How to search for linked RSS article contents?

I have created a search set with a single RSS feed set to crawl mode https://blog.testdouble.com/index.xml

I want to search the contents of the articles present on that feed.

Example: https://blog.testdouble.com/posts/2021-09-09-how-to-build-a-search-engine-with-ruby-on-rails/ contains the term “postgres” multiple times.

However if I search for postgres, that article doesn’t come up. It appears that DA only searches the RSS preview content, rather than the page content itself.

I have also tried setting “follow links” to no avail.

How can I set up a search set that will crawl all of the links in an RSS feed, and search the contents?

1 Like

After enabling following of links (one level) this should work as desired.

That’s what I thought, but it doesn’t, which is why I’m posting here. Here’s the search set:

Blog RSS.agentSet.zip (882 Bytes)

Thank you! Currently following of links is only applied to matching pages in case of feeds, the next release will improve this.

Great to hear! I think it would be useful to get similar behavior with a sitemap.xml or sitemap.txt as well.