Competitive Site Analysis: Crawl, Search, and Download Entire Site

I want to use DevonAgent to download and monitor the different competitor websites.

Using DevonAgent I have set up two separate Search approaches: Crawl a domain, and Search (results from Google for that domain).

I’m surprised at how different the crawl vs search count is:

  • Crawl: 45 results
  • Search: 1,664

and when I download the entire site using DT3 I see

  • Entire Site Pages: 2,344 pages for the domain (not counting assets)

Any reason for such large variance in results.

Aside from the technical issues I’m really just wanting to review and monitor several competitive domains. DA3 seems like the right tool but am trying to make sure I’m setting it up correctly for this purpose.

These approaches are quite different and not comparable. Crawling downloads & searches on its own, the number of pages depends on the settings (especially following of links). Searching uses the results provided by Google/Bing for this domain. Sometimes one or the other approach might return more results. Downloading a site on the other hand includes of course additional resources like images.

1 Like