DevonAgent monitor webpage changes

DEVONagent Tips
1

New to DevonAgent and was unable to find my answer elsewhere. What I am trying to do is create a custom search set that runs at a scheduled interval on a series of select webpages. When a change is detected between the current webpage and the archived one I want it to email me.

Other forum posts mention that this is possible but I am unable to figure it out. The webpage contents are just a directory listing of zip files. The first run of the search set will add the page into the archive but when it runs again even if new files are displayed on the page DA will filter the page.

My search set is configured as follows:
General
Default Query: *
Filter: Similar Pages is checked
Filter: Archived Pages is checked
Scanner: Linked Binary Files
Advanced
Everything is checked
Results: Only new pages
Sites
I have listed a test page i am hosting locally to keep from constantly pinging a public site.
Plugins
None
Actions:
Archive the results is checked
Send mail to:

Please let me know what i am doing incorrect or if what I am trying to do is even possible. In the end I just want to schedule a search of websites for new files so that I know to download them. Thanks

2

The search set should work as expected if you’re either not archiving the results and/or disabling filtering of archived results.

3

Unfortunately I am still unable to get devonagent to work right.

I shut off filtering of archived pages and it returns a single result containing 2 zip files under linked binary files. I want devonagent to archive both zip files found or check to see if they are new. Now when I alter the website to have 3 zip files and rerun the search devonagent does show the page having 3 zip files.

I am wondering why when filtering of archived pages is enabled I do not get any results returned. It seems as though devonagent is not comparing the results for changes or I have done my query incorrectly. Any help would be appreciated.

4

Only the URL is compared to the archived pages. Therefore the “Only new pages” option of search sets is usually more useful which is actually comparing pages to the former results.