DevonAgent - Restrict Searched Pages by Published Dates

Does anyone know if there is a way to restrict DevonAgent Search Set to only pull data that has been published within a certain date? e.g. within the past 1week, etc to reduce the amount of data already reviewed >1week if scheduling auto running of Search Sets.

There have been several threads about using publication dates to limit searches. As Christian has noted, there are very limited options.

Just to complicate matters, I’ll caution that scheduled searches do not necessarily result in complete searches of all the relevant pages out there, even if publication date ranges could be specified.

In DEVONagent Pro, choose Window > Search Set. Pick any set and click on the Plugin tab. Note (at bottom of the panel) that there’s a number set as the maximum number of pages that will be examined and pulled into the search results list, for each plugin. This is user adjustable. There are other filters that could exclude potentially relevant pages out in the Web.

Thus, if you are scheduling weekly searches for a set, don’t assume that all the potentially relevant pages have been examined and archived for any one run, regardless of publication date. In principle, it could take a number of runs (or not) to capture highly relevant pages published in any date range, and this could vary for each plugin.

I’m usually satisfied with a setting of 100 results per plugin. For scheduled searches, on many of the topics I follow it could take multiple searches to begin to assume that the most recent search runs are covering “new stuff” on that topic. Of course, a long-running scheduled search will trend in that direction. :slight_smile:

Lots to think about here then and that even before I have looked at converting a search into a search set and then importing useful results into DTPO :slight_smile:

In the scenario of only wanting to search a set of specific websites that have published new articles can I use the following feature or do I need to limit the search using s Plugin like the “Google (last Week)”?:

Ideally if a previous search has pulled 53 articles from one website and another 1 has been published since last running the search would the above show just that 1 item within the search, without having to select additional “files” to be crawled/searched?

If this is correct are there any caveats :question:

Sounds correct.

Thats cool. 8)

I have noticed that one of my Search Sets does not complete the search after running for a few hours and I have to manually stop the search.

I have customised the search tool bar to now include “Add Selected Pages To Archive” or “Add All To Archive” and “Open Archive”.

Do I need to Add pages to the Archive to stop them being searched in the future or is this section of the App only for future reference, like a reading list because the following is operated whether the search stops automatically or manually?

When the items are Archived is there anyway to view them within the search window because there are some built in scripts that only seem to run within the search and browser windows :question:

If you are running a scheduled (i.e., repeated) search set, it’s critically important to check the option Add to Archive under the Actions tab. That’s so that the next run will report only the new items found, having filtered out results already saved to the Archive.

I"m not sure about your other comment. Most of my workflows for additional analysis of search results involve sending the results of a search (or an Archive for the search criteria) to DEVONthink.

Thanks for the Add to Archive tip via schedule :smiley:

I will have to look into the sending of results / archive to DTPO which format is best. I presume DTPO is better at extracting data than DevonAgent?