I have some searches that I run each and every day. The searches are set to only return new results (its accuracy may be questionable). I would think that at some point it would tell me that there are no more results. However, each time I run it I come up with 300+ results. Not all of these can be new, its a relatively obscure topic that I’m searching.
Am I wrong to think there will be a point where there are no new results (or at least only 10-15)?
Look at the bottom of the Plugin tab of a search set, where there will be a limit imposed on the number of items downloaded per plugin for a search episode.
If a given search would find twice the number of items set by the limit (if the limit were at least doubled), it will take two scheduled search runs to download them, even if no new content had been added to the Web between the runs of the search set.
So, let me make sure I have this correct. If I search something somewhat specific like “oil and gas drilling in North Dakota”, google gives me 1.1 million results. By the 10th page, or so, the results become much less relevant and/or timely.
In DA, the default limit to the number of results per search engine is 100 and it looks like the mac would be 999 (3 digits). Even at 999, that would take me 1,100 runs of the search to work through Google’s results. And that’s just one search engine.
Is there a better way/workflow/setting to use DA to monitor a given industry or topic? I would have to think that with the included ability to create a search set, the program was designed with the concept of an “often repeated search” in mind.
I love the program, I’m just trying to bend it to work with my needs (or modify my needs to work with the program…).
Thanks,
Brad
DEVONagent is smart enough to filter and prioritize the results of a search, so that you won’t need to parse through all those pages included in Google’s results.
But my previous post indicated why a scheduled search will sometimes continue to find interesting “new” results in cases where a plugin would pull more results if the default limit were raised. The point is that there will be cases such that a list of new results doesn’t mean that the “new” results were not present on the Web at the time a previous search was run. Continue running that search set long enough, though, and at some point the results will be pages that were not present at the time the previous search was run.