Level of search sliders and aggregator sites

Hello your helpfulness,

Is there a way to get DA to search an aggregator site such as ccel.org/? Right now I am following the links down to where the target text file resides and then cut & pasting. What I would like is to build a set that finds every link, then automatically goes down ‘n’ levels before surrendering.

I also do not understand the relationship between the two ‘levels’ sliders in the set general prefs and the one under the sprocket. Do they move the same parameter, or are they distinct?

Thanks again for the best tech support in the business.

David:

It would be possible to develop a plugin for the CCEL site, but I doubt you want to do that.

Another approach is to open the site at one of its topical search pages and copy the URL. For example, go to http://www.ccel.org/index/subject.html, which lets you browse the site for various topics.

Let’s create a new search set for the CCEL site that as default will search for the Philosophy category.

[1] In DA select Tools > Edit Search Sets. Choose Internet (Fast Scan).

[2] Click on the cogwheel at the bottom left and choose Duplicate. Name this new Search Set as “CCEL” (without quotes).

[3] As you wish to restrict this set to only searching the CCEL site, click on the Sites tab. In the Location field triple-click to select the contents and press Command-V to paste in the URL you previously copied. Now click on the Plugins tab and delete the existing check for plugins (click in the checkbox that shows a “-” symbol to clear it). Result: Your search set will start with the designated CCEL site.

[4] Now lets set up a query. Click on the General tab. In the default field enter “Philosophy” (without quotes). Comment: You may want to create several unique search sets for this site. If so, you can name each one differently.

[5] Next, let’s enter the name of a particular philosopher in the second query field. Example: Pascal. Now the search set will look for and download only pages that contain this term.

[6] But we need to go further. Pascal’s name doesn’t appear anywhere on that first page DEVONagent will examine, so we need to tell DEVONagent to “drill down” and look for other pages under the link for Philosophy that may contain the term Pascal. To force DEVONagent to check further down among all links enter the term “*” (apostrophe, without quotes) in the Follow Links field.

[7] Next, decide how many levels of links DEVONagent will drill down in its search for pages containing “Pascal”. Warning: the more links, the longer the search will take, especially as some of the servers linked to the CCEL site seem rather slow. Two or three levels will probably be sufficient, so adjust the slider as you wish.

[8] Close the Edit Search Sets window.

Now we’ll use the set for a search:

[9[ Select File > New Search. Click on the Loupe (magnifying glass) triangle symbol and, from the list of displayed sets, choose the name of your custom set, e.g. CCEL. Click on the Start button. I’ve already done a test, so I know that at 3 levels you will find some search results. But it will be slow. Realize that you are “hammering” the site with lots of hits. If it’s a slow server, it will take time.

Comment: If you want a lot of results more quickly for the philosopher Pascal, just choose Internet (Fast Scan) and enter his name.

Note: Your custom search set is “hard-wired” to search for Pascal. If you want to use it to search for Aristotle you will need to open the set again in Tools > Edit Search Sets and change the second query field from Pascal to Aristotle. Yes, my test search set did include a page that included Aristotle as well as Pascal.

Hope this helps.

Yes but this requires a dedicated XML plugin.

Strange. There’s only one slider in the “General” tab of the “Search Sets” panel over here. Which system language do you use?

The Follow Links sliders in the Search Sets panel and in the drawer of Search windows are identical - you can use the drawer to temporarily change settings of the selected set.