search set problems

DEVONagent Tips
1

I sincerely would like to use DA’s search feature for search sets but I don’t find I am getting the results I should be.

I have a search set for journal articles. I took the exact same URL’s and created a Google custom search. I am getting the results I should from Google but not from the DA search.

Does DA search for all URL’s with the pattern provided? In other words, if I have jnt.sagepub.com will it find every URL that begins with that? It doesn’t seem to be doing it (and I do have it following 5 levels, the max allowed).

If this is not how the searching works, then can I request that it be this way for the next release? Also (and I have suggested this to Google co-op as well) you should create the ability to have ‘wildcard’ URL’s.

For instance, Cambridge journals, very unfortunately, does not have a clear URL pattern for each specific journal. But each individual periodical has an id number. So the URL of an article may be cambridge.journals.com/ajdklfjd_id=7y8_jahdjfkdhghjahjfdh. Everything that I have underlined is always unique to the journal that I want to search. I would love to be able to specify cambridge.journals.com/ajdklfjd_id=7y8* in the DA search set, so that the * can represent anything following.

Danny

2

Following links is not based on matching URLs, it’s based on the link description/title or if the link is located near a matching text phrase. The only alternative right now is to use “*” (without quotes) as the term to follow.

3

Sorry Christian I just need clarification because I must be understanding. Does this mean I need to put an asterisk after every URL I have designated in my search set?

I just tested it. I had the URL jhj.sagepub.com/
I added an asterisk jhj.sagepub.com/*

And searched for a name ‘Crossley’. I am getting no hits. I know that it isn’t working, because when I put this search in Google
Crossley site:http://jhj.sagepub.com/

I get 23 hits, about half of them are PDF’s.

What is it I am doing wrong?

4

The URLs listed in the Sites tab of the Search Sets panel are just starting points (see Marketing search set for example). DEVONagent downloads those URLs and scans them for the specified search term. But neither a search engine is used nor do the listed URLs specify which links to follow.

5

Okay I think I undestand a little better, but in the example I gave above, webpages with “Crossley” on it are within 3-5 links of the page specified in the search set. I have the search set searching up to 5 links deep and it is still not finding them.

As I said, I want to use DA’s search for my research but it is not giving me the results that a google custom search is giving me.

6

DEVONagent doesn’t follow all links by default. But if you want to follow all links, enter “*” (without quotes) as the term used to follow links. But following all links will definitely need “some” time and therefore it’s probably better to create a custom XML plugin for this site.

7

Thanks, this clarifies things. I think an xml plugin would be a good idea, but I am not quite savvy enough to do it (I read the user manual on it).

Is there anyone who has made an xml plugin based on a Google custom search? If so, it would be easier for me to replicate for my own purposes.

8

Here’s a quick start:

  1. Select DEVONagent in the Finder and choose “Show Package Contents”
  2. Navigate to Contents > PlugIns > Search
  3. Copy Google.plist to ~/Library/Application Support/DEVONagent/Plugins
  4. Open the copy in TextEdit (use UTF-8 for opening & saving)
  5. Modify the EngineURL - insert for example +site:devon-technologies.com after agentQuery:
    google.com/search?as_q=agen … entOffset
  6. Change the Info, Name and Identifier strings to whatever you like (a different Identifer is necessary)
  7. Launch DEVONagent and select the new plugin via the pop-up menu of the search window
9

Unless I am doing it wrong this doesn’t seem to be working. Your instructions seem very different from the User Guide explanation of adding a search engine (i.e. do a wide ranging search and take the resulting URL, etc.—I get lost after that).

My plist URL, according to your instructions above, is google.com/search?as_q=agentQuery+site:

The custom search that I am currently trying to make a plugin for is underlined above.

Danny

10

First, there are unncessary and tags in the URL and therefore this can’t work. Second, you’re not specifying a site but a URL to a Google custom search. This doesn’t work too as the goal of the above instructions is to create a plugin searching a dedicated website (e.g. devon-technologies.com) but not to create a plugin for a new search engine (including Google custom searches).