Developing Science plugins

Hi all
long-time user of DTPO here who is only now beginning to see the potential of DevonAgent. One thing I noticed that a number of portals I frequently query in the course of my work do not work in the way described in the “develop your own plugin for DA” section here and in the manual. The symptom is that the search term in the query does not show up in the resulting URL after the search.
Some of these databases are a little more complex in that they allow the user to build a basic search AND/OR syntax and limit the query to certain fields (author, journal etc.) but perhaps this can be ignored during the first steps.

Is there anything that can be done?

Web of Knowledge
Zoological Record
Aquatic Sciences and Fisheries Abstracts (ASFA) & Oceanic Abstracts
Biological Abstracts Archives
Web of Science
Zoological record

Could you please post the used URLs or (even better) the code of the plugins you’ve created? Thanks!

Hello Christian

ok, here goes. There may be problems due to the fact that some or all of them are subscription-based. Unless you are connected through VPN to an institution with a subscriptiin it may or may not work for you.

Web of Knowledge or Web of Science home page

URL of a search run through WoS page … eralSearch
This is the page with the search results already, so I don’t see a variable for the search term I could target to adapt this to work with DevonAgent.

ASFA databbse
results of a search for “Antarctic” through ASFA … untid=8451

Zoological record database home:
Search results

I will have a look at ASFA again, looks like I missed the search term in the rather convoluted URL.

This might indeed be the case, I can’t access these sites unfortunately. Could you post the code of the plugins or send them to me? That way I could at least check whether the plugins should work as expected.

I would like to bump this thread as this type of search engine seems quite common.

One more example, the “aggregate” full-text search engine of New York University:

(please note that this link will not work if you are not authenticated, which in turn requires university credentials)

As you may note, the search engine produces a “group” (in this case #2011-11-26-001607) and “result set” (#007246 in this case). Obviously these numbers are unique for each search query.

The situation may not be hopeless, however, as the above URL is the final URL in a chain of automatically loaded URLs following the user’s entry of the search query. And the first URL in the chain (i.e. the URL immediately produced once the user submits the query) is:

where I have already replaced the query string with devonQuery

This last URL is perfectly manageable by DevonAgent. However, as I mentioned, it does not lead directly to the first page of search results. Instead, it loads a second (and often a third) page, in which the query string is represented by the “group” and “result set” numbers.

Can DevonAgent handle such situations? If not, could it be empowered in a future release to do so?

In principle, a DA script would need to be able to:

  1. Follow the chain of automatically loaded URLs (at the moment it seems unable to do so).
  2. Allow the script developer to insert the agentOffset key in a “wildcard”-containing URL, such as:

Thank you.

Could you please send me the complete URL chain plus the HTML source code of the results page? Then I might be able to figure out if it’s already possible or not. Thanks!

I’m also very interested in having more plugins that can search sites like Web Of Knowledge.
Any updates on those plugins?


What’s the URL of this website? Because I found several services named like this, sometimes requiring a subscription.