Looking for help in diagnosing a specialized search engine plugin

[jstarek]

I need to (once again) integrate the BASE search engine (Bielefeld Academic Search engine) into DEVONagent. This was working a few years ago, but now, when trying to create the plugin, DEVONagent does not seem to see any results at all on the results page.

Steps to reproduce:

• A typical query URL would be https://www.base-search.net/Search/Results?type=all&lookfor=LOFAR&ling=0&oaboost=1&name=&thes=&refid=dcresde&newsearch=1
  • Note that the engine lists around 10.000 hits on its results page, most under with dx.doi.org
• Open DEVONagent → Modules & Scanner window. Create new module. Paste search URL. Replace “LOFAR” with “_agentQuery_ .
• Click test.
• There seem to be only a few irrelevant addresses in the result set. Not a single doi.org entry.

Can anyone suggest how I could get this to work? The results page looks to be pretty standard HTML to me, so I don’t know how to proceed…

[cgrunenberg]

@jstarek

This was working a few years ago, but now, when trying to create the plugin, DEVONagent does not seem to see any results at all on the results page

This page uses now an anti-bot detection to validate that it’s only accessed by humans.

[jstarek]

Oh, I see. Thanks for the quick reply. I’ll see if I can give it a custom user agent string…