I have recently purchased Devon Agent. I have tried to create a plugin to search the NY Times site using as a template their site search string, replacing the terms in the string with ‘agentQuery’: thus “http://query.nytimes.com/search/sitesearch/#/agentQuery”
I then wrap this plug-in in a search set that enables me to enter a query string for the plugin and specifies very little except the plugin and 'Follow: on the same host".
I get no results in the digest. The Log shows that there are “15 links” from that query but none of them are provided by Dev Agt.
I have no exclusions specified in the search set.
I have a log-in for the NYT site that allows me to download the documents. I would like to have Dev Agt find the actual documents if possible. But entering the login details in the relevant tab of the Search Set window has no effect.
What am I doing wrong? Why don’t I get a list of the links to relevant documents? Or at the very least the page displaying those links on the NYT site?
Coud you please post the code of the plugin plus a screenshot of the search set? Thanks in advance!
Here they are… appreciate information on what I’m doing wrong.
The Plugin
<?xml version="1.0" encoding="UTF-8"?>
Description
Search for documents on the NYTimes site
EngineUrl
http://query.nytimes.com/search/sitesearch/#/agentQuery
Identifier
query.nytimes.com
Info
NYTimes Documents Plugin
Name
NYTimes Documents
OffsetPerPage
1
Operators
59
ParseLinks
ResultsPerPage
10
Start
0
Version
1.0
The search engine of the New York times is based on JavaScript, XML plugins can’t support such search engines. One workaround is to create a new plug-in via File > New plug-in with the URL nytimes.com/. The plug-in won’t use the search engine of the New York Times but Google/Bing instead. However, the results might be incomplete/not up-to-date.
Thank you Christian. NY Times appears to expose a ‘site-map’ to Google at the URL ‘spiderbites.nytimes.com’. So as you suggested I created a plug-in to query this site with Google.
The URL that Dev.Agt. composes from my query + plugin (“https://www.google.com.au/search?q=international+chamber+Qu�bec+1949+site%3Aspiderbites.nytimes.com”) produces nothing: “no links” in the log window.
But when I copy that URL into the address bar of Firefox I get a page-full of links to plausible NYTImes archive results.
May I ask for your guidance once more? What am I doing wrong? I cannot see anything in the ‘settings’ tab that would prevent the return of the links by Dev. Agt.
Thank you again (plugin text below):
<?xml version="1.0" encoding="UTF-8"?>
Description
Google query at spiderbites.nytimes.com
EngineUrl
https://www.google.com.au/search?q=agentQuery+site%3Aspiderbites.nytimes.com
Identifier
www.google.com.au
Info
NYTimes Site Map search Plugin
Name
NYTimes Site Map search
OffsetPerPage
1
Operators
59
ParseLinks
ResultsPerPage
10
Start
0
Version
1.0
That’s an issue of DEVONagent’s parsing of links (see ParseLinks key), the upcoming version 3.6 will fix this and should be available soon.
