American Chemical Society plug-in is not working and help with plug-ins for scientific reports and conference proceedings

[samrose]

I am trying to learn how to create plugins that search websites containing conference proceedings, such as https://asmedigitalcollection.asme.org/IDETC-CIE. To do that, I tried to go through the example in the Help documentation to create a plugin for the ACS. I could not follow the instructions to the letter because for some reason, when I created the new ACS plugin, the Test option is greyed out, so I cannot go through and exclude any domains. However, I tried to just write the plug-in as specified in the help, but I get no results. Then I tried using the built-in plug-in for the American Chemical Society (under Science), but I could not get that to work either. When I select that plug-in and search for Mercury, I get no results. Now I am just very confused.

Are there any known compatibility issues or updates that would make it so that the built-in American Chemical Society plug-in would return no results when searching for Mercury? I am using the latest version of DevonAgent (3.11.7) and MacOS 14.1.2. I tried clearing the DevonAgent cache and toggling whether javascript was disabled or not, but get the same result.

Here are some screenshots to show how I am trying to use the built-in ACS plug-in:

Once I figure that out, if you have any advice or examples that would help me create a plug-in for a site such as https://asmedigitalcollection.asme.org/IDETC-CIE, I would appreciate it. I have tried many many iterations of trying to follow the instructions in the help manual and cannot get good results (i.e., I cannot get results that include what I find when I search the same term on the website).

Thanks in advance.

[BLUEFROG]

It is the holiday season and i am not at my Mac. Disable JavaScript in Preferences > Web then press Command-N to open a DEVONagent browser. Go to the site and try a search. If JavaScript is required you can’t build a plugin for it.

[samrose]

I disabled javascript and followed your instructions. I am able to go to https://search.acs.org (slightly different than the address you use in the help example, but it takes me to the ACS search page) and search for Mercury. I get plenty of results.

Glad you are taking a break =). When you get back to it, any other suggestions would be appreciated.

[BLUEFROG]

Side note from the site you mentioned:

[samrose]

Thank you, it is good to know that I won’t be able to use a plug-in for the ASME site or any site that requires javascript. However, there are some other sites with pdf reports that I would like to automate searches for, provided that they don’t require javascript. I would thus like to figure out how to make the example plug-in work. The American Chemical Society plug-in is a built-in function, and the ACS site does not seem to require javascript based on the test above. Do you have any insight on why it might still be showing me zero results when I search for a common term like Mercury using this plug-in?

I also want to confirm that you are saying DevonAgent cannot automate searches of any site that requires javascript. I believe this is true if I cannot create a plug-in for the site, which you confirmed above, but I want to confirm that there is no other way. I thought that perhaps I could create a search set that has the ASME site under Sites, but then I am not sure what plug-in to use. If there is some way to automate a search of a site that uses javascript, please let me know.

[BLUEFROG]

You could try this…

That actually uses search engines to do a domain search. But bear in mind, this site is issuing many 429 errors, meaning it’s rate limiting the search engines.

I don’t know if we have any way or working around that.

[samrose]

Thanks. That was what I originally tried to do, but I couldn’t find a configuration that would reproduce the search results that I would get using the ASME search field on the original site, which is why I was trying to create a plug-in. It would give me only 1 - 2 results instead of about 10 and the results kept changing when I reran the search set with the same search terms.

There are now two issues: (1) finding a way to automate the ASME search using DevonAgent (which may not be possible) and (2) finding a way to automate the search of reports and proceedings on other sites, which leads to the issues I have getting the built-in ACS plug-in to work. Presumably, there are some report or proceedings sites that DevonAgent is capable of automatically searching (i.e., those without javascript and without 429 errors), suggested by the fact that there is a built-in plug-in for ACS. However, I cannot get the ACS plug-in to work as expected. I am doing something wrong or the plug-in no longer works. Since this is the plug-in used in the help documentation to teach us how to create our own plug-ins, I am not able to create my own plug-ins at all. Do you have any guidance on this? Does the ACS (American Chemical Society) built-in plug-in show you results when you search for a common term like Mercury, or does this plug-in no longer work?

[BLUEFROG]

@samrose

Does the ACS (American Chemical Society) built-in plug-in show you results when you search for a common term like Mercury, or does this plug-in no longer work?

Yes, but I think it may be throttling the search similar to what Google does. If you run too many concurrent or successive searches with a Google plugin, it can lock you out for a period of time. That could be several hours and it’s nothing we control. I was getting ACS search results following the tutorial but ran into the zero results after a time.

[kewms]

FWIW, I haven’t tried the ACS plug-in, but run into similar issues searching the IEEE library. To such an extent that I’ve effectively given up on DevonAgent.

[BLUEFROG]

Not sure of the issue but I just ran this search…

[kewms]

I get nothing like those results.

I checked, and I have the latest version of DevonAgent, but I do not have a DT-supplied IEEEXplore plug-in. Is it downloadable from somewhere?

Edit: Or, to put it another way, what has changed since this thread?

Search help: IEEE Xplore

I have to admit, I’ve always been a bit flummoxed by DevonAgent, especially when it comes to searches where it’s not just a pretty front end for Google. But I now have a task where it would come in pretty handy, and I’m stuck. I’d like to use it as a front end to search the IEEE digital library, the entrance page for which is here: IEEE Xplore My naive attempt to create a search plugin resulted in DA using both Google and Bing to search that specific page with, of course, zero results. Help? (…

[BLUEFROG]

Ahh, I switched machines and that’s in an internal build for the next release.

[kewms]

I guess I should be eagerly looking forward to the next release, then. Sigh…

[samrose]

I actually do have IEEEXplore in my current version of DevonAgent (3.11.7). It is under Science. I am also able to see the results that you see under IEEEXplore when I search for modular. This is promising! However, I still get no results for the American Chemical Society. Below is the log when I try to search for Mercury using the ACS plug-in. Since I would like to get sites similar to the ACS to work, and it does not require javascript, and it does not show 429 errors like the ASME site – do you have any insight on why this might not be working for me?

One strange thing is that now, after running the ACS search in DevonAgent, I am no longer able to search the ACS site on my default browser. It shows 0 results no matter what I search for. Is this a sign that I am making too many requests with DevonAgent and getting locked out? I don’t know enough about how searches typically work.

My hope is to create a search set that includes about 10 databases of reports and conference proceedings that I can easily search with new search terms or possibly set up an automated search. These include things like ASME IDETC (which looks like it won’t work out, based on the above), the World Bank reports (Open Knowledge Repository), JMP (Reports | JMP), FAO (Home | Food and Agriculture Organization of the United Nations), and Issue Lab (https://www.issuelab.org/special-collections) among others. If DevonAgent is not the right tool to do this, please let me know and I won’t spend more time on it.

[kewms]

Ah ha! DevonAgent lied to me! It said I was up to date, but I was actually back on 3.11.6. I’ve now installed 3.11.7, and the IEEEXplore plugin has appeared and gives good results.
(Pending further testing when I’m not pretending to be on vacation.)

Thank you @samrose!

[samrose]

I am still struggling to make my own plug-ins for searchable sites with reports. The one I am trying now is https://climatechange.issuelab.org. My test search is for the keyword ‘equity’ which brings up 16 results when I search on the issuelab site. I am trying to get a plug-in or search set to show me these results when I search for equity.

I’ve tried many variations, but the current plug-in I am using is

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Description</key>
	<string>Searches the web using &lt;climatechange.issuelab.org&gt;.</string>
	<key>EngineUrl</key>
	<string>https://climatechange.issuelab.org/1?publisher=&amp;doctype=&amp;issue_area=&amp;wikitopic_categories=&amp;keywords=_agentQuery_&amp;language=&amp;pubdate_start_year=1&amp;pubdate_end_year=1&amp;sort=&amp;categories=&amp;offset=_agentOffset_&amp;pageSize=50</string>
	<key>Identifier</key>
	<string>climatechange.issuelab.org</string>
	<key>Info</key>
	<string>Issue Lab Climate Plugin</string>
	<key>LinksMatching</key>
	<string>*climatechange.issuelab.org/resource/*</string>
	<key>Name</key>
	<string>Issue Lab Climate</string>
	<key>OffsetPerPage</key>
	<integer>1</integer>
	<key>Operators</key>
	<integer>59</integer>
	<key>ParseLinks</key>
	<true/>
	<key>ResultsPerPage</key>
	<integer>50</integer>
	<key>Start</key>
	<integer>0</integer>
	<key>Version</key>
	<string>1.0</string>
</dict>
</plist>

(note I have also tried iterations with OffsetPerPage = 10)
I am trying to create the most general search set to try to replicate the results from the webpage search:

Under Plug-ins, I am only using the plug-in above with nothing under Sites (though I have tried iterations with climatechange.issuelab.org/resource/ and climatechange.issuelab.org under Sites with both crawl and search).

When I search for ‘equity’ using this search set, I see the following log:

When I click the top item, I see the results of the keyword search showing the 16 results that I want to see. However, for some reason, DevonAgent doesn’t give me any of these results. I see only two:

Since DevonAgent is clearly able to find the results in the log, I don’t understand why it won’t show me these results in the search. Can you help me create either a plug-in or a search set that will allow me to see the 16 results that come up on the search site?

Other iterations I have tried are to search for html & xhtml pages with a scanner to look for linked documents, thinking I could find the search term in the abstract or title. However, this also doesn’t work. I have also tried to change the EngineURL in the plug-in to be just climatechange.issuelab.org/resource/ because that is where all of the pdfs of the reports live, but this also doesn’t work.

I ultimately want DevonAgent to be able to search through all of the attached pdfs of the reports on this site for my search term, but I also want it to return hits that have my search term in the keywords, so figuring this out seems like a first step. However, if you can help me define a search that would return anything with my search term in the keywords or the pdfs of the reports, that would also be helpful.

[cgrunenberg]

@samrose

Here are some screenshots to show how I am trying to use the built-in ACS plug-in:

The next release will include an updated ACS plug-in.

[cgrunenberg]

@samrose

When I click the top item, I see the results of the keyword search showing the 16 results that I want to see. However, for some reason, DevonAgent doesn’t give me any of these results. I see only two:

Just disable filtering of similar results (see Settings tab), afterwards 12 results should be returned.

[BLUEFROG]

To underscore why Criss suggested disabling the filtering… see this in the Log tab…

Those are being ignored by the filter in the Settings tab.