Hello, I am trying to develop a plugin that searches the ArXiv using their API. I am aware that there are issues crawling the main site, but my understanding is that the “export” version should work for this purpose. I am using the following code:
but, when I test it, it returns zero links even though when I click on the addresses that are searched I can see some results in ATOM format. I have removed arxiv from the excluded domains but this didn’t solve the problem. Can anyone please help me troubleshoot this? Thanks!
There’s an unnecessary space right before https:. In addition, the results page is actually not HTML but XML (Atom) and this requires that the plugin specifies the keys ResultsKeyPath, LinksKeyPath and optionally e.g. DescriptionKeyPath and TitleKeyPath. At least theoretically because in this case it doesn’t work due to the MIME type, a future release should improve this.
Than you for your prompt reply and your help, I would have never figured this out by myself!
I had read the documentation before posting this question, but hadn’t appreciated that the keys ResultsKeyPath and LinksKeyPath were required for XML – they are listed together with DescriptionKeyPath and TitleKeyPath in the “JSON AND XML-SPECIFIC KEYS” section. I am wondering if it would make sense to move them to the section “NECESSARY KEYS” instead, with the added caveat that they are necessary for XML.
I will play around with your code and might post some follow up questions if I can think of ways to improve it that I am unable to implement by myself.