When I try to reach clusty.com/forbidden.html.
However opening clusty.com with Firefox or Safari on the same machine works fine.
I have just updated to DEVONagent 2.0 but it is the same behaviour as in the previous version.
Do only I have this problem? Are there any tips regarding this behaviour?
Thank you very much for your suggestions,
Stephan
Clusty does not like bots and as someone did a heavy scan of their website using DEVONagent, the user agent of DEVONagent is now blocked by them.
Clusty does not like bots and as someone did a heavy scan of their website using DEVONagent,
the user agent of DEVONagent is now blocked by them.
Thank you for getting back to me so fast. However, your answer gives rise to my next question:
Does anybody know, whether I have any possibility to change the user agent of DEVONagent - at least for the Browser utility of DEVONagent (so they may block the search utility of DEVONagent as long as they want to)?
Since I usually use Clusty and can only add with DEVONagent web archives easily to DEVONthink I really would like to be able to access Clusty via DEVONagent.
Thank you for your help,
Stephan
Clusty provides a contact address on its web site: clusty@clusty.com
Why not write to them and explain the situation? DA must have several thousand users by now; it’s unfair to punish all of them for one bot’s abuse.
Maybe if a lot of us write, the situation would change. I just sent them a note.
Well, guess I was a little naive about the power of communication.
Here’s the response from clusty.com:
"DevonAgent does not respect the robots.txt file on our site which prohibits crawling our search results. This creates so many requests that it overwhelms our servers.
We have contacted Devon Technologies to alert them that they are not following our crawling restriction and they did not think this was a problem. If Devon Technologies complies with our robots.txt directive, we will unblock them. Please feel free to contact them yourself: devon-technologies.com/about/contact.html"
So they are blaming the software, instead of the rogue user. Go figure