Most sites not searched with code "Disallowed"

DEVONagent Feedback
1

I’m running my first DEVONAgent search in a while, and noticed I am getting almost no results for simple queries that should return tons. When I look at the log, almost every site DA tries to access returns “Disallowed.” WikiPedia, Github, Medium, all are returning disallowed. Is this something where nobody wants their site indexed by AI, and DEVONAgent is just getting caught up in it? Is there any way around this? Anyone else having this issue?

2

These websites don’t allow crawlers & bots. There’s a preference to ignore these instructions but some websites might block your IP address temporarily in case of huge amounts of traffic caused by you.

3

It doesn’t help. Not sure what the error I got was, but disabling that setting did not work. What a shame, it drastically reduces the usefulness of DEVONAgent, a tool that I love. Internet search has become such a mess, and it looks like it is getting worse.

4

I can’t speak to GitHub or Medium, but the Wikipedia plugin still works perfectly for me. When you say sites return “Disallowed”, is that in the error column under the Log tab? (I don’t think I’ve ever seen that error, which doesn’t help with figuring out what’s going on here.) Just to eliminate the obvious, does switching the Settings between Search and Crawl make any difference?

5

You aren’t reporting default behavior here.

image
image1744×1584 414 KB

  • Did you try creating your own plugin?
  • Are you using a search set?
    • If so, which one?
    • If not a built-in one, select it in Search Sets, open the action menu (gear icon), and export the set. Post it or open a support ticket so we can inspect it.
6

I was using the built-in web “Deeper” plugin. I started a search just now so I could get a screenshot to send in with the error report, and it is not doing it now. ???

I can only guess that it was some kind of really weird network thing on my end. I’ve never seen it before. But right now all seems well. I am really relieved, because I seldom find the kinds of obscure stuff I am looking for with regular search engines anymore, DEVONAgent is a life saver for me.

In the immortal words of Gilda Radner: “Never mind.”

1 Like