Some returned results are "Too big"

#1

Hi,

When crawling pages like dblp.org/db/conf/aaai/index.html, and following the “contents” links (one level only), I often get “Too big” log errors. I would like to know if there is a customizable size limit for this kind of situation. Or am I doing something obviously wrong?

Thanks in advance,
Best regards

#2

Yes. It’s in DEVONagent’s Preferences > Search > Max. Download Size. I have mine set to 8192 KB, but I don’t suggest going higher than that, in general.