Would it be possible to allow an external filter to be attached to a queued download, so that links were followed if the external filter answered “true” when handed a link URL that the user wanted, and “false” when handed one the user didn’t want?
Specifically, I’m thinking it would be great to be able to say “follow all links that contain subdirectories with these names” type things, but implementing the parser into your code would be an unnecessary pain, and place unnecessary limits. If, on the other hand, I could specify an external (ie) Perl script, I could write regular expressions that were as sophisticated as I liked, and have the script hand back decisions for DTP to decide whether it should download that link target.