Capture remote directory by URL like wget?

I have having difficulty finding what I am looking for, in part, because I am having difficulty describing it. It seems like something that DEVONAgent would do if I knew how to use it properly. Maybe this is built into DEVONThink?

I would like to download a remote URL, directory-like, location and store it in DEVONThink as a web archive or pdf file. I think of it in terms of two steps.

  • Download the remote URL into a local directory, with the links altered to local links. Something like the following.
    wget --execute=“robots=off” --mirror --convert-links --no-parent --wait=3 --local-encoding=utf-8 --header=‘Accept-Charset: utf-8’ http://www.asite/location

There is an example for this here.

  • Convert the newly created directory of web content into a web archive, pdf, etc. DEVONThink does this well for a single page, but I do not know how to do this for more that one page or file. I see scripts in DEVONAgent that will grab linked images, but in this case they are linked text/HTML pages.

    pandoc -s -f html -t latex -o archive.pdf *.htm

Any suggestions?

Something like that but in a DEVON technology?

DEVONthink 3 Pro and Server have the Download Manager that can download remote resources. You can copy a URL and add it to Window > Download Manager.
This is covered in Help > Documentation > Windows > Download Manager.

You could then potentially use a smart rule that detects the HTML files, converts them to webarchives, then moves them to another location (or whatever).

1 Like