Convert URLs to Markdown v2 (no Applescript).

This approach uses Keyboard Maestro alone - if you don’t own it already get the demo from keyboardmaestro.com/main/ - I couldn’t use a mac without it (alongside Alfred and Default Folder, it’s one of the first programs I install on a new Mac).

As was kindly pointed out by Jim Neumann, you can initiate the Web Clipper via File > Import - so this has allowed me to dump the Applescript aspect of my previous approach. I’ve also fine tuned the timing aspects of the Macro and ensure that all interface control is automated via DEVONthink’s menu system (less errors this way).

I’ve attached the KM macro so you can easily import it into Keyboard Maestro but this screen-grab shows the core of what’s going on.

Korm asked that I explain my usage scenario a little more so here goes:

I have a huge archive of Pinboard bookmarks and I add at least 50 new bookmarks every week. Whilst Pinboard is great to search via tag, it’s full text search is flakey to say the least and lacks DEVONthink’s sophisticated search AI. For me to get the most out of all the research I store in Pinboard I download my bookmarks into DEVONthink (via the Applescript available via the Support Assistant in DT’s help menu), and then use my KM macro to bulk convert those bookmark files to Markdown content via the Readability API so the resulting Markdown is focused on the content only.

The reason I prefer Markdown as my storage format of choice within DEVONthink is that a captured page takes up a very small amount of space (because it’s a plain text file) - approx. 10k vs 1mb for the same content captured via PDF or Rich Text.

Some important things to consider when using the Macro:

  • You need to have set up the Web Clipper with your preferred options before you run the Macro (the Web Clipper always uses the last set of options by default

  • The Macro allows 5 seconds between each web page capture which should be enough time for the Web Clipper to do it’s thing but you can’t predict for slow websites so it’s good to chunk your import tasks to a manageable number (100 records at a time works for me) and watch for errors.

  • Some websites don’t allow themselves to be parsed via Readability (Fast Company & Read/Write spring to mind) but Readability allows for this and captures the source URL within the Markdown. You can use this source link to recapture the page as a PDF directly from the Markdown record (this occurs less than 5% of the time for my bookmarks but your own use case scenarios may differ).

  • Whenever capturing from the web it’s best to run automation tasks in the morning. Websites tend to respond quicker in the morning, particularly here in Europe when most of the USA is sleeping!

I find it best to automatically have DEVONthink render my Markdown documents in ‘Best Alternative’ view: View > Best Alternative. To make this the default view for Markdown documents run the following command in Terminal:


defaults write com.devon-technologies.thinkpro2 RenderMarkdown -bool TRUE

(use FALSE to change the behavior back to the standard)

If you want to edit the Markdown document, you can make it editable via View > Text Alternative

Apologies if this all seems a bit long winded but it was pointed out to me that many users of DEVONthink aren’t regular users of Markdown.

Hope you find the Macro useful.
Convert DEVONthink bookmarks to Markdown.zip (1.1 KB)

Thanks for the offering and the detailed notes. :smiley:

Pleasure.

For me the workflow is more interesting than the macro but I’m just wired that way I suppose. :slight_smile:

jm

I like this but I’ve noticed it will loop endlessly through a list, but I haven’t just done bookmarks with it yet — I’ve been using it to convert some web archives. Is that because I’m doing something different or did I mangle something in my import of the macro set?

The last step of the macro looks for the escape key to be pressed. And you can also easily stop the Keyboard Maestro engine via the menu bar icon.

Without using Applescript there’s no intelligent way of stopping the loop. The DEVONthink Applescript Suite lacks the ability to use the Readability API when converting to Markdown so I developed this alternative Keyboard Maestro approach but it is a workaround that lacks the programming sophistication of Applescript.

You can download an Applescript for managing Markdown conversion below, but the resulting Markdown is nowhere near as readable as it captures all of the HTML page elements including things like page navigation (due to the lack of the Readability API). It will however stop automatically at the end of the task without the need for user interaction.

To be honest Korm. I’m happy with it as it stands as it does what I need but if anybody wants to adapt it, that’s cool too. :slight_smile:

The Applescript Suite will be updated at some point to include the Readability API and at that point I’ll go swap back to the Applescript approach.

For some reason I was mashing Esc and not getting it to stop but I’ll try again. The menu item method is cumbersome because if it spawns a window it takes focus away from the menu.

I need to figure out how it styles the alternative view in DEVONthink because I’m finding the rendered document in DEVONthink is sized too small.

Thank you, by the way; I don’t mean to be whining about things when you’ve generously shared something. I don’t want to come off as ungrateful at all!

No problem. The thing with DEVONthink is that it’s infinitely tweakable so everybody has their install set up slightly differently. If you find the text is too small in DEVONthink’s window you can tweak this in the second tab of the preferences window. I have mine set to Avenir Next (point size 14) for both plain and rich text fonts. The pixel density of both my laptop & desktop is high so pick a point size that best suits your screen. By having both plain & rich text font settings kept the same I find it a smoother transition when swapping between reading/editing tasks via the ‘text view/best alternative’ command. Some people prefer a different font for editing tasks (e.g. a Monospaced font), it’s all down to user preference.

With regard to escaping the macro via the Quit Keyboard Maestro Engine command you can allocate a system wide key binding to this too (you can control KM itself via KM) which is pretty fail safe.

Hope this helps.