Convert Markdown to PDF / DOCX in Devonthink using Pandoc

Thanks @Bernardo_V & @chrillek for your tips.

I implemented both step by step, but still facing an error (in this example I am trying to convert a Doc in Devonthink named Thinking.docx into a md. doc:

pandoc: Thinking.docx.md: openBinaryFile: does not exist (No such file or directory)

Searching for this error code points to some path configuration problems but in my case path should be okey.

				-- Setup Your Temporary Folder Here:
				set theOutput to "/Users/MY_USER/Downloads/" & theName & ".md"
				
				-- Construct your personal command line options here:
				do shell script "export PATH=/Library/TeX/texbin:$PATH && /usr/local/bin/pandoc --wrap=none --extract-media=images" & " Path_to_Docx" & "\" -o \"" & theOutput

Should you have any other ideas, I would appreciate any help!

Is your input file

Or “thinking.docx”?

I checked again, it is “Thinking.docx”

image
Looking on the error message I wonder is there is something wrong with the line which appends .md to the name of the file.

set theOutput to "/Users/MY_USER/Downloads/" & theName & ".md"

Couldn’t tell from the error whether it occurred on input or output, and thought it might be trying to open a (non-existent) thinking.docx.md.

The other question is: Do you really have a folder named

(Do a Shift-Command-G in the finder and copy/paste the folder name in the dialog box’ input line). I’d suppose not - MY_USER is a placeholder for the real user name (like “jooz”, maybe).

This translates to
--extract-media=images Path_to_DocX on the shell level. Which is most probably NOT what you want, since Path_to_DocX is not a command or anything the shell understands. It is probably (!) an AppleScript variable, in which case the line might work better as

do shell script "export PATH=/Library/TeX/texbin:$PATH && /usr/local/bin/pandoc --wrap=none --extract-media=images " & Path_to_Docx & "\" -o \"" & theOutput

(Note the space after images!) I don’t know why the -o should be quoted here, but I don’t really care, that shouldn’t set off the shell one way or another.

However, if the path to your document(s) contain spaces, you might have to quote them like so

do shell script "export PATH=/Library/TeX/texbin:$PATH && /usr/local/bin/pandoc --wrap=none --extract-media=images \"" & Path_to_Docx & "\" -o \"" & theOutput & "\""

(not tested).
Unfortunately, all this is overly complicated. I’d rather go for a simple AppleScript that calls a shell script with the input file name as its only parameter. Then let the shell script do all the rest. Otherwise, you have all there problems with quoting, overly verbose and illegible shell calls etc.

Hei @chrillek i really appreciate all the detailed recommendations!

  • "/Users/MY_USER/Downloads/" does exist with the proper user name. That was the first thing i tested as well once i hit this error
  • Path_to_DocX sounds like a variable to me. The script above sets it as set Path_to_Doc to path of theRecord
  • if the path to your document(s) contain spaces > yes, good catch. It does indeed may include spaces
  • I tested the

do shell script “export PATH=/Library/TeX/texbin:$PATH && /usr/local/bin/pandoc --wrap=none --extract-media=images “” & Path_to_Docx & “” -o “” & theOutput & “””

And we are coming back to the undefined variable as you alluded to above.

image

Unfortunately, all this is overly complicated. I’d rather go for a simple AppleScript that calls a shell script with the input file name as its only parameter.

My initial idea was to use a shell script which would be fired up by Keyboard Maestro. But the integrated smart rule in DT3 is something I would prefer for this translation to happen automatically. I will look into your recommendation to re-build the flow and just use a simple shell embedded into apple script.

Btw, the reason I am trying to get this working is to finally have direct integration between iThoughts mindmaps and DT3. iThoughts is able to export into docx but not into md with embeeded pictures … so one needs to go via this workaround.

In the ideal world, it would be cool to have DT3 be able do translation from docx into markdown + images directly (either via a bespoke script in their library) or native feature.

iThoughts is able to export into docx but not into md with embeeded pictures … so one needs to go via this workaround.

Have you contacted iThoughts about them possibly adding support for this?

Yes sir. I hope that Craig (iThoughts developer) will consider implementing this. He was very responsive to the feedback in the past.
For now, I need to get this working with some bash.

Argh. I’m sorry, I didn’t notice that the backslashes where gone in this code snippet. The end should read
images \"" & Path_to_Docx & "\" -o \"" & the Output & "\""
What you’d want to see as a shell command is
export PATH=/Library/TeX/texbin:$PATH && /usr/local/bin/pandoc --wrap=none --extract-media=images "Document.docx" -o "Converted.md"
So you need quotes around the original document name and the converted one, to protect spaces and other special characters. And you need to make sure that not the string “Path_to_Docx” is passed to the shell but the value of the variable of that name.

I’m not quite sure that what you want to achieve (export a mindmap from iThoughts to Markdown) is feasible. As I see it, the mindmaps can be fairly complicated two-dimensional graphs. Is this really something that can be represented in Markdown? How so? On the other hand, wouldn’t PDF be a more useful format (and supported by iThoughts, too) because it can preserve the graphics? And a PDF should be searchable, too (like Markdown).

Fantastic. Thanks a lot for the help, I think I have moved a bit again.
The next error I am getting is:

Any ideas what it might be? :slight_smile:

I’m not quite sure that what you want to achieve (export a mindmap from iThoughts to Markdown) is feasible. As I see it, the mindmaps can be fairly complicated two-dimensional graphs. Is this really something that can be represented in Markdown? How so? On the other hand, wouldn’t PDF be a more useful format (and supported by iThoughts, too) because it can preserve the graphics? And a PDF should be searchable, too (like Markdown).

TL;DR: iThoughts for food for thoughts. DT for real thoughts.

That is pretty straightforward and it works like magic for me. I use iThoughts for mindmapping (as you say two dimensional graphs). For visual people like me that is great start instead of a blank piece of md paper. What iThoughts is doing during the export is that it creates a bullet list document based on parent-child relationships from your mindmap and preserves all embedded images (which is the best part!). Once you have a docx you can convert it into md and put into your knowledge management system/zettelkasten/howeveryoucallit in Devonthink for further processing, editing, etc.

I hope this is helpful.

I’d say that you’re (or the script is) trying to save the images to a read only file system.
Frankly, it’s there in plain English, there’s nothing anybody can do to make it clearer.

Check the pandoc documentation, figure out where it tries to save the images to and make sure that directory exists and is writable by you. Run the script from a command line, make sure it works there and then implement it in DT.

Don’t use docx. Don’t use pandoc.

I guess in a few moments you’ll be happy you mentioned him. Some time ago I wanted to know if there’s a better way than UI scripting to get a DEVONthink link from a selected node into an AppleScript. When I read “iThoughts developer” I realized that I had forgotten what it was that I asked him, so I reread his answer. He wrote that iThoughts itmz files are basically zips and that I could get what I was looking for directly from the file. Reading this made me curious if it would work in your scenario too. It does.

However it’s not fully automated. It takes four clicks to index a map as markdown record with pictures. If the destination group doesn’t change you could hard code it with get record with uuid or create location which would make it three clicks. If you know you’ll be using just one map for some time it could also make sense to hard code its path instead of choosing it every time.

The good news: Depending on the way you work (whether the mindmap doesn’t change after you’ve left iThoughts, or if you go back and forth between both apps and make changes) I thought it could be useful to update an already indexed markdown record, so if you choose a map which is already indexed as markdown it will be updated instead of indexed again. I guess this is quite near

One important thing: if a markdown record is updated the entire contents of its folder are moved to trash so DON’T PUT OTHER FILES IN THESE FOLDERS!

Let me know if it’s working for you.

-- Index iThoughts mindmap as markdown record (with pictures)

property theDefaultLocationPath : POSIX path of (path to documents folder) & "iThoughts" -- set default location of choose file dialog 
property theOutputPath : POSIX path of (path to desktop) & "Indexed iThoughts Markdown" -- set folder where markdown subfolders should live

tell application "SystemUIServer"
	activate
	set thePath to POSIX path of (choose file of type {"com.toketaware.uti.ithoughts.itmz"} default location theDefaultLocationPath)
end tell

set theName to (characters 1 thru -6 in (do shell script "basename " & quoted form of thePath)) as string
set thisOutputPath to theOutputPath & "/" & theName

try
	tell application "Finder" to move entire contents of folder (POSIX file thisOutputPath as alias) to trash
	do shell script "mkdir -p " & quoted form of thisOutputPath & " && unzip " & quoted form of thePath & " 'assets/*' 'text.md' -d " & quoted form of thisOutputPath
	do shell script "mv " & quoted form of (thisOutputPath & "/text.md") & space & quoted form of (thisOutputPath & "/" & theName & ".md")
on error
	do shell script "mkdir -p " & quoted form of thisOutputPath & " && unzip " & quoted form of thePath & " 'assets/*' 'text.md' -d " & quoted form of thisOutputPath
	tell application id "DNtp"
		try
			activate
			set theGroup to display group selector "Index iThoughts Markdown in:"
			set theRecord to indicate ((thisOutputPath & "/text.md") as string) to theGroup
			set name of theRecord to theName
			
		on error error_message number error_number
			if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
		end try
	end tell
end try

@jooz did you try the script?

Hi @pete31 - thanks a lot for your highly detailed explanation. I tried the script back in the summer but never got it working and somehow forgot to report back …

I am sure I am using the script incorrectly - i tried to use it as a script (not a smart rule).

Can you advise me on the first two lines:

property theDefaultLocationPath : POSIX path of (path to documents folder) & “iThoughts” – set default location of choose file dialog
property theOutputPath : POSIX path of (path to desktop) & “Indexed iThoughts Markdown” – set folder where markdown subfolders should live

I am supposed to change anything here?

Basically if I select an .itmz file and let this script run via a script DT3 menu - nothing happens on my end.

Did you try running the script in Script Editor? There you usually get an error message that could point you in the right direction.
From what I understand, you get to the point where you interactively select a itmz file? Then the two lines you mentioned are predicably not at fault. But then I might be wrong, @pete31 would know for sure.

property theDefaultLocationPath must be set to an existing folder, e.g. POSIX path of (path to documents folder). The path I used in the script is my iThoughts folder. Should have made this clear.

Anyway I’ll post a better version in a few minutes

Hi @jooz, the old script probably didn’t work as expected because you have no “iThoughts” folder in your “Documents” folder. I forgot to remove my paths and to make clear that you need to set the first property to an existing path.

Anyway, this old script was more of a test script that you could try to see if it would do what you’re looking for. By the way if something doesn’t work or you’re not sure how to use it it’s always a good idea to report that as otherwise there’s no chance to change it. :slight_smile:

Here’s a new version.

Setup:

  • Set property theChooseFromListDefaultPath to the folder which you want to see in the dialog (ideally the folder where you store your iThoughts files).

  • Set property theOutputPath to the folder in which the script should create subfolders in. Please make sure to use an empty folder (as the script tries to delete a sub-subfolder “assets” on each run). You won’t see this folder or a subfolder in DEVONthink as only the markdown file is indexed.

Don’t put any other files in the sub-subfolder “assets” as this folder will be deleted on each run.

-- Index or update iThoughts mindmap as markdown (with pictures)

-- Setup:
-- Set property theChooseFromListDefaultPath to the folder which you want to see in the choose from list dialog
-- Set property theOutputPath to the folder in which subfolders should be created. If the path doesn't exist it will be created.

-- Usage:
-- If you've made changes in iThoughts save them.
-- Run the script
-- Choose a mindmap
-- If the chosen mindmap is not indexed it will be indexed (you'll be asked in which group)
-- If the chosen mindmap is already indexed it will be updated (select it to see the updated content)

property theChooseFromListDefaultPath : "/Users/USER/Documents/iThoughts" -- set path of the folder which you want to see in the choose from list dialog
property theOutputPath : "/Users/USER/Desktop/test output" -- set path of the folder in which subfolders should be created. If path doesn't exist it will be created.

tell application "Finder"
	try
		try
			set theChooseFromListDefaultFolder to (POSIX file theChooseFromListDefaultPath as alias)
		on error
			error "Error: Path \"" & theChooseFromListDefaultPath & "\" doesn't exist."
		end try
		
	on error error_message number error_number
		activate
		if the error_number is not -128 then display alert "Finder" message error_message as warning
		return
	end try
end tell

tell application id "DNtp"
	try
		activate
		set thePath to POSIX path of (choose file of type {"com.toketaware.uti.ithoughts.itmz"} default location theChooseFromListDefaultPath)
		set theName to (characters 1 thru -6 in (do shell script "basename " & quoted form of thePath)) as string
		set thisOutputPath to theOutputPath & "/" & theName
		
	on error error_message number error_number
		if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
		return
	end try
end tell

tell application "Finder"
	try
		try
			set theOutputFolder to (POSIX file thisOutputPath as alias)
			set folderExists to true
		on error
			set folderExists to false
		end try
		
	on error error_message number error_number
		activate
		if the error_number is not -128 then display alert "Finder" message error_message as warning
		return
	end try
end tell

if folderExists = false then
	my extractFiles(thePath, thisOutputPath)
	
	tell application id "DNtp"
		try
			activate
			set theGroup to display group selector "Index iThoughts Markdown in:"
			set theRecord to indicate ((thisOutputPath & "/text.md") as string) to theGroup
			set name of theRecord to theName
			
		on error error_message number error_number
			if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
			return
		end try
	end tell
	
else
	try
		tell application "Finder" to move (POSIX file ((thisOutputPath & "/assets") as string) as alias) to trash
	end try
	my extractFiles(thePath, thisOutputPath)
	do shell script "mv " & quoted form of (thisOutputPath & "/text.md") & space & quoted form of (thisOutputPath & "/" & theName & ".md")
	
	tell application id "DNtp"
		try
			if exists (viewer window 1) then
				if (content record of viewer window 1) ≠ missing value then
					set theRecord to content record of viewer window 1
					if ((path of theRecord) as string) = ((thisOutputPath & "/" & theName & ".md") as string) then
						set selection of viewer window 1 to {} -- this is needed as the "synchronize" command doesn't update the displayed content
						set selection of viewer window 1 to {theRecord}
					end if
				end if
			end if
		on error error_message number error_number
			if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
		end try
	end tell
	
end if

on extractFiles(thePath, theOutputPath)
	do shell script "mkdir -p " & quoted form of theOutputPath
	set theListing to do shell script "unzip  -l " & quoted form of thePath
	if theListing does not contain "assets/" then
		do shell script "unzip " & quoted form of thePath & " 'text.md' -d " & quoted form of theOutputPath
	else
		do shell script "unzip " & quoted form of thePath & " 'assets/*' 'text.md' -d " & quoted form of theOutputPath
	end if
	tell application "Finder" to set modification date of (POSIX file ((theOutputPath & "/text.md") as string) as alias) to current date
end extractFiles

FWIW a footnote:

Is that right ? I think the reason for needing to export the path may actually be slightly different.

I believe AppleScript just uses a fresh vanilla instance (without any imported variable settings) of whatever the default system shell is set to be, which may well be zsh on recent systems.

For example, on this Catalina 10.15.7 system, evaluating:

do shell script "printenv"

yields a listing of the fairly sparse set of names bound in a vanilla shell environment, which, as you say includes only a very minimal value for PATH but, does, in fact does start with:

SHELL=/bin/zsh

Two practical conclusions:

  • We do need to explicitly set any PATH and other environment variable values which we are relying on. The do shell script shell instance is not initialised in the same way as Terminal.app instances, but
  • we may also need to be aware that the flavour of shell syntax available by default to AppleScript’s do shell script can vary between systems, and is determined (by default) by the system’s default shell setting ?

But I may be getting this wrong – perhaps others can advise us.

I notice that do shell script "$0" just yields a raw sh

1 Like

hei @pete31!

I was able to make this scrip run - and it works pretty nicely! I did test a bunch of mindmaps.

  • one of them resulted in an assets folder generated but no md file (still not sure what is so specific about that mindmap - will let you know if I find out)
  • encrypted mindmaps do not work which is WAI I assume

The absolute nirvana would be a bi-directional script which would be able to convert/update itmz -> md and the other way around depending on where the last changes have been made … :slight_smile: Not sure this is even feasible - just thinking out loudly.

Fantastic work! You made these two great apps work with each other really nicely!

And again this is an example of such an amazing community here on DT forum :slight_smile: Thank you very much @pete31 & @chrillek.

@pete31 you could suggest DT folks to include your script (if that is indeed ok for you) into the DT script library - this is an awesome asset!

1 Like