Colon in the first line of a Markdown text creates pseudo-empty note

Two Months later. Problem still not fixed, still in danger of loosing content when the first paragraph of a Markdown note contains a colon and the markdown note is converted to RTF.

An uncommunicative friend it is. Just bits and pieces that without doubt are helpful for experienced Apple Script programmers but not for an occasional user like me. Or to put it another way: A dictionary alone does not make you speak a language. A grammar book is needed too.

Maybe a thorough explanation of how the elements work together would not even be necessary if there was a wide-ranged number of example scripts one could steal from.

on performSmartRule(theRecords)
    	tell application id "DNtp"
    		repeat with theRecord in theRecords
    			set the first character of the rich text of theRecord to "#"
    		end repeat
    	end tell
end performSmartRule

This is not much of a script but already not working. Is it as suspected the “set” line? I did try a number of variations of this but to no avail. Or is the whole theRecord of theRecords thing wrong?

Wether it is the first character or the first paragraph I change does not matter. I simply do not get access to it.

Still looking for the script to alter rich text. Anybody, please?

Try this (carefully on one test file…) since I am no expert.

make new paragraph at beginning of text of theRecord with data "#"

Problem still not fixed, still in danger of loosing content when the first paragraph of a Markdown note contains a colon and the markdown note is converted to RTF.

This is not a problem to be fixed. It is part of the MultiMarkdown specification.
Also, just adding an empty line at the top of the file when you start composing will ignore the line as metadata.


Regarding a simple method of adding text to an RTF document, here’s a simple approach noting it only works on the currently displayed rich text file.

tell application id "DNtp"
	set sel to content record -- This is the active document
	tell text of sel -- Talking to the text of the document
		make new paragraph with data ("#" & return) at beginning
	end tell
end tell

Thank you for the script. Can it be used in a Smart Rule? And could you provide the second script that will remove the first paragraph again (after the conversion to RTF) too?

I know it is part of the MMD specification. Still there lies a problem: Converting from Markdown to RTF eats the first paragraph(s) when there is a colon in it. Wether it is actual meta-data or, like mostly in my case, just some content with a colon in it—it gets lost in the conversion. An option to ignore MMD meta-data in the conversion (and maybe in the rendering of the Markdown file too) might be useful. Or the meta-data could be generally kept in the conversion to RTF.

Can it be used in a Smart Rule?

No it can’t be used in a smart rule as it only acts on the rich text file being actively shown in the view/edit pane.

And could you provide the second script that will remove the first paragraph again (after the conversion to RTF) too?

tell application id "DNtp"
	set sel to content record -- This is the active document
	tell text of sel -- Talking to the text of the document
		set paragraph 1 to "" -- Set the first paragraph to an empty string, this removing it.
	end tell
end tell

And no, this can’t be used in a smart rule either.

Still there lies a problem: Converting from Markdown to RTF eats the first paragraph(s) when there is a colon in it. Wether it is actual meta-data or, like mostly in my case, just some content with a colon in it—it gets lost in the conversion.

Again, this is not a bug. It is functioning properly. The MMD metadata is not rendered, therefore it will not be part of text of the conversion.

But is there any script possible that can be used in a Smart Rule?

I got it that it is not a bug (this topic is still filed under “bugs” though because I can’t change that anymore). A bug would be something unintended. This is intended—and wrong because it might destroy (meta) data in the conversion to other formats.

To prevent this some Markdown apps have options on how to treat MMD meta-data. Be it a general “compatiblity mode” (i. e. just basic Markdown) or a more elaborate solution of turning on and off the rendering of single MMD features. See Drafts (screenshot below), Marked, or Fletcher T. Penney’s very own MultiMarkdown Composer.

Drafts MMD settings

But is there any script possible that can be used in a Smart Rule?

Modifying text in an RTF file in this way requires the document to be selected and previewed, so it can’t be used in a smart rule.
Also, editing the underlying code in an RTF is possible but not a simple thing to accomplish.

Now that is finally a definite statement. Just not the one I was hoping for.

So what can I hope for? A setting in DEVONthink that allows to ignore MMD meta-data? A note taking app for iOS/iPadOS that allows to write instantly after launching it, automatically saves to, say, iCloud drive so DT on the Mac can index it—like 1Writer but only with rich text?

I did try this and that and as far as I got—can’t say which obstacles I would have to surmount later in the process—what made it not a simple thing was not the RTF specifications but Markdown. RTF requires a document wrapper and wrappers for italics, bold, and footnotes. That would cover the basics.

I again faced problems because of Markdown’s usage of asterisks and square brackets &c. as markup. But asterisks and square brackets—like the colon, sigh—do have non-Markdown meanings already. So a script would have to tell apart the two different usages of these characters. For example, imagine a footnote with […] in it. The script must not take the closing bracket of […] as the closing bracket of the footnote. And so forth …

Before I dive into this once again I have to know if it does make sense at all. Would after all a script changing the plain text of a record work in a Smart Rule?

I am now able to answer my question myself: Yes. Or better: YES!!!

I’ve done it. After all this time I found a solution. A solution with two restrictions though.

And to everyone else who might be interested in de-metadataing the MMD metadata here is what I did: I simply replaced every : in the Markdown file by \\:. With this Smart Rule script:

on performSmartRule(theRecords)
	tell application id "DNtp"
		repeat with theRecord in theRecords
			set theText to plain text of theRecord as string
			set theText to my findAndReplaceInText(theText, ":", "\\:")
			set plain text of theRecord to theText
		end repeat
	end tell
end performSmartRule

on findAndReplaceInText(theText, theSearchString, theReplacementString)
	set AppleScript's text item delimiters to theSearchString
	set theTextItems to every text item of theText
	set AppleScript's text item delimiters to theReplacementString
	set theText to theTextItems as string
	set AppleScript's text item delimiters to ""
	return theText
end findAndReplaceInText

The double \\ is necessary to actually get a single \ into the file.

DEVONthink’s current MMD engine displays the \: only as : in the rendered Markdown file. Marked, for example, that works with MMD displays them rendered as-is.

And that’s the first restriction, at least for those who want to keep the MMD file in DEVONthink. The alleged metadata is automatically converted into data and the rendered file looks fine in DT. But not necessarily in other Markdown apps because of possibly visible backslashes.

Since I want to convert the MMD files into RTF files that does not matter to me. To compensate, the second restriction applies to me: Since adding the execution of my “Markdown to RTF preliminaries” script to the Smart Rule it does not delete the MMD file anymore so in the end I got both it and the RTF file.

Markdown to RTF Preliminaries

Any idea why the deletion does not work anymore? I have tried different positions of the script execution in the Smart Rule already but without effect.

In DEVONthink, hold the Option key and select Help > Report Bug. Do NOT send the email. Double-click the Console.log and check it for errors near the end.

And this is what I found (a whole bunch of them actually, each occurring when the MMD to RTF Smart Rule kicked in):

2020-04-27 17:41:15.888 DEVONthink 3[17424:273301] WARNING: Couldn't lock file: KRecord <BC122039-2FB5-443C-BC4B-5D1124F7C437> (Plain Text: "Test Text")

I didn’t see any issue with this. See if it persists after quitting and relaunching DEVONthink.

It does. I have done actually quite a number of restarts of DEVONthink today after I had found out it sometimes cached my constantly evolving script instead of using the latest and saved version.

@cgrunenberg would have to intepret that Console entry.

My memory: not quite as good as it was back in the days but still kicking in every now and then.

After fruitlessly trying literally all possible orders of the different steps of my Smart Rule I remembered something from the early days of playing around with the new Smart Rules in the DEVONthink 3 beta. I had noticed something I wanted to dedicate a topic to here in these forums but—well, the memory, see above.

What I noticed was that there is a divergent behaviour of a converting Smart Rule when moving an item to the trash and when deleting it. And no, I am not referring to the obvious and intended difference: Moving to trash moves a file to DEVONthink’s trash while Delete moves it to the OSses trash.

No, after a conversion of a file deleting or moving to trash could lead to different handlings of the original file. In one case it was in the respective trash can, in the other it wasn’t moved at all and kept together with the new file in the converted format. I am pretty sure “Move to trash” kept the source file without moving it because when I was experimenting with Smart Rules although I wanted to delete the source file I chose to only move it to DT’s own trash as a precaution. And only after I had switched to “Delete” to my surprise the conversion Smart Rule actually worked.

But here it is different: Before my script was in the Smart Rule “Delete” removed the old file and just kept the new. Now with the script in the Smart Rule “Move to Trash” removes the old file and keeps the new. I’d love to know why.

But what matters is, it works.

In both cases the old file is removed and the new one kept? What exactly is different then? That you have to use a different action?

Usually a harmless message and just for debugging. In most cases this is logged by background tasks which can’t access a file as it’s currently in use (e.g. by AppleScript, the user or another background task). But automatic retries will be performed anyway.

So far I couldn’t reproduce this:

  1. Is the Markdown document actually an indexed one? The first action tries to move it into the database but there’s no such condition

  2. Is the destination of the “Move” action in the same or a different database?

Before the script to replace the colons by backslash-backslash-colons in the MMD file was part of the Smart Rule it worked fine with “Delete” as the last step: RTF file was there, the source MMD file was deleted. After the Script was in the (beside that unmodified) Smart Rule “Delete” did not delete the MMD file anymore. But “Move to Trash” worked and moved it to the trash.

The Markdown files come from 1Writer on my iPad and are saved in its iCloud drive folder. DEVONthink indeed indexes the folder and moves every file in it into a database where it gets converted into an RTF file.

The indexed folder is part of the database the file is moved into, if that is what you mean.

A screenshot of Preferences > Media might be useful - maybe this issue requires certain settings.