Are there any plans to offer a custom pre-processing option for Markdown in DEVONthink (something similar to the corresponding Marked 2 feature)?
No, currently there are no such plans. The request is noted but consider this: who will get the support tickets when the preprocessor doesnât work as you hope it would?
DEVONthink is not a bespoke Markdown application. It is not going to offer all the flavors, features, and extensions of Markdown. If you need something that custom, many thirdy-party apps play nicely with DEVONthink.
1 Like
Whatâs the result of that process supposed to be? Another text format or HTML?
In the former case: you can script that in DT. In the latter case: you can script that in DT
No need to invent anything. Just select your markdown (or HTML) document in DEVONthink and Open With > Marked 2.
Marked 2 is used for converting and saving markdown or HTML files into a different format. Of course that can be done with DEVONthink.
The only real âbenefitâ of using Marked 2 with a markdown file from DEVONthink, as I described, is the ability to assign a different style sheet before exporting. (But, that can be done in DEVONthink by adding the code to the file itself.)
(Downside is Marked 2 is buggy and crash prone.)
4 Likes
The pre-processor takes the original Markdown text (on STDIN) and outputs a (potentially) modified Markdown text (on STDOUT). The purpose of this option is to enable effecting temporary changes to the original Markdown text in order to improve compatibility with the Markdown processor/renderer present in DEVONthink.
The value of a pre-processor is that you can have Markdown documents created and maintained by third-party Markdown editors (of which there are many that are far superior to the Markdown editing facilities available in DEVONthink), whilst still being able to view and interact with these documents in the context of DEVONthink databases as intended when the Markdown document was created/edited in the external editor.
There are many âflavoursâ of Markdown across editors and processors/renderers, often exhibiting subtle differences that can have big impacts on the user experience when viewing or interacting with the rendered Markdown.
A good example is custom IDs for headings in Markdown documents. Quite a few Markdown editors use â{#AnchorID}â, others (including DEVONthink) use â[AnchorID]â, while Obsidian, for example, uses â^AnchorIDâ. Furthermore, there are different approaches to automatically generating heading IDs â with some Markdown editors/processors simply discarding whitespaces and other reserved symbols, while others replace them with dashes. As a result, it is very difficult to get links to headings to work when using several Markdown tools with the one Markdown document.
In general, if one uses an external Markdown editor for notes stored in DEVEONthink (possibly, together with other dedicated tools for producing and publishing the end results â e.g. Marked 2, Pandoc, etc), the different Markdown syntaxes present in all these different tools can cause many inconveniences which impact on the user experience in DEVONthink.
It would be unreasonable to expect DEVONthink (or any other third-party tool) to support all Markdown âflavoursâ, or even just the more popular ones. However, having the option of a pre-processor (or some form of an API to hook into the Markdown processing/rendering engine), enables end-users to address compatibility issues without having to ask the DEVONthink development team to make changes to their Markdown processing/rendering engine.
Some of the compatibility issues can be addressed through custom CSS and JavaScript, but it tends to be more cumbersome (as itâs done âafter the factâ â i.e. post-processing).
1 Like
Just curious. In the meantime, or instead-of, canât you use the âOpen Withâ feature of DEVONthink to open with any editor with the flavour you prefer? Does not work for you?
1 Like
I understand the reluctance to âopen upâ the Markdown processing/rendering engine present in DEVONthink, and the (legitimate) concerns regarding potential side-effects that may be introduced by custom pre-processing. However, I would argue that these concerns already exist with custom CSS and JavaScript â features already present in DEVONthink. And, indeed, custom CSS+JavaScript can be employed to address some of the issues that can be solved with a Markdown pre-processor (though, not all) â but it is (generally) more cumbersome and less efficient. Also, the resulting HTML (i.e. if exported) would end up being a lot âmessierâ.
Furthermore, the few end-users that would go through the trouble of implementing a custom pre-processor would likely have the programming and debugging skills to be self-sufficient. I would expect more support tickets linked to custom CSS and JavaScript than what you might get from custom pre-processors.
I would also argue that debugging pre-processors is a lot easier than custom CSS and JavaScript. Given how they work â (STDIN â STDOUT) with both input and output being Markdown â it is very easy to inspect and debug results by comparing input with output using any of the many text diff-ing tools available. They are also easier to profile (in terms of runtime performance), as they can be executed in isolation.
It is exactly because of this that offering the option of a Markdown pre-processor makes sense. This way, compatibility across the many Markdown âflavoursâ can be addressed outside of DEVONthink, reducing demands on the DEVONthink development team. In fact, it could also be a vehicle for the DEVONthink development team to offer âquick-fixesâ (or, even, diagnostics) for end-users encountering certain problems â i.e. via (potentially temporary) custom/bespoke pre-processing scripts.
I would argue that offering the option of having custom Markdown pre-processors in DEVONthink is the âbiggest bang for the buckâ you could get in relation to Markdown facilities in DEVONthink. The implementation should be rather trivial â as all it takes is a small intermediate step of taking the Markdown text and passing in via STDIN to an external executable and, after getting its output via STDOUT, hooking it back into your existing Markdown processing code. Sure, there will be some overhead in terms of error handling, but it shouldnât be too onerous (keeping in mind that the target audience is technical and with a small footprint).
CSS has nothing to do with what you want. Itâs simply indispensable to achieve the desired visual appearance of HTML. And it certainly does not âmess it upâ if itâs put in an external file, as it should be.
And why is the HTML âmessinessâ even of any concern? Itâs not meant for humans to read, and as long as itâs valid, the browser will deal with it.
To me, itâs not quite clear what your goal is.
Do you want to have HTML files for publication? Then why do you need DT for that?
Do you want to write MD notes in ever which way and have them appear as you want inside DT, ie as preview? Why do you need different MD dialects in this case?
Try this in several Markdown editors (including DEVONthink), as well as in Marked 2 (or other Markdown processors/renderers):
# Section 1
Some text
# Section 2
See [Section 1](#section-1)
The above will work in Typora and Nota (and on GitHub), but not in DEVONthink (or in Marked 2).
Now, try this:
# Section 1 {#section-1}
Some text
# Section 2
See [Section 1](#section-1)
Now it works in Marked 2 (and most Markdown editors â though not in Obsidian, without plug-ins), but still doesnât work in DEVONthink.
This works in DEVONthink, but it doesnât in most other Markdown tools:
# Section 1 [section-1]
Some text
# Section 2
See [Section 1](#section-1)
Obsidian uses â^Anchorâ for custom heading IDs, which doesnât work in pretty much any other Markdown tool.
This is just one use-case scenario. There are many others that have to do with differences between Markdown âflavoursâ.
Of course, if you only use DEVONthink for editing, viewing, and exporting Markdown, itâs a non-issue. I use all of the Markdown tools I mentioned above (and more). I come across such compatibility issues often. Some are more inconvenient than others. Some can be âfixedâ with custom JavaScript (embedded in Markdown, or in the resulting HTML), but it is often quite cumbersome and results in a âmessierâ HTML (which may matter, if you publish the resulting HTML).
The option of a custom Markdown pre-processor in DEVONthink would help address incompatibility issues in an elegant and efficient manner. A fairly trivial regular expressions script could âfixâ the custom heading IDs issues showed in the examples above (e.g. replace â{#...}â with â[...]â at the end of headings, or replace â^...â with â[...]â if using Obsidian).
I have no problem understanding that there are different flavors of MD. What I do not understand: what do you want to achieve.
Say you have an MD file in Obsidian with markup thatâs particular to this app. Now, you want to import that file into DT. Than a simple smart rule with an included script could fix the fixable markup so that DT preview shows what you want.
Or do you want to index that file in DT and have the preview in DT show what you want? In that case, DT would have to create an intermediate files that it than passes through is MD processor and previews. And it would have to do that every timeâŚ
It seems to me, that thereâs already a solution for this 2nd use case, namely what you described as the preprocessor in Marked2. Why not simply user that fort previewing the indexed files?
This 
I spend a lot of time in DEVONthink, as well as in Obsidian (and other Markdown editors). I create and update information in Obsidian (mostly; but I use other Markdown editors and tools for other purposes â e.g. stand alone documents, maintaining blogs, producing documentation sets, writing books, etc). I capture and research information in DEVONthink. I have hundreds of PDFs indexed in my DEVONthink databases. Most of the Markdown documents I produce also end up indexed in one (or more) of my DEVONthink databases. Btw, I think the ability to index (and then cross-reference) external documents in DEVONthink is a âkiller featureâ.
Anyway, I do not want to keep switching between applications when Iâm âin the flowâ. If Iâm researching something, I tend to sit in DEVONthink. I even try to avoid hitting the (external) browser, if I can. Iâve built up extensive knowledge bases in my DEVONthink databases, that I tap in on daily basis. I often spend several hours a day in DEVONthink (not necessarily continuously, of course).
I edit my Markdown documents in different (external) tools, depending on what the document is for. Some I edit in Obsidian, some in Typora, some in Nota, some in Visual Studio Code, etc. However, if thereâs a small edit I need to effect on a Markdown note, I just do it directly in DEVONthink. This is one of the beauties of Markdown â itâs ârichâ text without vendor lock-in.
Other than a few niggling issues, I think the Markdown viewer in DEVONthink is excellent. And the fact that I can cross-reference the information I create with the information I capture from other sources is of tremendous value. The search in DEVONthink is excellent, too (only slightly inferior to FoxTrot Search â at least for my needs).
So, DEVONthink is my âgo toâ place for information. I would just like to retain (as much as possible) the same user experience when viewing and interacting with Markdown in DEVONthink as I have in the external editors (and other tools) I use. Itâs funny how, at least for me, the content of a document triggers âmuscle memoryâ when it comes to how I expect to interact with it and how I expect it to be displayed. Maybe Iâm just an edge case, I donât know⌠I just find it grating when I try to click on a link and it doesnât work, or when I see something that I expect to be hidden in preview mode (like a custom heading ID), or I come across an admonition that is not rendered accordingly, etc.
I could take care of all of these issues (and many more) myself if DEVONthink would offer the option of a custom pre-processor for Markdown. However, I may be biased in my approach, as I already have several pre-processor scripts that I use for Marked 2 â which, by the way, would also give me a head-start for pre-processing Markdown for DEVONthink (in fact, I think Iâd already be about 90% there, since the preview in DEVONthink is actually quite good, and not that far off from Marked 2 â in terms of Markdown syntax supported).
1 Like
A lot of words here, probably not convincing DEVONtech to change, but again, as @rmschne and I suggested, why not just use Open With to open the document in Marked 2. We can use whatever syntax we want in a text file or markdown file in DEVONthink and get the desired effects by opening the file in Marked 2.
2 Likes
Iâll wager that in my last 16 years here watching and listening, I could probably predict that more than 90% of enhancement requests on this forum and its predecessor can be met by using âopen withâ to send documents to external apps or editors.
4 Likes
I can also think of several instances of people being told that there is no way theyâre going to support X feature of Markdown only to have it mysteriously appear later in the release notes. âWe only support Multimarkdown syntaxâ then later checkboxes show up, or strikethrough with ~~, for example (neither of which are Multimarkdown features). Way back in 2019 Bluefrog suggested, in the ~~ for strikethrough example, that no they canât account for other flavors of Markdown yet here we are in 2023 and heâs saying the same thing in this thread. The OP isnât asking for âall flavors, features, and extensions of Markdownâ (despite the team seemingly adding them slowly), theyâre asking for a fully-optional custom preprocessor option that runs prior to generating the preview.
Does anyone remember when CriticMarkup, and Mermaid.js, and prism.js for code syntax highlighting were added? I do but I donât remember hearing much complaining when those features were added oddly enough. And funny this is what Fletcher Penney says about CriticMarkup:
The CriticMarkup syntax is fairly straightforward. The key thing to remember is that CriticMarkup is processed before any other MultiMarkdown is handled. Itâs almost like a separate layer on top of the MultiMarkdown syntax.
Huh. Processed before. Almost sounds like a preprocessor to me. 
So no, I donât think this is a bad idea at all. Itâs quite similar to adding Mermaid or prism support. And if itâs anything like Marked 2 it would be a couple of config items that require someone to not only check a box to enable but provide paths to the preprocessor binaries that run prior to the preview being generated. This is not average user stuff.
Is it ahead of the 5 billion other things the team probably has on the list? Not necessarily but doesnât mean it shouldnât be done. And I suspect if/when it is the folks chiming in here with âwell Y other feature is good enough because thats what we doâ will start using the preprocessor but never let us know 
2 Likes
I said this:
I use search and I use linking (including cross-references) in DEVONthink to navigate my knowledge bases. I jump around to find relevant information.There are many types of documents I keep in DEVONthink: web pages (archive), PDFs, Markdown documents, etc. They can all be previewed in DEVONthink. This is outstanding!
Of course I can open web pages in the browser, PDFs in Preview (or other PDF viewers/editors), Markdown in the many editors/viewers that are available. Equally, I can search for text across documents of different types using Spotlight (or one of the many more advanced front-ends to Spotlight â eg. HoudahSpot) or, even better, FoxTrot Search. I also have Hookmark to help with cross-references.
Does this mean I should just use these other applications (which are each, arguably, better at performing the corresponding functions) and give up on DEVONthink altogether?
All Iâm asking for is a simple way (implementation-wise â i.e. for the DEVONthink team) of extending/customising the way Markdown is displayed in DEVONthink, so that I can spend more time in DEVONthink than I do in Marked 2, for example. Surely, this is not a bad thing for DEVONthink (or its user base)âŚ
Of course many DEVONthink enhancement requests can be met via external apps or editors. Thatâs where the requests come from; users being accustomed to certain features available in third-party applications and asking for them to be implemented in DEVONthink â so that, like I said above, they can spend more time in DEVONthink and not have to switch to these third-party applications.
Funny you should mention CriticMarkup, because this is another use-case (for me) where I could use a Markdown pre-processing option in DEVONthink. I didnât have to address this in my Marked 2 pre-processors, because Marked 2 has excellent support for CriticMarkup.
Ironically, providing a Markdown pre-processing option could remove quite a few of these â5 billion other thingsâ â from the subset that is related to Markdown, anyway.
Generally speaking, providing an API and/or hooks in an application results in dramatic improvements to its functionality through community effort â and quite quickly, too (look at Obsidian, to give just one example; or Visual Studio Code, to give another).
DEVONthink already has powerful scripting facilities â which is great. It also offers the ability to customise its Markdown viewer through CSS+JavaScript â this is also great. As far as Iâm concerned, all that itâs missing in terms of customising the Markdown viewer is the option of a pre-processor. Because this could take care of several Markdown-related things that are missing (or could be improved) in DEVONthink.