Searching for documents based on Markdown front-matter

[stuartro]

Any way to expose metadata in Markdown documents to DT?

Removing quotation marks from titles (where they wrap titles in YAML, but aren’t part of the names) just became a lot easier (for me) in version 3.5. It can now be done in a Smart Rule using regular expressions. I’ve scanned the name for the regular expression ^[\"\'](.+)[\"\']$ which means: “Look for a name with single or double quotes at the beginning and end, and if you find that, capture the text between them.” Then I replace the name with the captured text, which is just \1, or in other wo…

I have started using Markdown front-matter for tags (instead of hash-tags), for example converting:

#someTag
#someOtherTag

to

---
tags: someTag, someOtherTag
---

This works very well and, if Settings > Files > Import > Convert hashtags to tags is checked, then tags entered via Markdown front-matter are correctly “detected”.

As I’ve extended my use of Markdown front-matter, I’ve started adding other front-matter attributes. For example, I use an indexLevel attribute to “label” documents, e.g.:

---
tags: someTag, someOtherTag
indexLevel: 1
---

I’ve noticed, however, that Smart Groups and/or searches don’t “find”/search the front-matter text. That is, to continue the example, if I search for “indexLevel” I don’t find any documents—even though they have the text “indexLevel” in the Markdown front-matter.

Is there a way to have DEVONthink search on attributes / values declared in Markdown front-matter? If not, might it be possible to extend DEVONthink to enable such?

[BLUEFROG]

The raw Markdown source is not indexed by default.
Go into the built-in Help > Documentation > Appendix > Hidden Preferences and click the On link for IndexRawMarkdownSource.

[stuartro]

Wow, that was fast! Thank you. DEVONthink and the DT dev team never cease to amaze me.

Two questions:

1. The documentation contains the, rather ominous, text: “Note: These hidden preferences are not an official feature and could be changed or removed with any update.” Is IndexRawMarkdownSource and the associated functionality likely to be removed at some point, or can I safely depend on it sticking around?

2. After enabling (i.e. quitting DT, running defaults write com.devon-technologies.think3 IndexRawMarkdownSource -bool TRUE from the command-line, and then re-launching DT) I see that existing notes aren’t “automatically” re-indexed. If I edit a note and save, though, then its Markdown contents is fully indexed. Is there a way to force a re-index of all existing notes?

[BLUEFROG]

You’re welcome and thanks for being amazed

1. Considering the functionality was just added, I’d say it’s safe to assume it won’t be removed soon.
2. Make sure your local backups (i.e., not sync) are current and do a File > Rebuild Database.

[stuartro]

Rebuild worked flawlessly, and Markdown front-matter attributes/values are now found via Smart Groups and search. Thank you.

[BLUEFROG]

You’re very welcome

[stuartro]

Is there some way I can create a Smart Group that searches for specific Markdown files based on Markdown front-matter attributes? I see there is a metadata search option in the Smart Group editor, but it doesn’t seem to do what I was hoping.

Following on from the notetype example mentioned above, I’d like to be able to create a Smart Group that finds only notes where notetype:meeting, say.

[BLUEFROG]

Your metadata is nothing but words in the frontmatter, so you can just search for text:"notetype:meeting".

And obviously you’ll need to have already activated IndexRawMarkdownSource in the Appendix > Hidden Preferences section of the built-in Help and manual.

[stuartro]

Thanks Jim.

In the Smart Group editor dialog, I couldn’t find any way to specify "text: " but using the “Content” search works just as well:

And, yes, I have IndexRawMarkdownSource turned on.

[BLUEFROG]

text: is the search prefix used in a toolbar search. content: is also synonymous. And you care correct, that it’s Content in the smart group criteria.