Any way to expose metadata in Markdown documents to DT?

DEVONthink
21

I just checked and, yes, IndexRawMarkdownSource is enabled on both Macs. Bug?

22

I’m still running into this issue (and another related issue, which I’ll detail below). @BLUEFROG, any insight / updates on whether this is a bug?

The related issue is that, for some reason I haven’t been able to discern, it seems that front-matter attributes (which correctly indexed by turning on enabling IndexRawMarkdownSource) seem to “disappear” over time. From time to time I will search for notes based on the notetype front-matter attribute and find that not all notes are showing up. That is, if I have 5 notes which all have notetype=ABC, searching for notes with notetype = ABC returns only 3, say.

23

Does editing the files fix this issue? Do you edit them with DEVONthink To Go or external editors too?

24

No, editing doesn’t seem to fix the issue. To be honest, though, I haven’t spent much time trying to reproduce the issue. I will see if I can reproduce the issue with a test database tonight and will post here any "lessons learned’

I use Typora to edit all my notes.

25

As an aside: It would be super useful (to me, at least) if I were able to display front-matter attributes as columns in the various search and other “list” views. Not sure how practical this is from an implementation standpoint, but DT already has the various front-matter attributes “on hand” (since it is indexing them) and so, he said hopefully, might it be possible to allow front-matter attributes to be exposed similar to “custom metadata”?

26

A quick test shows the answer to the question “Does editing the files fix this issue?” is “Yes”.

Below is a screenshot showing all notes on SwiftUI, where all notes that have the blue SwiftUI icon have front-matter attribute notetype: code/swiftui.

image
image1696×2820 334 KB

Searching for all notes with search text notetype:code/swiftui* returns only two results:

image
image1702×884 75.6 KB

The second result (Convert SVG to inline CSS form) doesn’t have notetype: code/swiftui but instead has notetype: code/css. Seems the “front-matter indexing getting lost” issue is more widespread than I’d thought. And it has incorrectly indexed the note with notetype: code/css as matching notetype: code/swiftui.

After editing two files (All SwiftUI property wrappers edited within DT, and Options for custom navigation in SwiftUI edited within Typora), both files now correctly show up in a search for notetype:code/swiftui.

image
image1698×912 104 KB

Rebuilding the database works, but is a temporary solution as the “decay” slowly returns. And, rebuilding the database is not ideal (for me, at least) since the DB UUID changes, which breaks a number of Keyboard Maestro macros that move notes (and which which refer to the DB UUID). This is a small annoyance (since the UUID is defined in a single place in KM), though, and can “work” for now.

Further background on my setup:

  • DT version: v3.9.6 Pro Edition
  • Two Macs:
    • 16" MacBook Pro M1 Max (Apple Silicon), 32GB RAM
    • 27" iMac 5K 2019 (Intel), 96GB RAM
  • macOS: Sonoma 14.5 on both Macs
27

Maybe the hidden preference wasn’t enabled in case of older notes?

The search syntax is invalid, nodetype is not a valid prefix. Try "nodetype code css" instead.

28

Currently, this is not possible nor planned. Why don’t you actually use custom metadata as that is supported for displaying as columns?

search text notetype:code/swiftui*

In addition to the previous comments, the frontmatter is not actual metadata; it’s just text. So the proper search in your case would be text:"notetype: code/swiftui".

29

@BLUEFROG Thanks for correcting my mindunderstanding of searching for indexed front-matter attribs… I will use the pattern text:"notetype: code/swiftui" in future.

I have code which I use to parse my notes which is able to extract notetype (et al) from Markdown notes, and will see if I can create a Smart Rule which copies the value (from the Markdown front-matter attribute) into a “regular” DT custom metadata field.

30

That should be fairly simple to do… maybe. :slight_smile:

31

Works perfectly. DEVONthink truly is an amazing piece of software. DT’s comprehensive AppleScript interface is superb—I cannot think of any other piece of sofware that I have that is as “open” to automation.

The approach used is as follows:

  • When Markdown files are modified their front-matter is parsed and the notetype attribute value extracted
  • The notetype value is stored into the “Note Type” (mdnotetype) custom metadata value.
  • This is automated by a Smart Rule that runs on save and also every one minute. It needs to run every minute to pick up files that have been modified outside of DT (since I mostly edit Markdown notes using Typora).

I now have, essentially, all of what I was looking for:

  • Note Type can be displayed as a column :+1:
  • mdnotetype can be used to rapidly + precisely search for notes of specific types :+1:
  • I can work purely with Markdown and the Smart Rule automatically updates the mdnotetype value if I change the type of a note (which I sometimes do as I work). :+1:

Below is a (admittedly contrived) search for all notes whose Note Type contains code/s whose names contain the word create:

image
image1536×1532 173 KB

And now my previous search for notes of type code/swiftui finds all of the notes (instantly):

image
image1844×2816 358 KB

It would be (very) cool if DEVONthink could watch for files that are modified externally and allow a Smart Rule to trigger “Upon Modification Externally”. For my own code I use the Witness Swift package to monitor the filesystem for modified files and trigger updates in response.

32

We’ll consider this for future releases.