This seems so basic I feel sure I’m missing something but here goes…
When I search for a URL DT won’t find it in a Markdown document unless I switch it to source mode.
So if I have a markdown that contains:
[Some Website](https://discourse.devontechnologies.com)
a DT search won’t find https://discourse.devontechnologies.com but if I find the document and switch it to source view it will find it.
Am I missing something?
Thanks
If you searched for Some Website the document would be found.
DEVONthink no longer indexes the source of Markdown documents, only the rendered content.
Ug. Oh well, at least I’m not going mad.
Is it unreasonable to ask why the source is no longer indexed?
This problem arose because I was trying to check if I’d already made a note about a particular site I was looking at. When I create Markdown links by taking page title and url via a KM action it doesn’t always give me an easy to figure title search - hence the need to search for the domain.
Thanks
Because
@gg378 recently presented a solution that makes searching the markdown source possible again by writing the source in custom meta data. See Backlinks (aka Incoming Links) revisited: DTTG. You could adopt this for your needs. If you need help let me know
If you are not interest in the backlink issue, you can just use “part 1” of the Applescript that I posted in this forum post (don’t use the version in the original post, but the one I added a few replies down; just delete everything from the comment “part 2” to the end, and you have a script that takes the list of selected markdown files and copies the source into a custom field).
If you are interested in backlinks, you should use @pete31’s script instead.
Totally off topic: an original post can be edited until someone posts a reply and then it is fixed? If so, I should not put any initial code into the thread leader, as it will eclipse the corrected version lower in the thread? Rather, I should write the OT with a general description and then immediately reply to myself with the code, which then can be kept up-to-date with future edits?
Here is a script that works for me. My custom field is called GGText. Depending what you use in your DT install, you have to change the property in the script.
(*
Purpose: put Markdown source into a custom metadata field, so that it becomes searchable
GG, 2020.09.14, updated 2020.09.15
1. Select all markdown files you want to include (in the simplest
case, make a smart folder that lists all markdown files in the DB, and then select them all).
- note: the script does not check explicitly whether all selected files are indeed
markdown (could be added, but I'm not an Applescript whiz, and have to move
on to other stuff now). I have not checked yet what it does to other file types.
Proceed at your own risk.
2. Run the script, wait, and done. Hopefully there are no major bugs that eluded me.
*)
use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions
property MainUUID : "" -- Not sure what this means
property CustomFieldName : "GGtext" -- name of the custom metadata field for the .md source
tell application id "DNtp"
set theSelection to the selection
repeat with theRecord in theSelection
set theText to plain text of theRecord -- copy the plain text of a markdown document into a custom metadata field
add custom meta data theText for CustomFieldName to theRecord -- add markdown source into the custom metadata field
end repeat
display notification "Success!"
end tell
In my experience, you can always edit your own posts.
And here’s a Smart Rule script
Matching: Kind is Markdown
Event: On Save
I would set Location to only the group (or database) that you need to find links in. Doesn’t make sense to have the markdown source in custom meta data if not necessary. To process existing records run it once manually.
-- Smart Rule - Write markdown text in custom meta data (to make it searchable)
property customMetadataName : "ggtext" -- name of the custom metadata field for the .md source.
on performSmartRule(theRecords)
tell application id "DNtp"
try
repeat with theRecord in theRecords
if type of theRecord = markdown then
set theText to plain text of theRecord
add custom meta data my trimEnd(theText) for customMetadataName to theRecord
end if
end repeat
on error error_message number error_number
if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
end try
end tell
end performSmartRule
on trimEnd(str)
local str, whiteSpace
try
set str to str as string
set whiteSpace to {character id 10, return, space, tab}
try
repeat while str's last character is in whiteSpace
set str to str's text 1 thru -2
end repeat
return str
on error number -1728
return ""
end try
on error eMsg number eNum
error "Can't trimEnd: " & eMsg number eNum
end try
end trimEndIf you use Safari you could look up the url with this script (after you’ve used one of the other scripts in this thread).
-- Search current Safari URL in markdown custom meta data
property customMetadataName : "ggtext" -- name of the custom metadata field for the .md source. Make sure to use lowercase
tell application id "com.apple.Safari"
try
set theURL to URL of current tab of window 1
on error error_message number error_number
if the error_number is not -128 then display alert "Safari" message error_message as warning
return
end try
end tell
tell application id "DNtp"
try
set theResults to search "md" & customMetadataName & ":\"" & theURL & "\" kind:markdown"
if theResults ≠ {} then
set search results of (open window for record (root of current database)) to theResults
activate
else
display notification "URL not found"
end if
on error error_message number error_number
if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
return
end try
end tell
For a limited time, I don’t know specifically how long.
Right now, for my recent posts, all the ones that have no direct reply attached to them, still show the “edit” icon at the lower right, but those that have explicit replies attached (indicated on the lower left), are no longer editable.
Perhaps it’s the browser that you are using? On a Discourse forum, I’ve never seen the ability to edit posts go away, although I expect it may be an option available to the administrator of the forum. Here’s an earlier post of mine today, still able to edit even though I have a direct reply.
I just checked again: All my posts in this thread (whether with an explicit reply or not) are still editable, i.e. show the edit icon that you circled above. All my posts in this thread have the edit icon missing as of now (but originally it was there, as I edited some of my posts). So it does not look like a global browser issue (I’m using the latest Safari release). Maybe some specific setting for that other thread?
I used Google Chrome, and same thing.
Edits in that other thread are not editable. In that case, the edit icon shows up at the top of the post, and if you click it, it will show the edit history, but will not let you revise anything.
Thanks to all for the suggestions. I have to admit to feeling disappointed that I can’t find plain text in a plain text file. Adding metadata as a workaround to make DT work seems to go against the sensible simplicity of using Markdown to store notes. Or maybe it’s just me.
Having to remember to run a script whenever I add a link to a document, or even thinking of a way of having it run more automatically and/or regularly is extra work and more stuff to think about and maintain. I’m wondering about just indexing all Markdown - at least then Finder searches work.
But just to be clear, I am grateful for the suggestions and might come around to them if it looks like the “only way”.
It’s not only you, Yesterday I also had to find out that (new) markdown records don’t index footnotes anymore. I couldn’t find a note about that in the version history. I think such changes should at least be transparent to the users …
Once you’ve set up a Smart Rule with the script I’ve posted there should be no additional maintenance necessary. Did you try that?
Yes, I understand that this feels a bit deflating. Happened to me. However, upon some reflection, one comes to the conclusion that in the end, markdown files are not really plain text files. If you do treat them this way, you will gain searching un-rendered contents, but you will start having trouble finding rendered text, because the markup characters get in the way (searches for phrases will become hazardous, because they might contain formatting chars; that’s why RTF has to store text separately). So for the devs, it’s damned if you do, and damned if you don’t. I guess one solution could be to offer two kinds of searches “kind: md” and “kind:mdr” for “raw markdown”. But that’s evermore fragmenting things. So in the end, copying the plain full text into a metadata field is actually not a bad idea at all. I think for many use cases, markdown notes will contain a lot less content than the zillion pdfs and whatnot, so DB size and responsiveness would be largely unaffected. In other words, integrating the above posted Applescript into the main app would make this transparent to the end user, no need to set up scripts and smart rules.
It’s not only you, Yesterday I also had to find out that (new) markdown records don’t index footnotes anymore.
One thing that worries me about markdown is that it seems not really nearly as future proof as usually claimed. One is highly dependent on the rendering engines made available and how they do what. I have to hope that if I now commit a lot of md files in DT, the provided MMD renderer stays backward compatible for a long time. (It is definitely future proof in the sense that, when everything else fails, the raw source is always interpretable, but that’s really a last resort.)
Markdown is a bit weird. It’s a great idea. The first iterations were woefully lacking features. MMD is now kind of OK feature-wise. But the fragmentation into “flavours” is discouraging.
I have been using PmWiki for about 15 years, and find that its styling is very similar in the sense that things are rather readable in plain text. But it offers a lot more options (text sizes etc). I feel it could easily take the place of md. Markdown is a bit like iOS: great to get started, but denying the advanced user more options.
I have to admit to feeling disappointed that I can’t find plain text in a plain text file.
If you were using a plain text file, then yes you could search for the URL and find it.
Markdown is not merely a plain text file and indexing the underlying code adds noise to the index.
It’s been explained already. I get it. This in a non helpful reply. Congratulations on your customer alienation skills.
