At least in the last years DEVONthink indexed the Markdown source, not only the content. That’s what I‘ve build my whole Markdown system on.
I can’t search comments anymore.
I can’t search meta data anymore.
I used both extensively, mainly because
I like the idea of having everything in plain text which should never be a problem to access.
I don’t want to solely rely on e.g. an app’s meta data. Apps are discontinued etc.
Incidentally the fact that DEVONthink doesn’t treat Markdown anymore like it did in the last years just proved my fear of relying on an app.
„Unfortunately“ there‘s no other app I could use as there’s simply nothing that would even come close to DEVONthink.
I won’t give exact use cases, all I can say is that DEVONthink used to search the whole Markdown file, I‘ve built everything around that fact and suddenly what was good for years has been replaced.
I really don’t get why the new behavior replaced the old one. Adding would have been fine. Replacing is still unbelievable for me.
It’s actually not the real rendering but the parser strips certain elements which wouldn’t be part of the rendering. In case of foot notes this is just a bug and fixed.
We might add a hidden preference but you would have to rebuild the database afterwards.
Just noticed some extra strange behaviour. When I make a search targeting a text in the footnote, sometimes it does recognise it; other times it does not.
I’m attaching two screenshots that exemplify this. First I searched for Gibraltar and the term was found. Then I searched for Marinid (also in the same footnote text) and the term was NOT found.
We might add a hidden preference but you would have to rebuild the database afterwards.
That would be very helpful, Christian, thank you.
In the meantime, it would be helpful to have a full article on the implications of the changes, explaining exactly which elements of markdown files are no longer searchable, and whether this applies to all forms of search / smart groups / rules etc. E.g. At the moment, it’s not really clear what ‘rendered content’ means, and people are having to do their own testing. It would be helpful if there was a definitive statement.
Basically URLs of links/images (including the ones in the footnotes) and HTML tags. This effects all index based operations (search, see also, classify and concordance) and is actually intended to improve especially the results of these features and make them more consistent to RTF/HTML.
But that doesn’t include Metadata, which are neither URLs nor HTML.
Is internal metadata searched for or not? There’s a suggestion up thread that it isn’t, but I seem to be able to do it fine. This is a search for journal entries on Fridays…
Is that because I haven’t (knowingly) rebuilt this database? Or is metadata in fact still searchable?
Secondly, as far as I can see URLs are picked up — a search for ‘https’ or ‘x-devonthink-item’ seems to work fine. (It’s the ‘seems’ which is worrying… are some being missed?)
Again, is this by design (you do search certain parts of links), or will this (helpful) behaviour go away if I rebuild the database?
So, I think we need a far more detailed explanation for the changes than a sentence or two in this forum.
At the moment, this is causing some people to query whether the searches they have made are accurate or not — rightly or wrongly. DT3 is built on the idea of robust searching — a change in that process needs to be fully and loudly documented, even if (especially if) the concerns are largely illusory. I’m sure you don’t take these decisions lightly, but the consequences and implications need to be explained properly.
Finally, please could we have a preference to turn back to the old behaviour soon?
So, as far as you’re concerned, Metadata is searchable and will remain so?
That’s a relief, but it only underlines my point — someone coming across this thread will be 26 posts in before they get that confirmed.
BTW: it seems to go against the idea in the Release notes and earlier in the thread that only rendered text is searched — or it uses a definition of rendered which some will not have expected (does ‘rendered’ mean shown on the screen or not).
It seems to me that at the least help files need an explicit note in both the markdown and Search sections listing the new restrictions (not just saying ‘rendered’ as that’s open to interpretation). Apologies if it’s there, but I couldn’t find anything.
An article in the regular feed would also be helpful — if you’ve been using markdown search successfully for years, it’s unlikely you’ll check the help.
Hang on, now I’m really confused… The position seems to be changing and I can’t keep up.
Please can we have a definitive statement:
Can we currently search markdown files for metadata: i.e. data at the beginning of a file with the form Keyword: Data. Yes or no?
Are words within markdown links in the form [title](url) searchable or not? E.g. can we search for all items which contain say ‘https:’ or ‘x-devonthink-item’ within the (url) part of the link? Yes or not?
If the answer to either is No, then why can I find examples of both where the search definitely works? Is it because my databases haven’t been rebuilt yet (I assume)? Will I lose this behaviour if I do rebuild them?
The problem is, forum threads really aren’t a good way of getting accurate information out about this sort of issue: that’s why it needs to be addressed in one place as a specific, definitive response to which people can be pointed.
But metadata is indeed currently not indexed like links.
