In praise of Markdown: making Chat work hard and well

So, @BLUEFROG years ago motivated me to use markdown (and I believe @chrillek too), so I did.

These past few weeks I finally got a handle on using Chat/MCP to get good stuff that was all over the place in my databases into far more coherent buckets. Among them, thousands of food recipes.

Not only I managed to get Chat to categorize (e.g., Soups, Fish, etc.), but also to do a whole lot of consolidation - which would be harder using batch commands, smart groups (though not impossible, Chat is just such a breeze).

And I have been learning how to write propmts to get pretty complex things done. I always try on small subsets of duplicated records.

One thing I was quite interested in: lots of old (10, 15 years) lists whose links ended up broken (probably by moving them in and out of databases, some not even existing anymore).

So I tried using Chat to search for matches to broken links - and it did, very successfully (in the majority of cases, an original pdf or image whas still somewhere). What I did was asking Chat to APPEND each list with the new findings.

It was great to do it en masse, and so fast. But there was an issue: my lists were all rtf files. And Chat said “If I try to append an rtf, formatting will be lost” - that meant also losing links. And in fact it converted to plain text, and broke all links.

But I then converted the old rtf files to markdown, and Chat successfully appended search matches at the end of them. Image below. I am a happy camper.

I know “See also” and other DT tools can help, but this was quite targeted and selective. Please let me know if I’ve been reinventing the wheel :slight_smile:

5 Likes

So you saw the light :wink:

Links to …? External broken links are something not even AI can fix (yet?).

From the reminder of your post, I guess that you are referring to links to other items in the database. Probably item links.

I’m wondering how your AI finds broken links in an RTF. I just fiddled around a bit:

  • created a dummy text file in a test database
  • added an item link to it to an RTF file (same database)
  • link showed up in the link inspector (or whatever it’s called) and was reported by a trivial script checking for outgoing references
  • removed the dummy text file
  • link still showed up in the link inspector but no longer in the outgoing references
  • it is of course still present in the RTF, but clicking on it elicits only a feeble beep.

So it seems that the UI reports things differently than the scripting interface does: A broken item link is not present in the list of outgoing references. Which makes me think that your AI actually reads the RTF (but is too dumb to write to it … well) instead of using a script.

Now, if you’re working with MD, it’s a trivial task to find broken links and append them to the file (perhaps mark them as broken or whatever). Here’s a sample script written in JavaScript. It works only with a single record, but it’s trivial to change it so that it handles the currently selected records.

The broken links are appended to the end after a 2nd level heading. They are included in a code block which makes them visible as broken in the preview.

(() => {
  const app = Application("DEVONthink");
  const r = app.getRecordWithUuid('x-devonthink-item://BAEEFFA8-36EF-49F0-BCDB-F9B157EF5591');
  const txt = r.plainText();
  const itemLinks = [... txt.matchAll(/(\[.*?\])\((x-devonthink-item:.*?)\)/g)];
  const brokenLinks = [];
  itemLinks.forEach(link => {
    const record = app.getRecordWithUuid(link[2]);
	  if (!record instanceof ObjectSpecifier) { /* broken link, record does not exist */
	    brokenLinks.push(link[0]); /* save the complete MD link */
	  }
  })
  r.plainText = txt + `\n\n## BROKEN LINKS:\n\n\`${brokenLinks.join('\n')}\``;
})()

Hey thanks @chrillek !!

I will run the js when back from lunch :slight_smile:

The broken links are for items that probably got moved around so much (some from databases decades old) that now they’re orphaned, and the references don’t even point to anything in any existing database. So my thought was to try and find something that could be related, in the current databases.

So, “Agua Fresca” could have been an rtf file, the link won’t point to anything. But there might be an “Making Agua Fresca.pdf” I created 12 years later - and that is what I wanted to know. Of course, just searching does that, but I wanted to gather the waht DT found and add to the bottom of that rtf file.

So DT is searching text, really. But when I tried to append the search results, the fact that I was trying to append to rtf files was a big issue. So, markdown.