I have started to put together a database of notes in the spirit of the Zettelkasten (ZK). Backlinks (also referred to as incoming links) are a feature that lends itself particularly to digital versions of the ZK. For various reasons, I am not keen on yet another dedicated piece of software for the ZK, and online options such as Roam or Remnote did not convince me either (export of your information is a massive concern, but they also look insanely busy for a serene ZK, with all those hovering pop-ups). Hence, I started implementing the ZK in trusted DT using Markdown as the note format. At this point, DT does not do backlinks. Recently, Bernardo_V (forum thread) and JacobIO (forum thread) posted very nice solutions. However, for my needs, I want something slightly different. Here are my requirements:

• Backlinks must work in Devonthink To Go.
• Backlinks must be immune to future name changes of files.
• Web export of the DB must fully maintain the link structure ( ➝ 100% safe exit strategy, as generic html will always work).

This implies:

• Wikilinks are out (don’t work in DTTG and break when filenames are changed).
• Item links are the way to go, but finding them must be based on UUID, not filenames.

Bernardo_V’s scripts almost fit the bill, as they can handle item links. However, they use the name to match the link. In principle, it would be easy to match the UUID instead of the name. Unfortunately, DT these days searches the rendered markdown content, not the source (see e.g. this forum thread)

My solution: copy the markdown source of all markdown files in the ZK DB as plain text into a custom metadata field of type “multi-line text” (but exclude the potentially already existing “Backlinks” section, to avoid going in circles). Then, search this field for links based on their UUID.

When should one act with such a script? One could attach it to markdown files such that when a file is opened in DT, the backlinks are updated on the fly. However, this will not work for someone like me who reads the notes mostly in DTTG. Hence I decided to do bulk backlink updates periodically (say once a day and whenever necessary).

The attached script takes a list of selected markdown files (but does not check the filetype!), and first copies for all of them the source md into the custom metadata field. Then it again loops through the list and searches the custom field for backlinks, adopting one of Bernardo_V’s scripts, but searching for UUID instead.

That’s a lot of searching: for 96 files, this takes 29 seconds on my lowly 2016 1.2 GHz MacBook. Naively, I’d expect this to scale as N^2 for N markdown files in the DB: N searches and in each search N items to search. However, in a first, imperfect, test, I found a largely linear behaviour: 192 files took 54 secs, 384 took 123 secs (I only have a few, independent markdown files in this DB, so I duplicated them multiple times over, which, however, creates a large number of almost identical files, but on the other hand I get massive levels of backlinking, which is also unrealistic).

Seems that for a large ZK of several thousand notes, this can still easily finish over lunch or overnight.

Below is the code. Please note that I have only fleeting familiarity with Applescript. I also can’t figure out how to make the code appear with colour highlighting (update: a fairy swooped in and inserted the triple backquotes).

(*
to the end of the document, making the backlinks available in DevonThink to Go,
and in a DT web export of the note collection.

GG, 2020.09.14

Credits:
Based on an early version of a script provided by Bernardo_V, who deservers all
the credit for the difficult stuff
This version of Bernardo_V's base script was distributed with the ebook
"Taking Smart Notes with DevonThink" by Kourosh Dini
( https://www.kouroshdini.com/course-books/ ).

What's different: Unlike the other scripts, this one identifies item links by
their UUID, NOT THE FILENAME. This is the only way to protect links
from future file name changes. To me this is a must-have feature (ruling
wikilinks effectively out, which also won't work in DTTG).

The problem: UUID, while present in the .md file in the link, CANNOT
be searched in DevonThink, because it only indexes the RENDERED
markdown content. See

The solution: Prior to searching for backlinks, this script copies for all
selected files the markdown SOURCE into a custom metadata field
"GGtext", which subsequently, the is searched for links based on UUID.

the custom metadata field, otherwise they would be interpreted as

Usage and practical considerations:
- This script is run periodically to update all backlinks in bulk
- technically, one could use a trigger that runs this script for a single
file when it is opened in DT, so you get JIT backlinks updates whenever
you look at a file. However, no such mechanism is avaible in DTTG,
which I mostly use for looking at my notes. In that case periodic bulk
backlink maintenance is the way to go.

- this script has to loop twice over all selected markdown notes and carry
out DT searches for each file. It might get slow with a large number of notes.

I'm testing with 96 markdown files and it seems to take around 28 seconds
on my lowly 2016 1.2 GHz MacBook. Naively, 1000 markdown files might
then take around 290 seconds. Part 1 is much faster. The repeated searches
in part 2 slow things down.

For large markdown note collections, run it over lunch break or start it
last thing in the evening.

1. Select all markdown files you want to include in the backlinking process
(in the simplest case, make a smart folder that lists all markdown files in the
DB, and then select them all).

- note: the script does not check explicitly whether all selected files are
indeed markdown (could be added, but I'm not an Applescript whiz, and have
to move on to other stuff now). I have not checked yet what it does to other file
types. Proceed at your own risk.

2. Run the script, wait, and done. Hopefully there are no major bugs that
eluded me.

3. Once DB is synced, backlinks will be available in DTTG.

*)

use AppleScript version "2.4" -- Yosemite (10.10) or later
use script "RegexAndStuffLib"

property MainUUID : ""
property theDelimiter : "###### Backlinks" -- Delimiter of choice. e.g. # Backlinks
property PlainText : "GGtext" -- name of the custom metadata field for the .md source

-- ===================================================
-- PART 1: for the selected list of .md files, copy the markdown source into the

tell application id "DNtp"
set theSelection to the selection
repeat with theRecord in theSelection

-- uncomment one of the following

-- copy the document's UUID into the comment
--	set the comment of theRecord to uuid of theRecord

-- copy the plain text of a markdown document into the comment
set theText to plain text of theRecord
-- return theText

try
set oldDelims to AppleScript's text item delimiters
set AppleScript's text item delimiters to theDelimiter
set delimitedList to every text item of theText
set AppleScript's text item delimiters to oldDelims
on error
set AppleScript's text item delimiters to oldDelims
end try

-- return delimitedList

try
set theTruncatedText to item 1 of delimitedList
end try
-- return theTruncatedText

-- choose whether to write the plain text to the comment or a custom field
-- set the comment of theRecord to theTruncatedText
add custom meta data theTruncatedText for PlainText to theRecord

end repeat

end tell

-- ===================================================
-- PART 2: Go through the list of selected .md files again, and search for links,
-- based on UUID information in the custom metadata field, and append an
-- updated backlinks section to the .md file.

tell application id "DNtp"

set theSelection to the selection
repeat with theRecord in theSelection

-- set theRecord to (content record of think window 1)
set theText to plain text of theRecord
set theName to name of theRecord
set theUUID to uuid of theRecord

-- return theName
-- return theText
-- return theUUID

-- Prepare the search string
-- set theSearchList to "\"" & theName & "\""          -- search for the name of the selected document
set theSearchList to "\"" & theUUID & "\"" -- search for the UUID of the selected document

-- return theSearchList

-- Perform the search
set theGroup to get record with uuid MainUUID
-- set theRecords to search "content:" & theSearchList & space & "kind:markdown" in theGroup      -- search the contents
-- set theRecords to search "comment:" & theSearchList & space & "kind:markdown" in theGroup    -- search the comments
-- Note: if custom metadata field has name "custom", then the search addresses it with "mdcustom"
set theRecords to search "mdggtext:" & theSearchList & space & "kind:markdown" in theGroup -- search custom metadata

-- return theRecords

set theList to {}
repeat with each in theRecords

set the end of theList to "[" & name of each & "]" & "(" & reference URL of each & ")" & "&nbsp;&nbsp; ⚫︎ &nbsp;&nbsp;"
--set the end of theList to return & "* [" & name of each & "]" & "(" & reference URL of each & ")"

end repeat
set theList to my sortlist(theList)

try
set oldDelims to AppleScript's text item delimiters -- salvar o delimitador padrão
set AppleScript's text item delimiters to theDelimiter
set delimitedList to every text item of theText
on error
set AppleScript's text item delimiters to oldDelims -- restaurar delimitador padrão em caso de erro
end try

try
set theText to item 1 of delimitedList
set theText to my trimtext(theText, "", "end")

set the plain text of theRecord to theText & theDelimiter & return & theList as text
end try

end repeat
end tell

-- END OF MAIN PROGRAM

-- Handlers section

on replaceText(theString, old, new)
set {TID, text item delimiters} to {text item delimiters, old}
set theStringItems to text items of theString
set text item delimiters to new
set theString to theStringItems as text
set text item delimiters to TID
return theString
end replaceText

on trimtext(theText, theCharactersToTrim, theTrimDirection)
set theTrimLength to length of theCharactersToTrim
if theTrimDirection is in {"beginning", "both"} then
repeat while theText begins with theCharactersToTrim
try
set theText to characters (theTrimLength + 1) thru -1 of theText as string
on error
-- text contains nothing but trim characters
return ""
end try
end repeat
end if
if theTrimDirection is in {"end", "both"} then
repeat while theText ends with theCharactersToTrim
try
set theText to characters 1 thru -(theTrimLength + 1) of theText as string
on error
-- text contains nothing but trim characters
return ""
end try
end repeat
end if
return theText
end trimtext

on sortlist(theList)
set theIndexList to {}
set theSortedList to {}
repeat (length of theList) times
set theLowItem to ""
repeat with a from 1 to (length of theList)
if a is not in theIndexList then
set theCurrentItem to item a of theList as text
if theLowItem is "" then
set theLowItem to theCurrentItem
set theLowItemIndex to a
else if theCurrentItem comes before theLowItem then
set theLowItem to theCurrentItem
set theLowItemIndex to a
end if
end if
end repeat
set end of theSortedList to theLowItem
set end of theIndexList to theLowItemIndex
end repeat
return theSortedList
end sortlist


I’d use a try … on error … end try block around the repeat ... end repeat loops, not inside it. AppleScript is not very fast, and you’re slowing it down more than necessary by repeatedly setting the text delimiter and resetting it again. This can be done once outside the repeat, I think. In the same vein, I don’t see why you’d put a set operation in a try block without a corresponding error handler. Either the set can fail, than you should handle the error. Or it will always succeed, than there’s not point to the try.

Instead of popping up windows with display notification, you could use log message for a less intrusive way to follow progress. The messages will then appear in DT’s log window.

I’m not sure why you’d use HTML entities (&nbsp; in a non-HTML file)

This is the second time MainUUID is used in the script. The first time is at the top, where you set it to “”. To me, it looks as if you’re now searching for a record with uuid “”. Is that intended or do I overlook something?

Personally, I find comments stating the obvious (“return theName” and the like) more irritating than helpful. They make the script a lot longer than necessary and it’s very difficult to weed out the useless comments.

Since you call the trimtext handler only with “end”, you could remove theTrimDirection from it as well as the first ifblock. From the 2nd ifblock, you need only the repeat loop (no if, no end if). All this is a matter of taste, but shorter code is easier to understand and to maintain. And in this case, it should also make things a tiny bit faster.

Are you sure that you need the links sorted as per set theList to my sortlist(theList)? If not, you might just get rid of this line. Which might also make things run faster.

This one initially needs 90 seconds for 1000(!) records.

That’s possible by only writing text and/or meta data if necessary.

Same here.

After an initial run it takes 25 seconds. However as it’s unlikely that one changes hundreds of records without running the script again I think it should take under 30 seconds to process 1000 records. This could make it possible to use it in a Smart Rule.

If you or others test my script after your script was used mine needs a lot longer on the first run as it needs to reset what your script has done. After that a second run shows the speed difference.

EDIT: removed a stupid script

Thanks for a script that looks much more cleaned up, @pete31! I have no time to check it out in detail right now, but a few questions/observations.

If you or others test my script after your script was used mine needs a lot longer on the first run as it needs to reset what your script has done.

I ran your script. I’m confused. It puts the proper backlinks in if there is no pre-existing backlink section. If files have a legacy backlinks section from running my script, yours seems to not do anything. That’s counter what you state about the “initial run”.

This one initially needs 90 seconds for 1000(!) records.

On what machine? My benchmark was with probably the slowest, non-vintage Mac there is.

That’s possible by only writing text and/or meta data if necessary.

Two remarks:
(1) For a Zettelkasten DB, pretty much every note contains links.
(2) Running your script on my machine, I find its speed comparable to mine. I’m not really surprised, because part 1 in my script, the writing of the metadata, takes less than 10% of the overall run time. So not writing some of the records won’t help much. Part 2, searching all md files in the DB for item link matches and assembling the new backlink section, is a comparable effort in our scripts, apart from coding efficiencies (which still could make a difference).

Thanks for the feedback, @chrillek. Indeed, this script is not optimized. I took a script by Bernardo_V that took one file and added backlinks to it, and simply slapped a repeat loop around it to work down a (potentially large) list of files. So things that for a single file update are completely irrelevant in terms of speed suddenly can become important.

Are you sure that you need the links sorted as per set theList to my sortlist(theList)? If not, you might just get rid of this line. Which might also make things run faster.

Easy to check! Just one line to comment out. I measure no improvement at all, despite having a few files with 20 or more backlinks that would require heavy sorting. On top of that, yes, I am sure that I want the links sorted!

I’d use a try … on error … end try block around the repeat ... end repeat loops

I commented out all the try instances and see not change in speed. To be honest, I’d be surprised if they made such a big difference, in code that undertakes such heavy operations such as calling a DT search in a loop. And note that some do have a useful function: For example, if a folder is included in the selection, the try...on error will fix that in part 1, without it the script will fail. For this to work, the try has to be inside the loop.

I don’t see why you’d put a set operation in a try block without a corresponding error handler

Yes, oversight during copy/paste.

Instead of popping up windows with display notification , you could use log message for a less intrusive way to follow progress. The messages will then appear in DT’s log window.

Huh? Why not? Let this script run in DT, switch to a different virtual desktop to do something else, and get a nice system notification. Why should I have to go back to the DT log window? Comment it out, if you don’t like it.

I’m not sure why you’d use HTML entities ( &nbsp; in a non-HTML file)

To get rendered extra whitespace into Markdown source? If you use " | " (two whitespaces on each side of the vertical bar), Markdown and HTML will simply ignore anything beyond a single whitespace (because they are typesetters, not typewriters). Futhermore, HTML is entirely legal content of a Markdown file. How else would you for example enable MathJax rendering on an individual file by file level? Having said this, I now changed my script to put each link on a new line. I think I like that better.

Personally, I find comments stating the obvious (“return theName” and the like) more irritating than helpful. They make the script a lot longer than necessary and it’s very difficult to weed out the useless comments.

I should have removed those before posting. Those are not comments, but commented-out code; poor-man’s debugging, so to speak. I apologize for irritating readers

@cgrunenberg indicated that a future update of DT will included a metadata field that contains backlinks. While this will likely still not work for DTTG users, it will be easy to write a script that copies the links from that field to a backlinks section in a markdown file. This should be much faster than the script presented here, which will then only be a stopgap measure until said feature arrives in DT.

Below is a somewhat streamlined version of the script, with no operational changes.

(*
to the end of the document, making the backlinks available in DevonThink to Go,
and in a DT web export of the note collection.

GG, 2020.09.14, updated 2020.09.15

Credits:
Based on an early version of a script provided by Bernardo_V, who deservers all
the credit for the difficult stuff
This version of Bernardo_V's base script was distributed with the ebook "Taking
Smart Notes with DevonThink" by Kourosh Dini ( https://www.kouroshdini.com/course-books/ ).

What's different: Unlike the other scripts, this one identifies item links by their
UUID, NOT THE FILENAME. This is the only way to protect links from future file
name changes. To me this is a must-have feature (ruling wikilinks effectively out,
which also won't work in DTTG).

The problem: UUID, while present in the .md file in the link, CANNOT be searched
in DevonThink, because it only indexes the RENDERED markdown content. See

The solution: Prior to searching for backlinks, this script copies for all selected
files the markdown SOURCE into a custom metadata field "GGtext", which
subsequently, the is searched for links based on UUID.

Slight complication: already existing backlinks must not be copied into the custom
metadata field, otherwise they would be interpreted as forward links. Hence, the
"backlinks" section is excluded in part 1.

Usage and practical considerations:
- This script is run periodically to update all backlinks in bulk
- technically, one could use a trigger that runs this script for a single file when
it is opened in DT, so you get JIT backlinks updates whenever you look at a file.
However, no such mechanism is avaible in DTTG, which I mostly use for looking
at my notes. In that case periodic bulk backlink maintenance is the way to go.

- this script has to loop twice over all selected markdown notes and carry out DT
searches for each file. It might get slow with a large number of notes.

I'm testing with 96 markdown files and it seems to take around 28 seconds
on my lowly 2016 1.2 GHz MacBook. Naively, 1000 markdown files might
then take around 290 seconds. Part 1 is much faster. The repeated searches
in part 2 slow things down.

For large markdown note collections, run it over lunch break or start it last
thing in the evening.

1. Select all markdown files you want to include in the backlinking process (in the simplest
case, make a smart folder that lists all markdown files in the DB, and then select them all).
- note: the script does not check explicitly whether all selected files are indeed
markdown (could be added, but I'm not an Applescript whiz, and have to move
on to other stuff now). I have not checked yet what it does to other file types.

2. Run the script, wait, and done. Hopefully there are no major bugs that eluded me.

3. Once DB is synced, backlinks will be available in DTTG.

*)

use AppleScript version "2.4" -- Yosemite (10.10) or later

property MainUUID : "" -- Not sure what this means
property theDelimiter : "###### Backlinks" -- Delimiter of choice. e.g. # Backlinks
property PlainText : "GGtext" -- name of the custom metadata field for the .md source

-- ===================================================
-- PART 1: for the selected list of .md files, copy the markdown source into the custom metadata field "GGtext", but omit any existing "backlinks" section

tell application id "DNtp"

set oldDelims to AppleScript's text item delimiters
set AppleScript's text item delimiters to theDelimiter

set theSelection to the selection

repeat with theRecord in theSelection
set theText to plain text of theRecord -- copy the plain text of a markdown document into a custom metadata field
set delimitedList to every text item of theText
set theTruncatedText to item 1 of delimitedList
add custom meta data theTruncatedText for PlainText to theRecord -- add markdown source into the custom metadata field
end repeat

set AppleScript's text item delimiters to oldDelims

end tell

-- ===================================================
-- PART 2: Go through the list of selected .md files again, and search for links, based on UUID information in the custom metadata field, and append an updated backlinks section to the .md file.

tell application id "DNtp"

set theSelection to the selection
repeat with theRecord in theSelection

-- set theRecord to (content record of think window 1)
set theText to plain text of theRecord
set theName to name of theRecord
set theUUID to uuid of theRecord

set theSearchList to "\"" & theUUID & "\"" -- prepare the search string with UUID
set theGroup to get record with uuid MainUUID -- don't know what MainUUID means
set theRecords to search "mdggtext:" & theSearchList & space & "kind:markdown" in theGroup -- Note: if custom metadata field has name "custom", then the search addresses it with "mdcustom"

set theList to {}
repeat with each in theRecords
-- Put each backlink of a new line
set the end of theList to "[" & name of each & "]" & "(" & reference URL of each & ")" & "<br>"
end repeat
set theList to my sortlist(theList)

try
set oldDelims to AppleScript's text item delimiters
set AppleScript's text item delimiters to theDelimiter
set delimitedList to every text item of theText
set AppleScript's text item delimiters to oldDelims
on error
-- set AppleScript's text item delimiters to oldDelims
end try

try
set theText to item 1 of delimitedList
set the plain text of theRecord to theText & theDelimiter & return & theList as text
end try

end repeat
end tell

-- END OF MAIN PROGRAM

on sortlist(theList)
set theIndexList to {}
set theSortedList to {}
repeat (length of theList) times
set theLowItem to ""
repeat with a from 1 to (length of theList)
if a is not in theIndexList then
set theCurrentItem to item a of theList as text
if theLowItem is "" then
set theLowItem to theCurrentItem
set theLowItemIndex to a
else if theCurrentItem comes before theLowItem then
set theLowItem to theCurrentItem
set theLowItemIndex to a
end if
end if
end repeat
set end of theSortedList to theLowItem
set end of theIndexList to theLowItemIndex
end repeat
return theSortedList
end sortlist


Just a short explanation concerning the try blocks: I agree that the speed is probably not an issue here.

set oldDelims to AppleScript's text item delimiters
set AppleScript's text item delimiters to theDelimiter
repeat with theRecord in theSelection
try
set theText to plain text of theRecord
set delimitedList to every text item of theText
set theTruncatedText to item 1 of delimitedList
add custom meta data theTruncatedText for PlainText to theRecord
end try
end repeat
set AppleScript's text item delimiters to oldDelims


So basically, I’d move the things that have no relation to the repeat loop (i.e. setting the delimiters) outside of it. This is mainly a question of style, unless you are processing huge amounts of data in a loop. My original comment was wrong. There’s probably no error handler needed here because there’s nothing it can do anyway, and you want to go over all the true records in the selection anyway.

BTW: In the new version of the script, you have set theTruncatedText to item 1 of delimitedList right after the end try. If, as you said, the try block is useful to catch erroneously selected folders et al, the set part should be inside the try block, I think. As well as the next line (add custom metadata…). Otherwise, you’d set the custom meta data of a folder to theTruncatedText of the last successfully processed record.

As to the rest, I just wanted to make some suggestions (dialog vs. log message) and get some clarifications (  in markdown). I still don’t see how get record with uuid MainUUID works when MainUUID is set to “” and never changed, but if it works…

@chrillek, you are of course right about the delimiter swapping in part 1. In part 2 there is a similar swapping of delimiters, but there I think it is necessary, as some operations with the standard delimiter are carried out in between. I’ve updated the script in my reply two above (I don’t think I can edit the original post, now that replies have come in).

I’ve looked a bit more at the code, and also noted that the handlers replaceText() and trimtext() are not really used. The former is not called at all, and the latter is called, but with the request to trim "", i.e. nothing.

Those are pitfalls of grabbing someone else’s script, and fooling around with it without really studying it. I never even noted that replaceText is not called. All I saw is that the existing code would do what I want for a single file, and ergo, all I have to do is put it in a loop. I suspect Bernardo_V needed those functions in his more general scripts. I also don’t know what property MainUUID : "" does, but it’s clearly needed. For now, I can live with being ignorant about that.

With all the cruft removed, the script still runs roughly at the same speed. I suspect that the interactions with DT take their time. Naturally, I suspected the DT searches to be the biggest resource hog. I tested this by replacing the actual search with a hard-coded “search result”, and surprisingly, that did not make much of a difference. So the DT search must be fast compared to other (scripting) holdups.

Anyway, I think I’m good with this for now. This script does what it needs to do. It will take me a while to (hopefully) accumulate enough notes in this DB to make speed an issue. If that time ever comes, I’ll look into this again.

Sorry, I don’t know what I did there. Testing without helper scripts was not a good idea …

Here’s a helper script that removes the backlink section and the meta data. After running it the new version below should work.

Helper scrpt: Remove Backlinks Section and Meta Data
-- Helper script: Remove Backlinks Section and Meta Data

property customMetadataName : "ggtext" -- name of the custom metadata field for the .md source. Make sure to use lowercase

tell application id "DNtp"
try
set theSelection to selection of viewer window 1
if theSelection = {} then error "Nothing selected"

repeat with thisRecord in theSelection
set theText to plain text of thisRecord
if theText contains theDelimiter then set plain text of thisRecord to my getTruncatedText(theText)
if (get custom meta data for customMetadataName from thisRecord) ≠ missing value then add custom meta data "" for customMetadataName to thisRecord
end repeat

on error error_message number error_number
if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
end try
end tell

on getTruncatedText(theText)
set oldDelims to AppleScript's text item delimiters
set AppleScript's text item delimiters to theDelimiter
set delimitedList to every text item of theText
set AppleScript's text item delimiters to oldDelims
set theTruncatedText_trimmed to my trimEnd(item 1 of delimitedList)
end getTruncatedText

on trimEnd(str)
local str, whiteSpace
try
set str to str as string
set whiteSpace to {character id 10, return, space, tab}
try
repeat while str's last character is in whiteSpace
set str to str's text 1 thru -2
end repeat
return str
on error number -1728
return ""
end try
on error eMsg number eNum
error "Can't trimEnd: " & eMsg number eNum
end try
end trimEnd


2012 MacBook Pro, 2,5 GHz

I know. I fixed my stupid script and did some tests.

Tested with 100 records, each containing links to the 99 others.

Initial run:

Run again without changing records:

Run after removig one different link in 10 records

You don’t exclude unnecessary interactions. That’s what I mean with “only writing text and/or meta data if necessary”.

Look at the AppleEvents used (the number at the right corner). See how your script sends 20606 on each run (don’t know why it’s 20586 in the last capture)? Then see the difference on each run of my script.

For creating test records I used scripts I already had, if you want I’ll post them.

Here’s a new version

-- Append Reference URL backlinks in markdown text (via custom meta data)
-- Idea: @gg378
-- Improvements: @pete31

property theGroupUUID : "" -- optional search scope
property theDelimiter : "###### Backlinks" -- Delimiter of choice. e.g. # Backlinks
property customMetadataName : "ggtext" -- name of the custom metadata field for the .md source. Make sure to use lowercase

tell application id "DNtp"
try
set theGroup to get record with uuid theGroupUUID
set theSelection to selection of viewer window 1
if theSelection = {} then error "Nothing selected."

repeat with thisRecord in theSelection
set theText to plain text of thisRecord
if theText ≠ "" then
set theContent to item 1 of (my tid(theText, theDelimiter))
if theContent contains "x-devonthink-item://" then
else
if (get custom meta data for customMetadataName from thisRecord) ≠ missing value then add custom meta data "" for customMetadataName to thisRecord
end if
end if
end repeat

repeat with thisRecord in theSelection
set theText to plain text of thisRecord
set theResults to search "md" & customMetadataName & ":\"" & (uuid of thisRecord) & "\" kind:markdown" in theGroup

if theResults ≠ {} then
if theText contains theDelimiter then
set theBackLinksText to item 2 of (my tid(theText, theDelimiter))

if (count theResults) = ((count my tid(theBackLinksText, "x-devonthink-item://")) - 1) then
repeat with thisResult in theResults
if (reference URL of thisResult) is not in theBackLinksText then
exit repeat
end if
end repeat
else
end if

else
end if
else
if theText contains theDelimiter then set plain text of thisRecord to my trimEnd(item 1 of (my tid(theText, theDelimiter)))
end if
end repeat

on error error_message number error_number
if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
end try
end tell

tell application id "DNtp"
set theList to {}
repeat with thisResult in theResults
set the end of theList to "[" & (name of thisResult) & "](" & (reference URL of thisResult) & ")" & "&nbsp;&nbsp; ⚫︎ &nbsp;&nbsp;"
end repeat
set theList to my sort_list(theList)
if theText ≠ "" then
set plain text of theRecord to my trimEnd(item 1 of (my tid(theText, theDelimiter))) & linefeed & linefeed & theDelimiter & linefeed & theList as text
else
set plain text of theRecord to linefeed & linefeed & theDelimiter & linefeed & theList as text
end if
end tell

on tid(theText, theDelimiter)
set d to AppleScript's text item delimiters
set AppleScript's text item delimiters to theDelimiter
set theTextItems to text items of theText
set AppleScript's text item delimiters to d
return theTextItems
end tid

on trimEnd(str)
local str, whiteSpace
try
set str to str as string
set whiteSpace to {character id 10, return, space, tab}
try
repeat while str's last character is in whiteSpace
set str to str's text 1 thru -2
end repeat
return str
on error number -1728
return ""
end try
on error eMsg number eNum
error "Can't trimEnd: " & eMsg number eNum
end try
end trimEnd

on sort_list(theList)
considering numeric strings
set theIndexList to {}
set theSortedList to {}
repeat (length of theList) times
set theLowItem to ""
repeat with a from 1 to (length of theList)
if a is not in theIndexList then
set theCurrentItem to item a of theList as text
if theLowItem is "" then
set theLowItem to theCurrentItem
set theLowItemIndex to a
else if theCurrentItem comes before theLowItem then
set theLowItem to theCurrentItem
set theLowItemIndex to a
end if
end if
end repeat
set end of theSortedList to theLowItem
set end of theIndexList to theLowItemIndex
end repeat
end considering
return theSortedList
end sort_list

See how your script sends 20606 on each run (don’t know why it’s 20586 in the last capture)?

Presumably the upper part of the timing/events graphs shows my script? (Btw, what software do you use for that?)

I ran your scripts over my test files. Indeed, it is significantly faster on subsequent runs. For some reason, I interpreted your first post such that the “unnecessary interactions” mean “not copying md source to the custom field if there is no item link in there”, which would not work in my test, as all my files contain links.

Looking at your new script, I now realize that the speedup comes from testing which backlink sections actually need to be written back into the md file. Nice! I would not have guessed that, because writing to the DB doesn’t seem to be a problem when I copy all md-sources to the custom field in part 1 of my script. If that were true, my part 2 should talk roughly as long as my part 1, not 10x longer. So I don’t think I really understand what’s going on in terms of the timing.

Let me see how you go about deciding whether the backlink section (BLS) needs an update. You check:

1. Are there any search results? If not -> skip updating, and if necessary, delete old BLS
2. if yes, is there already a BLS? If no -> make full BLS update
3. if yes, need to check whether old and new are identical
4. if there are fewer search result than existing BLS -> definitely need update
5. if equal or more new results, check in detail

If that’s correct, I wonder about 5.

if ( count theResults) ≥ theBacklinkCount_old then

It’s not wrong, but why lump those cases together with ≥? If >, then an update is definitely needed. Only the case = requires the careful link-by-link check. Do I get that right? Of course, with ≥ it will still work, and in the case of seldom link updates, be still almost as efficient. Wouldn’t one naturally lump < and > into one “definite update” category and then proceed to the = case? Just wondering.

The speedup provided by this version would be very welcome (and your 2012 MBP might be slower than my 2016 MB, after all, my 2010 top-of-the-line 17" MBP certainly was!)

Yes, the upper is yours. That’s Script Debugger

Good catch! Changed it in the script above. Thanks!

That’s for setting the search scope. I wondered too that it’s no problem to tell DEVONthink to search in a group that wasn’t defined but it obviously works.

BTW You could use this group too to get the records automatically without the need of selecting them each time. Adding this line

set theSelection to children of theGroup

and commenting out the two lines that currently get the selection should do it. However if your records are nested into subgroups you’ll need a recursive handler instead.

Tested with much more realistic data:

500 records, each with a random link count ranging from 1 to 10.

Initial run:

Run again without changing records:

Run after removing one different link in 10 records

That’s fast!

You can even use search ... in missing value without “getting” the non-existing group first. It returns results from exactly one database (no idea how it decides on this database). If you leave out the “in …” part, it returns results from all databases.

I’m wondering if that’s a feature or something else.

It won’t be a (custom) metadata field but an extension of the Document > Links inspector instead. And of course it will be scriptable:

tell application id "DNtp"
repeat with theRecord in (selection as list)
repeat with theReference in incoming references of theRecord
set theURL to reference URL of theReference
...
end repeat
end repeat
end tell

A property for outgoing references will be available too of course.

3 Likes

For the record: @cgrunenberg said that this is a bug: What does search in a non-existing group do?
using an invalid parameter for in should give the same results as no in parameter at all.

1 Like