Backlinks (aka Incoming Links) revisited: DTTG

I have started to put together a database of notes in the spirit of the Zettelkasten (ZK). Backlinks (also referred to as incoming links) are a feature that lends itself particularly to digital versions of the ZK. For various reasons, I am not keen on yet another dedicated piece of software for the ZK, and online options such as Roam or Remnote did not convince me either (export of your information is a massive concern, but they also look insanely busy for a serene ZK, with all those hovering pop-ups). Hence, I started implementing the ZK in trusted DT using Markdown as the note format. At this point, DT does not do backlinks. Recently, Bernardo_V (forum thread) and JacobIO (forum thread) posted very nice solutions. However, for my needs, I want something slightly different. Here are my requirements:

  • Backlinks must work in Devonthink To Go.
  • Backlinks must be immune to future name changes of files.
  • Web export of the DB must fully maintain the link structure ( ➝ 100% safe exit strategy, as generic html will always work).

This implies:

  • Wikilinks are out (don’t work in DTTG and break when filenames are changed).
  • Item links are the way to go, but finding them must be based on UUID, not filenames.

Bernardo_V’s scripts almost fit the bill, as they can handle item links. However, they use the name to match the link. In principle, it would be easy to match the UUID instead of the name. Unfortunately, DT these days searches the rendered markdown content, not the source (see e.g. this forum thread)

My solution: copy the markdown source of all markdown files in the ZK DB as plain text into a custom metadata field of type “multi-line text” (but exclude the potentially already existing “Backlinks” section, to avoid going in circles). Then, search this field for links based on their UUID.

When should one act with such a script? One could attach it to markdown files such that when a file is opened in DT, the backlinks are updated on the fly. However, this will not work for someone like me who reads the notes mostly in DTTG. Hence I decided to do bulk backlink updates periodically (say once a day and whenever necessary).

The attached script takes a list of selected markdown files (but does not check the filetype!), and first copies for all of them the source md into the custom metadata field. Then it again loops through the list and searches the custom field for backlinks, adopting one of Bernardo_V’s scripts, but searching for UUID instead.

That’s a lot of searching: for 96 files, this takes 29 seconds on my lowly 2016 1.2 GHz MacBook. Naively, I’d expect this to scale as N^2 for N markdown files in the DB: N searches and in each search N items to search. However, in a first, imperfect, test, I found a largely linear behaviour: 192 files took 54 secs, 384 took 123 secs (I only have a few, independent markdown files in this DB, so I duplicated them multiple times over, which, however, creates a large number of almost identical files, but on the other hand I get massive levels of backlinking, which is also unrealistic).

Seems that for a large ZK of several thousand notes, this can still easily finish over lunch or overnight.

Below is the code. Please note that I have only fleeting familiarity with Applescript. I also can’t figure out how to make the code appear with colour highlighting (update: a fairy swooped in and inserted the triple backquotes).

(*
Purpose: Add UUID-based backlinks to a markdown note by appending them 
to the end of the document, making the backlinks available in DevonThink to Go, 
and in a DT web export of the note collection.
         
GG, 2020.09.14

Credits: 
Based on an early version of a script provided by Bernardo_V, who deservers all 
the credit for the difficult stuff 
(see https://discourse.devontechnologies.com/t/return-links-back-links/54390 ). 
This version of Bernardo_V's base script was distributed with the ebook 
"Taking Smart Notes with DevonThink" by Kourosh Dini 
( https://www.kouroshdini.com/course-books/ ).

What's different: Unlike the other scripts, this one identifies item links by 
their UUID, NOT THE FILENAME. This is the only way to protect links 
from future file name changes. To me this is a must-have feature (ruling 
wikilinks effectively out, which also won't work in DTTG).

The problem: UUID, while present in the .md file in the link, CANNOT 
be searched in DevonThink, because it only indexes the RENDERED 
markdown content. See 
https://discourse.devontechnologies.com/t/search-for-x-devonthink-item-link-not-working/57652 .

The solution: Prior to searching for backlinks, this script copies for all 
selected files the markdown SOURCE into a custom metadata field 
"GGtext", which subsequently, the is searched for links based on UUID.

Slight complication: already existing backlinks must not be copied into 
the custom metadata field, otherwise they would be interpreted as 
forward links. Hence, the "backlinks" section is excluded in part 1.

Usage and practical considerations:
- This script is run periodically to update all backlinks in bulk
     - technically, one could use a trigger that runs this script for a single 
file when it is opened in DT, so you get JIT backlinks updates whenever
 you look at a file. However, no such mechanism is avaible in DTTG, 
which I mostly use for looking at my notes. In that case periodic bulk 
backlink maintenance is the way to go.
	  
- this script has to loop twice over all selected markdown notes and carry 
out DT searches for each file. It might get slow with a large number of notes. 
	   
I'm testing with 96 markdown files and it seems to take around 28 seconds 
on my lowly 2016 1.2 GHz MacBook. Naively, 1000 markdown files might 
then take around 290 seconds. Part 1 is much faster. The repeated searches 
in part 2 slow things down.
	
For large markdown note collections, run it over lunch break or start it 
last thing in the evening.
	 
1. Select all markdown files you want to include in the backlinking process 
(in the simplest case, make a smart folder that lists all markdown files in the 
DB, and then select them all).

- note: the script does not check explicitly whether all selected files are 
indeed markdown (could be added, but I'm not an Applescript whiz, and have 
to move on to other stuff now). I have not checked yet what it does to other file 
types. Proceed at your own risk.
	 
2. Run the script, wait, and done. Hopefully there are no major bugs that 
eluded me.

3. Once DB is synced, backlinks will be available in DTTG.

*)


use AppleScript version "2.4" -- Yosemite (10.10) or later
use script "RegexAndStuffLib"
use scripting additions


property MainUUID : ""
property theDelimiter : "###### Backlinks" -- Delimiter of choice. e.g. # Backlinks
property PlainText : "GGtext" -- name of the custom metadata field for the .md source


-- ===================================================
-- PART 1: for the selected list of .md files, copy the markdown source into the 
-- custom metadata field "GGtext", but omit any existing "backlinks" section

tell application id "DNtp"
	set theSelection to the selection
	repeat with theRecord in theSelection
		
		-- uncomment one of the following
		
		-- copy the document's UUID into the comment
		--	set the comment of theRecord to uuid of theRecord 
		
		-- copy the plain text of a markdown document into the comment
		set theText to plain text of theRecord
		-- return theText
		
		-- Remove the backlink part
		try
			set oldDelims to AppleScript's text item delimiters
			set AppleScript's text item delimiters to theDelimiter
			set delimitedList to every text item of theText
			set AppleScript's text item delimiters to oldDelims
		on error
			set AppleScript's text item delimiters to oldDelims
		end try
		
		-- return delimitedList
		
		try
			set theTruncatedText to item 1 of delimitedList
		end try
		-- return theTruncatedText
		
		-- choose whether to write the plain text to the comment or a custom field
		-- set the comment of theRecord to theTruncatedText
		add custom meta data theTruncatedText for PlainText to theRecord
		
		
	end repeat
	
	display notification "Success part 1!"
	
end tell


-- ===================================================
-- PART 2: Go through the list of selected .md files again, and search for links,
-- based on UUID information in the custom metadata field, and append an 
-- updated backlinks section to the .md file.

tell application id "DNtp"
	
	set theSelection to the selection
	repeat with theRecord in theSelection
		
		-- set theRecord to (content record of think window 1)
		set theText to plain text of theRecord
		set theName to name of theRecord
		set theUUID to uuid of theRecord
		
		-- return theName
		-- return theText
		-- return theUUID
		
		-- Prepare the search string
		-- set theSearchList to "\"" & theName & "\""          -- search for the name of the selected document
		set theSearchList to "\"" & theUUID & "\"" -- search for the UUID of the selected document
		
		-- return theSearchList
		
		-- Perform the search
		set theGroup to get record with uuid MainUUID
		-- set theRecords to search "content:" & theSearchList & space & "kind:markdown" in theGroup      -- search the contents
		-- set theRecords to search "comment:" & theSearchList & space & "kind:markdown" in theGroup    -- search the comments
		-- Note: if custom metadata field has name "custom", then the search addresses it with "mdcustom"
		set theRecords to search "mdggtext:" & theSearchList & space & "kind:markdown" in theGroup -- search custom metadata
		
		-- return theRecords
		
		set theList to {}
		repeat with each in theRecords
			
			-- Without AutoWikiLinks, we need the full address
			set the end of theList to "[" & name of each & "]" & "(" & reference URL of each & ")" & "   ⚫︎   "
			--set the end of theList to return & "* [" & name of each & "]" & "(" & reference URL of each & ")"
			
		end repeat
		set theList to my sortlist(theList)
		
		
		-- Remove old Returnlinks section
		try
			set oldDelims to AppleScript's text item delimiters -- salvar o delimitador padrão
			set AppleScript's text item delimiters to theDelimiter
			set delimitedList to every text item of theText
			set AppleScript's text item delimiters to oldDelims -- restaurar delimitador padrão
		on error
			set AppleScript's text item delimiters to oldDelims -- restaurar delimitador padrão em caso de erro
		end try
		
		-- Add new ReturnLinks section
		try
			set theText to item 1 of delimitedList
			set theText to my trimtext(theText, "", "end")
			
			set the plain text of theRecord to theText & theDelimiter & return & theList as text
		end try
		
		
	end repeat
	display notification "Success part 2!"
end tell

-- END OF MAIN PROGRAM

-- Handlers section

on replaceText(theString, old, new)
	set {TID, text item delimiters} to {text item delimiters, old}
	set theStringItems to text items of theString
	set text item delimiters to new
	set theString to theStringItems as text
	set text item delimiters to TID
	return theString
end replaceText

on trimtext(theText, theCharactersToTrim, theTrimDirection)
	set theTrimLength to length of theCharactersToTrim
	if theTrimDirection is in {"beginning", "both"} then
		repeat while theText begins with theCharactersToTrim
			try
				set theText to characters (theTrimLength + 1) thru -1 of theText as string
			on error
				-- text contains nothing but trim characters
				return ""
			end try
		end repeat
	end if
	if theTrimDirection is in {"end", "both"} then
		repeat while theText ends with theCharactersToTrim
			try
				set theText to characters 1 thru -(theTrimLength + 1) of theText as string
			on error
				-- text contains nothing but trim characters
				return ""
			end try
		end repeat
	end if
	return theText
end trimtext

on sortlist(theList)
	set theIndexList to {}
	set theSortedList to {}
	repeat (length of theList) times
		set theLowItem to ""
		repeat with a from 1 to (length of theList)
			if a is not in theIndexList then
				set theCurrentItem to item a of theList as text
				if theLowItem is "" then
					set theLowItem to theCurrentItem
					set theLowItemIndex to a
				else if theCurrentItem comes before theLowItem then
					set theLowItem to theCurrentItem
					set theLowItemIndex to a
				end if
			end if
		end repeat
		set end of theSortedList to theLowItem
		set end of theIndexList to theLowItemIndex
	end repeat
	return theSortedList
end sortlist

Some comments on the script. I’m not an AppleScript aficonado, though.

I’d use a try … on error … end try block around the repeat ... end repeat loops, not inside it. AppleScript is not very fast, and you’re slowing it down more than necessary by repeatedly setting the text delimiter and resetting it again. This can be done once outside the repeat, I think. In the same vein, I don’t see why you’d put a set operation in a try block without a corresponding error handler. Either the set can fail, than you should handle the error. Or it will always succeed, than there’s not point to the try.

Instead of popping up windows with display notification, you could use log message for a less intrusive way to follow progress. The messages will then appear in DT’s log window.

I’m not sure why you’d use HTML entities (  in a non-HTML file)

This is the second time MainUUID is used in the script. The first time is at the top, where you set it to “”. To me, it looks as if you’re now searching for a record with uuid “”. Is that intended or do I overlook something?

Personally, I find comments stating the obvious (“return theName” and the like) more irritating than helpful. They make the script a lot longer than necessary and it’s very difficult to weed out the useless comments.

Since you call the trimtext handler only with “end”, you could remove theTrimDirection from it as well as the first ifblock. From the 2nd ifblock, you need only the repeat loop (no if, no end if). All this is a matter of taste, but shorter code is easier to understand and to maintain. And in this case, it should also make things a tiny bit faster.

Are you sure that you need the links sorted as per set theList to my sortlist(theList)? If not, you might just get rid of this line. Which might also make things run faster.

This one initially needs 90 seconds for 1000(!) records.

That’s possible by only writing text and/or meta data if necessary.

Same here.

After an initial run it takes 25 seconds. However as it’s unlikely that one changes hundreds of records without running the script again I think it should take under 30 seconds to process 1000 records. This could make it possible to use it in a Smart Rule.

If you or others test my script after your script was used mine needs a lot longer on the first run as it needs to reset what your script has done. After that a second run shows the speed difference.

EDIT: removed a stupid script

Thanks for a script that looks much more cleaned up, @pete31! I have no time to check it out in detail right now, but a few questions/observations.

If you or others test my script after your script was used mine needs a lot longer on the first run as it needs to reset what your script has done.

I ran your script. I’m confused. It puts the proper backlinks in if there is no pre-existing backlink section. If files have a legacy backlinks section from running my script, yours seems to not do anything. That’s counter what you state about the “initial run”.

Furthermore: If I start with a clean-slate (i.e. no backlinks) group of files, your script indeed adds the right backlinks on the first run. But if I then add new forward links into documents, and run your script again, the backlink section is not updated accordingly.

This one initially needs 90 seconds for 1000(!) records.

On what machine? My benchmark was with probably the slowest, non-vintage Mac there is.

That’s possible by only writing text and/or meta data if necessary.

Two remarks:
(1) For a Zettelkasten DB, pretty much every note contains links.
(2) Running your script on my machine, I find its speed comparable to mine. I’m not really surprised, because part 1 in my script, the writing of the metadata, takes less than 10% of the overall run time. So not writing some of the records won’t help much. Part 2, searching all md files in the DB for item link matches and assembling the new backlink section, is a comparable effort in our scripts, apart from coding efficiencies (which still could make a difference).

Thanks for the feedback, @chrillek. Indeed, this script is not optimized. I took a script by Bernardo_V that took one file and added backlinks to it, and simply slapped a repeat loop around it to work down a (potentially large) list of files. So things that for a single file update are completely irrelevant in terms of speed suddenly can become important.

Are you sure that you need the links sorted as per set theList to my sortlist(theList)? If not, you might just get rid of this line. Which might also make things run faster.

Easy to check! Just one line to comment out. I measure no improvement at all, despite having a few files with 20 or more backlinks that would require heavy sorting. On top of that, yes, I am sure that I want the links sorted!

I’d use a try … on error … end try block around the repeat ... end repeat loops

I commented out all the try instances and see not change in speed. To be honest, I’d be surprised if they made such a big difference, in code that undertakes such heavy operations such as calling a DT search in a loop. And note that some do have a useful function: For example, if a folder is included in the selection, the try...on error will fix that in part 1, without it the script will fail. For this to work, the try has to be inside the loop.

I don’t see why you’d put a set operation in a try block without a corresponding error handler

Yes, oversight during copy/paste.

The rest of your comments are more about choices and attitudes:

Instead of popping up windows with display notification , you could use log message for a less intrusive way to follow progress. The messages will then appear in DT’s log window.

Huh? Why not? Let this script run in DT, switch to a different virtual desktop to do something else, and get a nice system notification. Why should I have to go back to the DT log window? Comment it out, if you don’t like it.

I’m not sure why you’d use HTML entities (   in a non-HTML file)

To get rendered extra whitespace into Markdown source? If you use " | " (two whitespaces on each side of the vertical bar), Markdown and HTML will simply ignore anything beyond a single whitespace (because they are typesetters, not typewriters). Futhermore, HTML is entirely legal content of a Markdown file. How else would you for example enable MathJax rendering on an individual file by file level? Having said this, I now changed my script to put each link on a new line. I think I like that better.

Personally, I find comments stating the obvious (“return theName” and the like) more irritating than helpful. They make the script a lot longer than necessary and it’s very difficult to weed out the useless comments.

I should have removed those before posting. Those are not comments, but commented-out code; poor-man’s debugging, so to speak. I apologize for irritating readers :slight_smile:

@cgrunenberg indicated that a future update of DT will included a metadata field that contains backlinks. While this will likely still not work for DTTG users, it will be easy to write a script that copies the links from that field to a backlinks section in a markdown file. This should be much faster than the script presented here, which will then only be a stopgap measure until said feature arrives in DT.

Below is a somewhat streamlined version of the script, with no operational changes.

(*
Purpose: Add UUID-based backlinks to a markdown note by appending them 
to the end of the document, making the backlinks available in DevonThink to Go, 
and in a DT web export of the note collection.
         
	GG, 2020.09.14, updated 2020.09.15


Credits: 
Based on an early version of a script provided by Bernardo_V, who deservers all 
the credit for the difficult stuff 
(see https://discourse.devontechnologies.com/t/return-links-back-links/54390 ). 
This version of Bernardo_V's base script was distributed with the ebook "Taking 
Smart Notes with DevonThink" by Kourosh Dini ( https://www.kouroshdini.com/course-books/ ).

What's different: Unlike the other scripts, this one identifies item links by their 
UUID, NOT THE FILENAME. This is the only way to protect links from future file 
name changes. To me this is a must-have feature (ruling wikilinks effectively out, 
which also won't work in DTTG).

The problem: UUID, while present in the .md file in the link, CANNOT be searched 
in DevonThink, because it only indexes the RENDERED markdown content. See 
https://discourse.devontechnologies.com/t/search-for-x-devonthink-item-link-not-working/57652 .

The solution: Prior to searching for backlinks, this script copies for all selected 
files the markdown SOURCE into a custom metadata field "GGtext", which 
subsequently, the is searched for links based on UUID.

Slight complication: already existing backlinks must not be copied into the custom 
metadata field, otherwise they would be interpreted as forward links. Hence, the 
"backlinks" section is excluded in part 1.

Usage and practical considerations:
- This script is run periodically to update all backlinks in bulk
     - technically, one could use a trigger that runs this script for a single file when
	 it is opened in DT, so you get JIT backlinks updates whenever you look at a file. 
	 However, no such mechanism is avaible in DTTG, which I mostly use for looking 
	 at my notes. In that case periodic bulk backlink maintenance is the way to go.
	 
	 - this script has to loop twice over all selected markdown notes and carry out DT 
	  searches for each file. It might get slow with a large number of notes. 
	    
	             I'm testing with 96 markdown files and it seems to take around 28 seconds 
		    on my lowly 2016 1.2 GHz MacBook. Naively, 1000 markdown files might
			 then take around 290 seconds. Part 1 is much faster. The repeated searches 
			 in part 2 slow things down.
	 
	         For large markdown note collections, run it over lunch break or start it last
	     thing in the evening.
	 
1. Select all markdown files you want to include in the backlinking process (in the simplest 
case, make a smart folder that lists all markdown files in the DB, and then select them all).
        - note: the script does not check explicitly whether all selected files are indeed 
		markdown (could be added, but I'm not an Applescript whiz, and have to move 
		on to other stuff now). I have not checked yet what it does to other file types. 
		Proceed at your own risk.
	 
2. Run the script, wait, and done. Hopefully there are no major bugs that eluded me.

3. Once DB is synced, backlinks will be available in DTTG.

*)


use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions

property MainUUID : "" -- Not sure what this means
property theDelimiter : "###### Backlinks" -- Delimiter of choice. e.g. # Backlinks
property PlainText : "GGtext" -- name of the custom metadata field for the .md source


-- ===================================================
-- PART 1: for the selected list of .md files, copy the markdown source into the custom metadata field "GGtext", but omit any existing "backlinks" section

tell application id "DNtp"
	
	set oldDelims to AppleScript's text item delimiters
	set AppleScript's text item delimiters to theDelimiter
	
	set theSelection to the selection
	
	repeat with theRecord in theSelection
		set theText to plain text of theRecord -- copy the plain text of a markdown document into a custom metadata field
		set delimitedList to every text item of theText
		set theTruncatedText to item 1 of delimitedList
		add custom meta data theTruncatedText for PlainText to theRecord -- add markdown source into the custom metadata field
	end repeat
	
	set AppleScript's text item delimiters to oldDelims
	
	display notification "Success part 1!"
	
end tell

-- ===================================================
-- PART 2: Go through the list of selected .md files again, and search for links, based on UUID information in the custom metadata field, and append an updated backlinks section to the .md file.

tell application id "DNtp"
	
	set theSelection to the selection
	repeat with theRecord in theSelection
		
		-- set theRecord to (content record of think window 1)
		set theText to plain text of theRecord
		set theName to name of theRecord
		set theUUID to uuid of theRecord
		
		set theSearchList to "\"" & theUUID & "\"" -- prepare the search string with UUID
		set theGroup to get record with uuid MainUUID -- don't know what MainUUID means
		set theRecords to search "mdggtext:" & theSearchList & space & "kind:markdown" in theGroup -- Note: if custom metadata field has name "custom", then the search addresses it with "mdcustom"
		
		set theList to {}
		repeat with each in theRecords
			-- Put each backlink of a new line
			set the end of theList to "[" & name of each & "]" & "(" & reference URL of each & ")" & "<br>"
		end repeat
		set theList to my sortlist(theList)
		
		-- Remove old Returnlinks section
		try
			set oldDelims to AppleScript's text item delimiters
			set AppleScript's text item delimiters to theDelimiter
			set delimitedList to every text item of theText
			set AppleScript's text item delimiters to oldDelims
		on error
			-- set AppleScript's text item delimiters to oldDelims
		end try
		
		-- Add new ReturnLinks section
		try
			set theText to item 1 of delimitedList
			set the plain text of theRecord to theText & theDelimiter & return & theList as text
		end try
		
		
	end repeat
	display notification "Success part 2!"
end tell

-- END OF MAIN PROGRAM


on sortlist(theList)
	set theIndexList to {}
	set theSortedList to {}
	repeat (length of theList) times
		set theLowItem to ""
		repeat with a from 1 to (length of theList)
			if a is not in theIndexList then
				set theCurrentItem to item a of theList as text
				if theLowItem is "" then
					set theLowItem to theCurrentItem
					set theLowItemIndex to a
				else if theCurrentItem comes before theLowItem then
					set theLowItem to theCurrentItem
					set theLowItemIndex to a
				end if
			end if
		end repeat
		set end of theSortedList to theLowItem
		set end of theIndexList to theLowItemIndex
	end repeat
	return theSortedList
end sortlist

Just a short explanation concerning the try blocks: I agree that the speed is probably not an issue here.
What I wanted to suggest was this:

set oldDelims to AppleScript's text item delimiters
set AppleScript's text item delimiters to theDelimiter
  repeat with theRecord in theSelection
     try
     set theText to plain text of theRecord
     set delimitedList to every text item of theText
     set theTruncatedText to item 1 of delimitedList
     add custom meta data theTruncatedText for PlainText to theRecord
     end try
  end repeat
  set AppleScript's text item delimiters to oldDelims

So basically, I’d move the things that have no relation to the repeat loop (i.e. setting the delimiters) outside of it. This is mainly a question of style, unless you are processing huge amounts of data in a loop. My original comment was wrong. There’s probably no error handler needed here because there’s nothing it can do anyway, and you want to go over all the true records in the selection anyway.

BTW: In the new version of the script, you have set theTruncatedText to item 1 of delimitedList right after the end try. If, as you said, the try block is useful to catch erroneously selected folders et al, the set part should be inside the try block, I think. As well as the next line (add custom metadata…). Otherwise, you’d set the custom meta data of a folder to theTruncatedText of the last successfully processed record.

As to the rest, I just wanted to make some suggestions (dialog vs. log message) and get some clarifications (  in markdown). I still don’t see how get record with uuid MainUUID works when MainUUID is set to “” and never changed, but if it works…

@chrillek, you are of course right about the delimiter swapping in part 1. In part 2 there is a similar swapping of delimiters, but there I think it is necessary, as some operations with the standard delimiter are carried out in between. I’ve updated the script in my reply two above (I don’t think I can edit the original post, now that replies have come in).

I’ve looked a bit more at the code, and also noted that the handlers replaceText() and trimtext() are not really used. The former is not called at all, and the latter is called, but with the request to trim "", i.e. nothing.

Those are pitfalls of grabbing someone else’s script, and fooling around with it without really studying it. I never even noted that replaceText is not called. All I saw is that the existing code would do what I want for a single file, and ergo, all I have to do is put it in a loop. :frowning: I suspect Bernardo_V needed those functions in his more general scripts. I also don’t know what property MainUUID : "" does, but it’s clearly needed. For now, I can live with being ignorant about that.

With all the cruft removed, the script still runs roughly at the same speed. I suspect that the interactions with DT take their time. Naturally, I suspected the DT searches to be the biggest resource hog. I tested this by replacing the actual search with a hard-coded “search result”, and surprisingly, that did not make much of a difference. So the DT search must be fast compared to other (scripting) holdups.

Anyway, I think I’m good with this for now. This script does what it needs to do. It will take me a while to (hopefully) accumulate enough notes in this DB to make speed an issue. If that time ever comes, I’ll look into this again.

Sorry, I don’t know what I did there. Testing without helper scripts was not a good idea …

Here’s a helper script that removes the backlink section and the meta data. After running it the new version below should work.

Helper scrpt: Remove Backlinks Section and Meta Data
-- Helper script: Remove Backlinks Section and Meta Data

property theDelimiter : "###### Backlinks"
property customMetadataName : "ggtext" -- name of the custom metadata field for the .md source. Make sure to use lowercase

tell application id "DNtp"
	try
		set theSelection to selection of viewer window 1
		if theSelection = {} then error "Nothing selected"
		
		repeat with thisRecord in theSelection
			set theText to plain text of thisRecord
			if theText contains theDelimiter then set plain text of thisRecord to my getTruncatedText(theText)
			if (get custom meta data for customMetadataName from thisRecord) ≠ missing value then add custom meta data "" for customMetadataName to thisRecord
		end repeat
		
	on error error_message number error_number
		if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
	end try
end tell

on getTruncatedText(theText)
	set oldDelims to AppleScript's text item delimiters
	set AppleScript's text item delimiters to theDelimiter
	set delimitedList to every text item of theText
	set AppleScript's text item delimiters to oldDelims
	set theTruncatedText_trimmed to my trimEnd(item 1 of delimitedList)
end getTruncatedText

on trimEnd(str)
	local str, whiteSpace
	try
		set str to str as string
		set whiteSpace to {character id 10, return, space, tab}
		try
			repeat while str's last character is in whiteSpace
				set str to str's text 1 thru -2
			end repeat
			return str
		on error number -1728
			return ""
		end try
	on error eMsg number eNum
		error "Can't trimEnd: " & eMsg number eNum
	end try
end trimEnd

2012 MacBook Pro, 2,5 GHz

I know. I fixed my stupid script and did some tests.

Tested with 100 records, each containing links to the 99 others.

Initial run:

Initial Run

Run again without changing records:

Without changing records

Run after removig one different link in 10 records

Removig one different link in 10 records

You don’t exclude unnecessary interactions. That’s what I mean with “only writing text and/or meta data if necessary”.

Look at the AppleEvents used (the number at the right corner). See how your script sends 20606 on each run (don’t know why it’s 20586 in the last capture)? Then see the difference on each run of my script.

For creating test records I used scripts I already had, if you want I’ll post them.

Here’s a new version

-- Append Reference URL backlinks in markdown text (via custom meta data)
-- https://discourse.devontechnologies.com/t/backlinks-aka-incoming-links-revisited-dttg/58299
-- Idea: @gg378 
-- Improvements: @pete31

property theGroupUUID : "" -- optional search scope
property theDelimiter : "###### Backlinks" -- Delimiter of choice. e.g. # Backlinks
property customMetadataName : "ggtext" -- name of the custom metadata field for the .md source. Make sure to use lowercase

tell application id "DNtp"
	try
		set theGroup to get record with uuid theGroupUUID
		set theSelection to selection of viewer window 1
		if theSelection = {} then error "Nothing selected."
		
		repeat with thisRecord in theSelection
			set theText to plain text of thisRecord
			if theText ≠ "" then
				set theContent to item 1 of (my tid(theText, theDelimiter))
				if theContent contains "x-devonthink-item://" then
					add custom meta data my trimEnd(theContent) for customMetadataName to thisRecord
				else
					if (get custom meta data for customMetadataName from thisRecord) ≠ missing value then add custom meta data "" for customMetadataName to thisRecord
				end if
			end if
		end repeat
		
		repeat with thisRecord in theSelection
			set theText to plain text of thisRecord
			set theResults to search "md" & customMetadataName & ":\"" & (uuid of thisRecord) & "\" kind:markdown" in theGroup
			
			if theResults ≠ {} then
				if theText contains theDelimiter then
					set theBackLinksText to item 2 of (my tid(theText, theDelimiter))

					if (count theResults) = ((count my tid(theBackLinksText, "x-devonthink-item://")) - 1) then
						repeat with thisResult in theResults
							if (reference URL of thisResult) is not in theBackLinksText then
								my updateBacklinks(thisRecord, theText, theResults)
								exit repeat
							end if
						end repeat
					else
						my updateBacklinks(thisRecord, theText, theResults)
					end if
					
				else
					my updateBacklinks(thisRecord, theText, theResults)
				end if
			else
				if theText contains theDelimiter then set plain text of thisRecord to my trimEnd(item 1 of (my tid(theText, theDelimiter)))
			end if
		end repeat
		
		display notification "Done"
		
	on error error_message number error_number
		if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
	end try
end tell

on updateBacklinks(theRecord, theText, theResults)
	tell application id "DNtp"
		set theList to {}
		repeat with thisResult in theResults
			set the end of theList to "[" & (name of thisResult) & "](" & (reference URL of thisResult) & ")" & "&nbsp;&nbsp; ⚫︎ &nbsp;&nbsp;"
		end repeat
		set theList to my sort_list(theList)
		if theText ≠ "" then
			set plain text of theRecord to my trimEnd(item 1 of (my tid(theText, theDelimiter))) & linefeed & linefeed & theDelimiter & linefeed & theList as text
		else
			set plain text of theRecord to linefeed & linefeed & theDelimiter & linefeed & theList as text
		end if
	end tell
end updateBacklinks

on tid(theText, theDelimiter)
	set d to AppleScript's text item delimiters
	set AppleScript's text item delimiters to theDelimiter
	set theTextItems to text items of theText
	set AppleScript's text item delimiters to d
	return theTextItems
end tid

on trimEnd(str)
	local str, whiteSpace
	try
		set str to str as string
		set whiteSpace to {character id 10, return, space, tab}
		try
			repeat while str's last character is in whiteSpace
				set str to str's text 1 thru -2
			end repeat
			return str
		on error number -1728
			return ""
		end try
	on error eMsg number eNum
		error "Can't trimEnd: " & eMsg number eNum
	end try
end trimEnd

on sort_list(theList)
	considering numeric strings
		set theIndexList to {}
		set theSortedList to {}
		repeat (length of theList) times
			set theLowItem to ""
			repeat with a from 1 to (length of theList)
				if a is not in theIndexList then
					set theCurrentItem to item a of theList as text
					if theLowItem is "" then
						set theLowItem to theCurrentItem
						set theLowItemIndex to a
					else if theCurrentItem comes before theLowItem then
						set theLowItem to theCurrentItem
						set theLowItemIndex to a
					end if
				end if
			end repeat
			set end of theSortedList to theLowItem
			set end of theIndexList to theLowItemIndex
		end repeat
	end considering
	return theSortedList
end sort_list

See how your script sends 20606 on each run (don’t know why it’s 20586 in the last capture)?

Presumably the upper part of the timing/events graphs shows my script? (Btw, what software do you use for that?)

I ran your scripts over my test files. Indeed, it is significantly faster on subsequent runs. For some reason, I interpreted your first post such that the “unnecessary interactions” mean “not copying md source to the custom field if there is no item link in there”, which would not work in my test, as all my files contain links.

Looking at your new script, I now realize that the speedup comes from testing which backlink sections actually need to be written back into the md file. Nice! I would not have guessed that, because writing to the DB doesn’t seem to be a problem when I copy all md-sources to the custom field in part 1 of my script. If that were true, my part 2 should talk roughly as long as my part 1, not 10x longer. So I don’t think I really understand what’s going on in terms of the timing.

Let me see how you go about deciding whether the backlink section (BLS) needs an update. You check:

  1. Are there any search results? If not → skip updating, and if necessary, delete old BLS
  2. if yes, is there already a BLS? If no → make full BLS update
  3. if yes, need to check whether old and new are identical
  4. if there are fewer search result than existing BLS → definitely need update
  5. if equal or more new results, check in detail

If that’s correct, I wonder about 5.

if ( count theResults) ≥ theBacklinkCount_old then

It’s not wrong, but why lump those cases together with ? If >, then an update is definitely needed. Only the case = requires the careful link-by-link check. Do I get that right? Of course, with it will still work, and in the case of seldom link updates, be still almost as efficient. Wouldn’t one naturally lump < and > into one “definite update” category and then proceed to the = case? Just wondering.

The speedup provided by this version would be very welcome (and your 2012 MBP might be slower than my 2016 MB, after all, my 2010 top-of-the-line 17" MBP certainly was!)

Yes, the upper is yours. That’s Script Debugger

Good catch! Changed it in the script above. Thanks!

That’s for setting the search scope. I wondered too that it’s no problem to tell DEVONthink to search in a group that wasn’t defined but it obviously works.

BTW You could use this group too to get the records automatically without the need of selecting them each time. Adding this line

set theSelection to children of theGroup

and commenting out the two lines that currently get the selection should do it. However if your records are nested into subgroups you’ll need a recursive handler instead.

Tested with much more realistic data:

500 records, each with a random link count ranging from 1 to 10.

Initial run:

Initial Run (500)

Run again without changing records:

Without changing records (500)

Run after removing one different link in 10 records

Removig one different link in 10 records (500)

That’s fast!

:slight_smile:

You can even use search ... in missing value without “getting” the non-existing group first. It returns results from exactly one database (no idea how it decides on this database). If you leave out the “in …” part, it returns results from all databases.

I’m wondering if that’s a feature or something else.

It won’t be a (custom) metadata field but an extension of the Document > Links inspector instead. And of course it will be scriptable:

tell application id "DNtp"
	repeat with theRecord in (selection as list)
		repeat with theReference in incoming references of theRecord
			set theURL to reference URL of theReference
			...
		end repeat
	end repeat
end tell

A property for outgoing references will be available too of course.

3 Likes

For the record: @cgrunenberg said that this is a bug: What does search in a non-existing group do?
using an invalid parameter for in should give the same results as no in parameter at all.

1 Like