Alternatives for achieving transclusion within DT3 (wiki post)

This is a feature sorely missing in DT3’s Multimarkdown implementation and one can easily see the reason for it: MMD transclusion relies on files being on the same folder and files are rarely in the same folder in the database internals. (Of course, one could add the path to make it work, but the path could change without warning which would break the thing again.)

Alternative 1 (New)

Script posted in Github

Alternative 2 - Indexed files + External MMD Preview

All of this can be avoided by using indexed files and keeping them inside the same folder. One would then have to use Multimarkdown Composer, Marked or nvUltra for preview.

Variant 2.1 - Using iA Writers transclusion

iA Writer allows for transclusion using its specific syntax /name.md (Suggested by @ryanjamurphy)

Alternative 3 - Script by @ngan

Script: Refreshable/Portable merged view of files in mixed formats + direct[almost] editing/addition of source files + dynamically linked to the contents of groups/tags .

Alternative 4 - HTML

I found several solutions online, but the only one that sort of worked was using iframes.

Example:

<iframe src="..." seamless width="100%" height="600"></iframe>

One can even create a HTML file and transclude markdown files with this method, but it won’t look very good.


Now here is a question for the HTML wizards out there: is there other ways to achieve with HTML?

Can anyone think of yet another alternative?

This is a wiki post. Feel free to edit it.

UPDATES:
2020-05-20-13-29-58: Posted working version of the transclusion script to github.

1 Like

I have been trying to figure this out lately, too. I have two to add:

  1. iA Writer provides transclusion via its Content Blocks feature:

A neat thing about this implementation is that it also works for other formats, e.g., CSV (or DT sheets).

  1. A one-time transclusion-like implementation is DT’s Merge Documents.

Arguably you could script or macro Merge Documents to keep updating a master file via a set of DEVONthink UUIDs.

Of course, the dream is native implementation… hopefully someday soon!

Edit: the HTML concept is neat, though. I imagine it drops all raw markdown in the iFrame?

2 Likes

Precisely. I still have not looked into how to make the css be applied into the trascluded content.

The script linked above does this, but it relies on the UUIDs being in the finder comment section. I will try to cobble something together now to see if it could be made to work by adding them to the MMD header or to the body of the text it self.

1 Like

I have just come up with this. So far, it works as expected over here.

It replaces {{File.md}} with the content of the record, as long as the name is unique and it is in the same database. It will keep the reference in place, but comment it out (<!-- {{File.md}} -->).

Just in case, you might want to back stuff up before boarding this train.
Or send your dummy doll in for the journey instead.

(You’ll need the RegexAndStuffLib script in ~/Library/Script Libraries/)

use AppleScript version "2.4" -- Yosemite (10.10) or later
use script "RegexAndStuffLib"
use scripting additions


tell application id "DNtp"
	
	set theRecord to (content record of think window 1)
	set theDB to the database of theRecord
	set theMainText to the plain text of theRecord
	if theMainText contains "<!-- " then
		set theFiles to regex search theMainText search pattern "\\<!-- \\{\\{(.+)\\.md\\}\\} -->"
		
		repeat with theFile in theFiles
			set theReplaceTemplate to my replacetext(theFile, "<!-- ", "")
			set theReplaceTemplate to my replacetext(theReplaceTemplate, " -->", "")
			set theSearchPattern to my replacetext(theFile, "{", "\\{")
			set theSearchPattern to my replacetext(theSearchPattern, "}", "\\}")
			set theSearchPattern to theSearchPattern & "(.|\\s)+?\\*\\*\\*"
			set theMainText to regex change theMainText search pattern theSearchPattern replace template theReplaceTemplate
			
		end repeat
	end if
	
	set theFiles to regex search theMainText search pattern "^\\{\\{(.+)\\.md\\}\\}"
	repeat with theFile in theFiles
		--set theFile to item 1 of theFiles
		set theSubName to ""
		set theSearch to ""
		set theSubText to ""
		set theSubRecord to ""
		set theReplacement to ""
		
		try
			
			set theSearch to my replacetext(theFile, "{{", "")
			set theSearch to my replacetext(theSearch, ".md}}", "")
			set theSearch to search "name:\"" & theSearch & "\"" in theDB
			set theSubRecord to the item 1 of theSearch
			set theSubText to the plain text of theSubRecord
			set theSubURL to the reference URL of theSubRecord
			set theSubURL to "[Edit](" & theSubURL & ")"
			set theReplacement to "<!-- " & theFile & " -->" & return & return & theSubText & "  " & return & theSubURL & return & return & "***"
			set theMainText to my replacetext(theMainText, theFile, theReplacement)
			
		end try
		
	end repeat
	

	set the plain text of theRecord to theMainText
	
	
end tell

on replacetext(theString, old, new)
	set {TID, text item delimiters} to {text item delimiters, old}
	set theStringItems to text items of theString
	set text item delimiters to new
	set theString to theStringItems as text
	set text item delimiters to TID
	return theString
end replacetext

Interesting idea re: iA Writer for transclusion

While it works within iA Writer though, I cannot figure out a way to move such a document to DT3 so I can then use “Open With” and view the original source document in iAWriter.

@ryanjamurphy - Got it working

I saved my iA Writer Inclusion demo document in iCloud and then indexed the iA Writer folder in Devonthink

Now I can use “Open With” iA Writer inside Devonthink and the document appears with Transclusion as desired

I agree that native support of transclusion would be ideal, but this is a pretty straightforward and frictionless solution in the interim

1 Like

I added some new features and adapted it to a smart rule.

  • It is now possible to trasclude in two levels (file c contained in b will appear in a if a attempts to transclude b).
  • It is possible to transclude a section of a file using {{Section|File.md}}. It will fetch the whole text until another section with the same level (h1, h2 or h3) or until ***.

Script
use AppleScript version "2.4" -- Yosemite (10.10) or later
use script "RegexAndStuffLib"
use scripting additions

on performSmartRule(theRecords)
	tell application id "DNtp"
		set Path_A to false
		set Path_B to false
		
		set n to (count selection)
		
		if n = 1 then set Path_A to true
		if n > 1 then set Path_B to true
		
		if Path_A then
			
			set theRecord to (content record of think window 1)
			set theDB to the database of theRecord
			set theMainText to the plain text of theRecord
			
			if theMainText contains "<!-- " then
				set theFiles to regex search theMainText search pattern "\\<!-- \\{\\{(.+)\\.md\\}\\} -->" -- look for commented out transclusion marks
				
				repeat with theFile in theFiles
					set theReplaceTemplate to my replacetext(theFile, "<!-- ", "") -- reverse to uncommented
					set theReplaceTemplate to my replacetext(theReplaceTemplate, " -->", "") -- reverse to uncommented
					set theSearchPattern to my replacetext(theFile, "{", "\\{") -- Prepare to look for the commented mark using regex
					set theSearchPattern to my replacetext(theSearchPattern, "}", "\\}") -- Prepare to look for the commented mark using regex
					set theSearchPattern to my replacetext(theSearchPattern, "|", "\\|") -- Prepare to look for the commented mark using regex
					set theSearchPattern to theSearchPattern & "(.|\\s)+?\\*\\*\\*" -- Search pattern including transclusion mark and transcluded text
					set theMainText to regex change theMainText search pattern theSearchPattern replace template theReplaceTemplate
					
				end repeat
			end if
			
			set theFiles to regex search theMainText search pattern "^\\{\\{(.+)\\.md\\}\\}"
			if theFiles is not {} then
				repeat with theFile in theFiles
					--set theFile to item 1 of theFiles
					set theSubName to ""
					set theSearch to ""
					set theSubText to ""
					set theSubRecord to ""
					set theReplacement to ""
					set theSection to ""
					
					
					set theSearchName to my replacetext(theFile, "{{", "")
					set theSearchName to my replacetext(theSearchName, ".md}}", "")
					
					if theSearchName contains "|" then
						
						
						try -- parse variable
							set oldDelims to AppleScript's text item delimiters -- salvar o delimitador padrão
							set AppleScript's text item delimiters to {"|"}
							set theText to theSearchName
							set delimitedList to every text item of theText
							set AppleScript's text item delimiters to oldDelims -- restaurar delimitador padrão
						on error
							set AppleScript's text item delimiters to oldDelims -- restaurar delimitador padrão em caso de erro
						end try
						
						set theSection to item 1 of delimitedList
						set theSearchName to item 2 of delimitedList
						
					end if
					
					set theSearch to search "name:\"" & theSearchName & "\"" in theDB
					if theSearch is not {} then
						set theSubRecord to the item 1 of theSearch
						set theSubText to the plain text of theSubRecord
						if theSection is not "" then
							
							set n to (count my decoupe(theSection, "#")) - 1
							
							set theSectionSearch to theSection & "((.|\\s)+)" & "(#{" & n & "}|\\*\\*\\*)"
							
							set theSectionText to regex search theSubText search pattern theSectionSearch replace template "$1"
							
							set theSectionText to my replacetext(theSectionText, "***", "")
							
							set theSubText to theSectionText
							
						end if
						set theSubText to regex change theSubText search pattern "(aliases: .+)" replace template ""
						set theSubText to regex change theSubText search pattern "(tags: .+)" replace template ""
						set theSubURL to the reference URL of theSubRecord
						set theSubURL to "<div style=\"text-align: center\"><a href=\"" & theSubURL & "?reveal=1\">[Edit section]" & "</a></div>  "
						
						
						-- Randomize footnotes
						
						set theSubText to my replacetext(theSubText, "[^1]", "[^" & (random number 100) & "]")
						set theSubText to my replacetext(theSubText, "[^2]", "[^" & (random number 100) & "]")
						set theSubText to my replacetext(theSubText, "[^3]", "[^" & (random number 100) & "]")
						set theSubText to my replacetext(theSubText, "[^4]", "[^" & (random number 100) & "]")
						set theSubText to my replacetext(theSubText, "[^5]", "[^" & (random number 100) & "]")
						set theSubText to my replacetext(theSubText, "[^6]", "[^" & (random number 100) & "]")
						set theSubText to my replacetext(theSubText, "[^7]", "[^" & (random number 100) & "]")
						set theSubText to my replacetext(theSubText, "[^8]", "[^" & (random number 100) & "]")
						set theSubText to my replacetext(theSubText, "[^9]", "[^" & (random number 100) & "]")
						-- end randomize
						--				set theSign to "\\* \\* \\*"
						set theSign to "---"
						--				set theSign to "```" & return & theSearchName & return & "```" & return
						set theReplacement to "<!-- " & theFile & " -->" & return & return & theSign & return & theSubText & "  " & return & theSubURL & return & return & "***"
						set theMainText to my replacetext(theMainText, theFile, theReplacement)
						
					end if
					
				end repeat
			end if
			
			
			set theFiles to regex search theMainText search pattern "^\\{\\{(.+)\\.md\\}\\}"
			if theFiles is not {} then
				repeat with theFile in theFiles
					--set theFile to item 1 of theFiles
					set theSubName to ""
					set theSearch to ""
					set theSubText to ""
					set theSubRecord to ""
					set theReplacement to ""
					set theSection to ""
					
					
					set theSearchName to my replacetext(theFile, "{{", "")
					set theSearchName to my replacetext(theSearchName, ".md}}", "")
					
					if theSearchName contains "|" then
						
						
						try -- parse variable
							set oldDelims to AppleScript's text item delimiters -- salvar o delimitador padrão
							set AppleScript's text item delimiters to {"|"}
							set theText to theSearchName
							set delimitedList to every text item of theText
							set AppleScript's text item delimiters to oldDelims -- restaurar delimitador padrão
						on error
							set AppleScript's text item delimiters to oldDelims -- restaurar delimitador padrão em caso de erro
						end try
						
						set theSection to item 1 of delimitedList
						set theSearchName to item 2 of delimitedList
						
					end if
					
					set theSearch to search "name:\"" & theSearchName & "\"" in theDB
					if theSearch is not {} then
						set theSubRecord to the item 1 of theSearch
						set theSubText to the plain text of theSubRecord
						if theSection is not "" then
							
							set n to (count my decoupe(theSection, "#")) - 1
							
							set theSectionSearch to theSection & "((.|\\s)+)" & "(#{" & n & "}|\\*\\*\\*)"
							
							
							set theSectionText to regex search theSubText search pattern theSectionSearch replace template "$1"
							
							set theSectionText to my replacetext(theSectionText, "***", "")
							
							set theSubText to theSectionText
							
						end if
						set theSubText to regex change theSubText search pattern "(aliases: .+)" replace template ""
						set theSubText to regex change theSubText search pattern "(tags: .+)" replace template ""
						set theSubURL to the reference URL of theSubRecord
						set theSubURL to "<sup>[\\[Edit subsection\\]](" & theSubURL & "?reveal=1)</sup>  "
						
						
						-- Randomize footnotes
						
						set theMainText to my replacetext(theMainText, "[^1]", "[^" & (random number 100) & "]")
						set theMainText to my replacetext(theMainText, "[^2]", "[^" & (random number 100) & "]")
						set theMainText to my replacetext(theMainText, "[^3]", "[^" & (random number 100) & "]")
						set theMainText to my replacetext(theMainText, "[^4]", "[^" & (random number 100) & "]")
						set theMainText to my replacetext(theMainText, "[^5]", "[^" & (random number 100) & "]")
						set theMainText to my replacetext(theMainText, "[^6]", "[^" & (random number 100) & "]")
						set theMainText to my replacetext(theMainText, "[^7]", "[^" & (random number 100) & "]")
						set theMainText to my replacetext(theMainText, "[^8]", "[^" & (random number 100) & "]")
						set theMainText to my replacetext(theMainText, "[^9]", "[^" & (random number 100) & "]")
						
						-- end randomize
						
						
						set theReplacement to "<!-- " & theFile & " -->" & return & return & theSubText & "  " & theSubURL & return
						set theMainText to my replacetext(theMainText, theFile, theReplacement)
						
					end if
					
				end repeat
			end if
			
			set theMainText to my replacetext(theMainText, "-->
			
			---", "-->")
			
			
			set the plain text of theRecord to theMainText
			
		end if
		
		
		if Path_B then
			set theRecords to the selection
			repeat with theRecord in theRecords
				
				set theDB to the database of theRecord
				set theMainText to the plain text of theRecord
				
				if theMainText contains "<!-- " then
					set theFiles to regex search theMainText search pattern "\\<!-- \\{\\{(.+)\\.md\\}\\} -->" -- look for commented out transclusion marks
					
					repeat with theFile in theFiles
						set theReplaceTemplate to my replacetext(theFile, "<!-- ", "") -- reverse to uncommented
						set theReplaceTemplate to my replacetext(theReplaceTemplate, " -->", "") -- reverse to uncommented
						set theSearchPattern to my replacetext(theFile, "{", "\\{") -- Prepare to look for the commented mark using regex
						set theSearchPattern to my replacetext(theSearchPattern, "}", "\\}") -- Prepare to look for the commented mark using regex
						set theSearchPattern to my replacetext(theSearchPattern, "|", "\\|") -- Prepare to look for the commented mark using regex
						set theSearchPattern to theSearchPattern & "(.|\\s)+?\\*\\*\\*" -- Search pattern including transclusion mark and transcluded text
						set theMainText to regex change theMainText search pattern theSearchPattern replace template theReplaceTemplate
						
					end repeat
				end if
				
				set theFiles to regex search theMainText search pattern "^\\{\\{(.+)\\.md\\}\\}"
				if theFiles is not {} then
					repeat with theFile in theFiles
						--set theFile to item 1 of theFiles
						set theSubName to ""
						set theSearch to ""
						set theSubText to ""
						set theSubRecord to ""
						set theReplacement to ""
						set theSection to ""
						
						
						set theSearchName to my replacetext(theFile, "{{", "")
						set theSearchName to my replacetext(theSearchName, ".md}}", "")
						
						if theSearchName contains "|" then
							
							
							try -- parse variable
								set oldDelims to AppleScript's text item delimiters -- salvar o delimitador padrão
								set AppleScript's text item delimiters to {"|"}
								set theText to theSearchName
								set delimitedList to every text item of theText
								set AppleScript's text item delimiters to oldDelims -- restaurar delimitador padrão
							on error
								set AppleScript's text item delimiters to oldDelims -- restaurar delimitador padrão em caso de erro
							end try
							
							set theSection to item 1 of delimitedList
							set theSearchName to item 2 of delimitedList
							
						end if
						
						set theSearch to search "name:\"" & theSearchName & "\"" in theDB
						if theSearch is not {} then
							set theSubRecord to the item 1 of theSearch
							set theSubText to the plain text of theSubRecord
							if theSection is not "" then
								
								set n to (count my decoupe(theSection, "#")) - 1
								
								set theSectionSearch to theSection & "((.|\\s)+)" & "(#{" & n & "}|\\*\\*\\*)"
								
								set theSectionText to regex search theSubText search pattern theSectionSearch replace template "$1"
								
								set theSectionText to my replacetext(theSectionText, "***", "")
								
								set theSubText to theSectionText
								
							end if
							set theSubText to regex change theSubText search pattern "(aliases: .+)" replace template ""
							set theSubText to regex change theSubText search pattern "(tags: .+)" replace template ""
							set theSubURL to the reference URL of theSubRecord
							set theSubURL to "<div style=\"text-align: center\"><a href=\"" & theSubURL & "?reveal=1\">[Edit section]" & "</a></div>  "
							
							
							-- Randomize footnotes
							
							set theSubText to my replacetext(theSubText, "[^1]", "[^" & (random number 100) & "]")
							set theSubText to my replacetext(theSubText, "[^2]", "[^" & (random number 100) & "]")
							set theSubText to my replacetext(theSubText, "[^3]", "[^" & (random number 100) & "]")
							set theSubText to my replacetext(theSubText, "[^4]", "[^" & (random number 100) & "]")
							set theSubText to my replacetext(theSubText, "[^5]", "[^" & (random number 100) & "]")
							set theSubText to my replacetext(theSubText, "[^6]", "[^" & (random number 100) & "]")
							set theSubText to my replacetext(theSubText, "[^7]", "[^" & (random number 100) & "]")
							set theSubText to my replacetext(theSubText, "[^8]", "[^" & (random number 100) & "]")
							set theSubText to my replacetext(theSubText, "[^9]", "[^" & (random number 100) & "]")
							
							-- end randomize
							--				set theSign to "\\* \\* \\*"
							set theSign to "---"
							--				set theSign to "```" & return & theSearchName & return & "```" & return
							set theReplacement to "<!-- " & theFile & " -->" & return & return & theSign & return & theSubText & "  " & return & theSubURL & return & return & "***"
							set theMainText to my replacetext(theMainText, theFile, theReplacement)
							
						end if
						
					end repeat
				end if
				
				
				set theFiles to regex search theMainText search pattern "^\\{\\{(.+)\\.md\\}\\}"
				if theFiles is not {} then
					repeat with theFile in theFiles
						--set theFile to item 1 of theFiles
						set theSubName to ""
						set theSearch to ""
						set theSubText to ""
						set theSubRecord to ""
						set theReplacement to ""
						set theSection to ""
						
						
						set theSearchName to my replacetext(theFile, "{{", "")
						set theSearchName to my replacetext(theSearchName, ".md}}", "")
						
						if theSearchName contains "|" then
							
							
							try -- parse variable
								set oldDelims to AppleScript's text item delimiters -- salvar o delimitador padrão
								set AppleScript's text item delimiters to {"|"}
								set theText to theSearchName
								set delimitedList to every text item of theText
								set AppleScript's text item delimiters to oldDelims -- restaurar delimitador padrão
							on error
								set AppleScript's text item delimiters to oldDelims -- restaurar delimitador padrão em caso de erro
							end try
							
							set theSection to item 1 of delimitedList
							set theSearchName to item 2 of delimitedList
							
						end if
						
						set theSearch to search "name:\"" & theSearchName & "\"" in theDB
						if theSearch is not {} then
							set theSubRecord to the item 1 of theSearch
							set theSubText to the plain text of theSubRecord
							if theSection is not "" then
								
								set n to (count my decoupe(theSection, "#")) - 1
								
								set theSectionSearch to theSection & "((.|\\s)+)" & "(#{" & n & "}|\\*\\*\\*)"
								
								
								set theSectionText to regex search theSubText search pattern theSectionSearch replace template "$1"
								
								set theSectionText to my replacetext(theSectionText, "***", "")
								
								set theSubText to theSectionText
								
							end if
							set theSubText to regex change theSubText search pattern "(aliases: .+)" replace template ""
							set theSubText to regex change theSubText search pattern "(tags: .+)" replace template ""
							set theSubURL to the reference URL of theSubRecord
							set theSubURL to "<sup>[\\[Edit subsection\\]](" & theSubURL & "?reveal=1)</sup>  "
							
							
							-- Randomize footnotes
							
							set theMainText to my replacetext(theMainText, "[^1]", "[^" & (random number 100) & "]")
							set theMainText to my replacetext(theMainText, "[^2]", "[^" & (random number 100) & "]")
							set theMainText to my replacetext(theMainText, "[^3]", "[^" & (random number 100) & "]")
							set theMainText to my replacetext(theMainText, "[^4]", "[^" & (random number 100) & "]")
							set theMainText to my replacetext(theMainText, "[^5]", "[^" & (random number 100) & "]")
							set theMainText to my replacetext(theMainText, "[^6]", "[^" & (random number 100) & "]")
							set theMainText to my replacetext(theMainText, "[^7]", "[^" & (random number 100) & "]")
							set theMainText to my replacetext(theMainText, "[^8]", "[^" & (random number 100) & "]")
							set theMainText to my replacetext(theMainText, "[^9]", "[^" & (random number 100) & "]")
							
							-- end randomize
							
							
							set theReplacement to "<!-- " & theFile & " -->" & return & return & theSubText & "  " & theSubURL & return
							set theMainText to my replacetext(theMainText, theFile, theReplacement)
							
						end if
						
					end repeat
				end if
				
				set theMainText to my replacetext(theMainText, "-->
							
							---", "-->")
				
				
				set the plain text of theRecord to theMainText
				
				
				
				
			end repeat
		end if
		
	end tell
	
end performSmartRule

on replacetext(theString, old, new)
	set {TID, text item delimiters} to {text item delimiters, old}
	set theStringItems to text items of theString
	set text item delimiters to new
	set theString to theStringItems as text
	set text item delimiters to TID
	return theString
end replacetext

on decoupe(t, d)
	local oTIDs, l
	set {oTIDs, AppleScript's text item delimiters} to {AppleScript's text item delimiters, d}
	set l to text items of t
	set AppleScript's text item delimiters to oTIDs
	return l
end decoupe

EDIT (2020-05-18-15-59-26)
Other minor improvements. Now if multiple records are selected, it will work on them, but if only one is selected, then it will act on the record being currently displayed (and not on the one that is selected).

2 Likes

I am excited to try this! Thanks for all your shared scripting, Bernardo.

1 Like

Jim @BLUEFROG, sorry to bother, could you help figure out why this script works if fired from the editor but not as a smart-rule?


-- on performSmartRule(theRecords)
tell application id "DNtp"
	set Path_A to false
	set Path_B to false
	
	set n to (count selection)
	
	if n = 1 then set Path_A to true
	if n > 1 then set Path_B to true
	
	if Path_A then
		set theRecord to (content record of think window 1)
		display notification "Path A"
		
	end if
	
	
	if Path_B then
		repeat with theRecord in theRecords
			
			display notification "Path B"
			
		end repeat
	end if
	
end tell

-- end performSmartRule

P.s.: it goes without saying that it has nothing to do with the commands specific to the smart-rule that one has to comment out to fire the script from the script editor.

A smart rule shouldn’t be processing a selection. The inly time you’d use that is if you have explicitly selected items in a smart rule and are running it On demand.

Sure, but even if I select records and run it on demand it will not work.

Now add a criteria using content matches to the smart-rule.

No issue…

Weird. Doing exactly the same thing, that is, selecting a records and running it, it works if I remove the criteria, but otherwise, it doesn’t.

In DEVONthink, hold the Option key and select Help > Report Bug. Do NOT send the email. Check the Console.log for errors.

2020-05-19 14:16:37.580 DEVONthink 3[1315:82664] Dragging multiple files using the deprecated NSFilenamesPboardType and dragging API. Please update to the NSDraggingSession API.

Doesn’t seem related. @cgrunenberg is going to have to weigh in on this.

It’s a harmless message and indeed not related.

Chris, I guess this is a bug report then. I am experiencing the same issue on two different machines even after rebooting both of them. There is no error msg in the log nor in the console. Perhaps you could try using the same criteria as I have and try to apply it to markdown files to recreate the issue.