A group merge view would be handy

Another nice little tweak! :slight_smile:

Again, this script is a big step forwards for me, as I previously had to add the titles (and the transclusion links themselves) manually. Thank you.

Still, I’m looking for ways to reduce the maintenance aspect of keeping transclusions up-to-date even further: I have a significant number of “atomic notes” for around 50 topic areas. I would like to use transclusions per topic area to review these insights regularly (including on the go) using text-to-speech.

However, re-creating this number of transclusions regularly would still require significant effort even with the help of @Stephen_C’s script, and could be impractical in the busyness of daily life.

I recognize my request for a smart rule - based script is a tall ask based on a single use case (mine).

So the next step forwards would be Pete’s script with some slight modifications. I’ve already figured out how to modify the script such that the transclusion record is neither selected nor opened in a new window, as this simplifies batch processing. My workflow would be to simply select all groups (topics) I have transclusions for once maybe on a weekly basis, which should not take more than a minute or so.

@pete31 I’m now primarily looking for a way to also include the item names of transcluded items in the transclusion. I’d hugely appreciate it, if you can find the time to make this modification. Feel free to send me your paypal, as I recognize there’s “no free lunch” and you’re surely busy as well :wink:

Meanwhile, what have you tried to get where you want to be?

I managed to come up with a pretty good workaround:

  • Modified Steven’s script to no longer ask for the name of newly created transclusions, and instead to give them the name of the containing group…
  • Created KB Maestro macro, which (with a group selected), opens that group and searches it for Markdown items, then selects and transcludes all found Markdown items using the script.

So now my workflow is essentially to select a group and activate the macro with a hotkey to transclude all contained items. A smart rule adds dates to the new transclusion notes and deletes those, which are older than a week. Obviously, generating the transclusions still needs to be done one by one periodically but it’s a good start.

I tried but didn’t manage to modify your script, such that it also adds the item names next to contents.

Well, when transcluding items I think it’s quite reasonable to want to include the heading. Otherwise there’s no context for the contents. But to each his own and, as mentioned, I’m satisfied with my workaround now.

I appreciate your sentiment that this is a place where people share as part of the „community spirit”. But I also think there’s nothing wrong in offering to buy you a beer as a small thank you for your efforts :wink:

You can’t do that via a Smart Rule because in a Smart Rule script the current record (i.e. the one that’s currently processed) doesn’t know the Smart Group’s scope. It would be possible to use the record’s location, however this may not work reliable in case of replicated records.

But it’s possible via a triggered script. Let me know if you want that.

Hi @pete31,

Really appreciate your offer to take another look at this. Even though a smart rule would be the ultimate solution for minimizing maintenance, a triggered script would certainly help a lot to further streamline things compared to my current workaround.

Quick clarification regarding the previous request to “include the heading” to ensure that if we’re ruling out a smart rule script, we’re doing so based on the same understanding of the objective:

What I meant to write here is actually that I want to include the title of the record as a formatted heading (##) in front of the record’s content in the transclusion document. So the end result would ideally be:

  • (##) Record A title (linebreak) Record A content
  • (two line breaks)
  • (##) Record B title (linebreak) Record B content
  • etc

It would be surprising to me if a smart rule could access a record’s contents but not its title. However, there may just be things about the mechanics of scripting in DT that I don’t yet understand. And again, a triggered script would be great as well otherwise!

One thing that would be important is that the script either needs to respect the visible order of items, or it needs to be possible to specify the field, by which records should be ordered in the transclusion. For my use case, I would want them to be ordered by a custom field that I created to group related items.

As a bonus, nice-to-have addition, another thought:
I noticed that the current version of the triggered script only creates a new transclusion, if none has already been created in the respective group. From my perspective, if it is feasible, an automatic replacement of the transclusion would be ideal. This way, the group would always contain the most up-to-date version including new Markdown items that were added into the group. Currently, one would need to check for each group whether it already contains a transclusion and delete it, before triggering the script. With this change, this additional step would no longer be needed.

It’s of course possible to get every property of a record via a Smart Group script. But a record can have multiple parents and it’s not possible to reliably get the right parent (i.e. the group that you’ve set as the Smart Group’s scope).

As there’s no way to get the group (that you’ve set as a Smart Group’s scope) via AppleScript the only (reliable) way would be to set the group’s reference URL in a script property.

That’s only possible if you sort by unsorted.

What type is your Custom Meta Data?

The triggered script already updates the Transclusion record’s contents. You need to deselect and select the group to trigger a triggered script.

1 Like

Thanks for explaining, I understand.

Understood. Maybe the ordering based on a custom field would be an option instead?

Currently, it’s single-line text. This could be changed to any other type though for the kind of information contained in these fields, in case one works better than others for scripting.

I did re-select the group, but was looking at the “added” rather than “modified” field to determine a change… Great that this already works!

This needs a little setup as

  • we can’t use AppleScriptObjC inline in a Smart Rule script
    i.e. you need to create a Script Library

  • we only want the Smart Rule to trigger the script one time
    i.e. you need to make sure it only matches one group

Create a Script Library

  • In Finder:

    • press ⇧⌘G
    • paste ~/Library/Script Libraries
    • if no window is opening
      • paste ~/Library/
      • create folder Script Libraries
  • In Script Editor.app

    • create new document
    • paste this script
    • press ⌘S
    • press ⇧⌘G
    • paste ~/Script Libraries/
    • save with extension scpt
-- Script Library

use AppleScript version "2.4"
use framework "Foundation"
use scripting additions

on sortedArrayUsingDescriptors(theList, sortAscending)
	try
		set theArray to current application's NSMutableArray's arrayWithArray:theList
		set theDescriptor to current application's NSSortDescriptor's sortDescriptorWithKey:"SortBy" ascending:sortAscending selector:"localizedStandardCompare:"
		set theArray_sorted to (theArray's sortedArrayUsingDescriptors:{theDescriptor})
		set theList_sorted to theArray_sorted as list
		
	on error error_message number error_number
		activate
		if the error_number is not -128 then display alert "Error: Handler \"sortedArrayUsingDescriptors\"" message error_message as warning
		error number -128
	end try
end sortedArrayUsingDescriptors


Create a Smart Rule

Prepare group

  • copy the group’s reference URL (Edit > Copy Item Link)
  • add the reference URL to the group’s aliases (see Inspector)

Prepare script

  • replace property theGroup_ReferenceURL with [the group’s reference URL]
  • replace Custom Meta Data key with md + identifier (see Preferences > Data)
  • replace Script Library name

Setup Smart Group

  • Location: the group’s parent group
  • Matching: aliases contains [the group’s reference URL]
  • Action: AppleScript
-- Smart Rule script - Create or update transclusion record

property theGroup_ReferenceURL : "x-devonthink-item://F8BC32DD-AED1-46A3-ADB3-051F5BD0DEF3" -- replace with your group's reference URL

on performSmartRule()
	tell application id "DNtp"
		try
			set theGroup to (get record with uuid theGroup_ReferenceURL)
			set theChildren to children of theGroup whose type = markdown and comment is not "Transcluded group contents"
			if theChildren = {} then return
			
			set theList to {}
			
			repeat with thisChild in theChildren
				try
					set thisChild_CustomMetaData to custom meta data of thisChild
					set thisChild_CustomMetaData_Value to mddescription of thisChild_CustomMetaData -- replace mddescription with "md" + your identifier (see Preferences > Data)
				on error
					set thisChild_CustomMetaData_Value to ""
				end try
				set end of theList to {|SortBy|:thisChild_CustomMetaData_Value, |RefURL|:(reference URL of thisChild), |NameWithoutExtension|:(name without extension of thisChild)}
			end repeat
			
			set theList_sorted to script "TEST"'s sortedArrayUsingDescriptors(theList, true) -- replace TEST with your Script Library's name
			
			set theResults to search "kind:markdown comment:\"Transcluded group contents\"" in theGroup
			if theResults ≠ {} then
				set theTransclusionRecord to item 1 of theResults
			else
				set theTransclusionRecord to create record with {name:"_Transclusion - " & (name of theGroup), type:markdown, source:"", comment:"Transcluded group contents"} in theGroup
			end if
			
			set theTransclusionRecord_Source_list to {}
			
			repeat with i from 1 to (count of theList_sorted)
				set end of theTransclusionRecord_Source_list to "# " & (|NameWithoutExtension| of item i of theList_sorted) & linefeed & linefeed & "{{" & (|RefURL| of item i of theList_sorted) & "}}"
			end repeat
			
			set theTransclusionRecord_Source to my tid(theTransclusionRecord_Source_list, linefeed & linefeed & linefeed)
			
			set plain text of theTransclusionRecord to theTransclusionRecord_Source
			
		on error error_message number error_number
			if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
			return
		end try
	end tell
end performSmartRule

on tid(theInput, theDelimiter)
	set d to AppleScript's text item delimiters
	set AppleScript's text item delimiters to theDelimiter
	if class of theInput = text then
		set theOutput to text items of theInput
	else if class of theInput = list then
		set theOutput to theInput as text
	end if
	set AppleScript's text item delimiters to d
	return theOutput
end tid

Tested and working! If anyone could actually make this viable via a smart rule, I knew it had to be you, Pete :slight_smile:

A couple questions to better understand the inner workings:

  1. In the instructions you wrote that the scope for the smart rule should be the parent group of the group that contains items to be transcluded. Could the scope not be the entire database as well? In this case, I think the correct group should still be identified via the criteria that the alias should contain its item link.

  2. Are the following assumptions correct?

  • One separate smart rule is needed for each group, for which a transclusion should be created
  • The setup for each additional automated transclusion needs to be done individually in a new smart rule

More than happy to put in that prep effort upfront in exchange for full automation, just want to verify that this is indeed necessary.

Allow me to share one more idea for potential further optimization:

If the script could pull the property theGroup_ReferenceURL from the records returned by the smart rule (based on each record’s alias), then wouldn’t it potentially be possible to set up one smart rule to process multiple groups? The script would need to process each record one after the other with the steps it currently applies to the single group that is currently in the smart rule’s scope.

However, I realize this may be highly theoretical and again, the smart rule as-is will 100% address my use case. Thanks a million!!!

Yes! That way it’s possibly to do everything in one Smart Rule. I wrote the script before thinking about how the Smart Rule could match a group (i.e. did step 2 before step 1) and then obviously didn’t think about the possiblity to do everything in one Smart Rule. That’s of course far better.

Setup

Create a Script Library

See instructions here

Create a Smart Rule

Prepare group(s)

  • For each group:
    • copy the group’s reference URL (Edit > Copy Item Link)
    • add the reference URL to the group’s aliases (see Inspector)
    • paste the reference URL somewhere (it’s later needed in the Smart Rule)

Prepare script

  • replace Custom Meta Data key with md + identifier (see Preferences > Data)
  • replace Script Library name

Setup Smart Group

  • Location: the database’s root (or all databases)
  • Matching:
    • any
      • aliases contains [reference URL of group 1]
      • aliases contains [reference URL of group 2]
      • etc.
  • Action: AppleScript
-- Smart Rule script - Create or update transclusion record

on performSmartRule(theGroups)
	tell application id "DNtp"
		try
			repeat with thisGroup in theGroups
				set theChildren to (children of thisGroup whose type = markdown and comment is not "Transcluded group contents")
				
				if theChildren ≠ {} then
					
					set theList to {}
					
					repeat with thisChild in theChildren
						try
							set thisChild_CustomMetaData to custom meta data of thisChild
							set thisChild_CustomMetaData_Value to mddescription of thisChild_CustomMetaData -- replace mddescription with "md" + your identifier (see Preferences > Data)
						on error
							set thisChild_CustomMetaData_Value to ""
						end try
						set end of theList to {|SortBy|:thisChild_CustomMetaData_Value, |RefURL|:(reference URL of thisChild), |NameWithoutExtension|:(name without extension of thisChild)}
					end repeat
					
					set theList_sorted to script "TEST"'s sortedArrayUsingDescriptors(theList, true) -- replace TEST with your Script Library's name
					
					set theResults to search "kind:markdown comment:\"Transcluded group contents\"" in thisGroup
					if theResults ≠ {} then
						set theTransclusionRecord to item 1 of theResults
					else
						set theTransclusionRecord to create record with {name:"_Transclusion - " & (name of thisGroup), type:markdown, source:"", comment:"Transcluded group contents"} in thisGroup
					end if
					
					set theTransclusionRecord_Source_list to {}
					
					repeat with i from 1 to (count of theList_sorted)
						set end of theTransclusionRecord_Source_list to "# " & (|NameWithoutExtension| of item i of theList_sorted) & linefeed & linefeed & "{{" & (|RefURL| of item i of theList_sorted) & "}}"
					end repeat
					
					set theTransclusionRecord_Source to my tid(theTransclusionRecord_Source_list, linefeed & linefeed & linefeed)
					
					set plain text of theTransclusionRecord to theTransclusionRecord_Source
					
				end if
			end repeat
			
		on error error_message number error_number
			if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
			return
		end try
	end tell
end performSmartRule

on tid(theInput, theDelimiter)
	set d to AppleScript's text item delimiters
	set AppleScript's text item delimiters to theDelimiter
	if class of theInput = text then
		set theOutput to text items of theInput
	else if class of theInput = list then
		set theOutput to theInput as text
	end if
	set AppleScript's text item delimiters to d
	return theOutput
end tid


This really is incredible! Works flawlessly.

I have simply set the scope to “is group” and “contains x-devonthink-item in alias” to simplify even more. Of course, this only makes sense because I don’t add item links to groups’ aliases for any other reason than these transclusions.

One thing I did notice: Ordering by custom meta data field doesn’t seem to work on my end. I entered md + my field identifier, so in the script the line reads like this:

set thisChild_CustomMetaData_Value to mdinsight_relations of thisChild_CustomMetaData

I’ve also tried leaving out the md at the front, in both cases though the order of transcluded items is the same as when I enter something random as the identifier.

Any advice what I could try to troubleshoot? This would be one last component to make everything perfect.

Restart DEVONthink. Other than that I‘ve no idea.

@pete31 Quick update: I found what was causing this and have successfully worked around it now.

I was using A1, A2, A3,… up to double-digits in my custom metadata field to sort items, so at some point there is A11, A25 etc. DevonThink displays items with the above values in this order:
*A1
*A2
*A3
*A11
*A25

The order in the transclusion follows a slightly different logic. For the values above, they would be displayed like this:

  • A1
  • A11
  • A2
  • A25
  • A3

With this knowledge, I simply switched to using 1A, 1B, 1C, 1D, 2A, 2C etc. to order and group items. As said, it’s easy to address this, and maybe pointing it out helps someone.

Pete, I now officially declare your script as the unexpectedly perfect solution to what I was trying to accomplish. Again, truly appreciate it.

I used the wrong method. It’s localizedStandardCompare instead of localizedCompare.

Updated the Script Library handler.

1 Like

This script creates/updates a Transclusion record for the group’s supported records and selects or opens it in a new window.

Supported file types

  • Markdown
  • Text
  • HTML
  • RTF
  • RTFD
  • Formatted Note
  • Webarchive
  • Sheet

From help:

Attaching a triggered script

  • Select the item you want to attach a script to.
  • Select Tools > Get Info or Tools > Inspectors > Generic.
  • Click the down arrow next to Script and choose Select. Note the script can be located anywhere, but it must remain in that location for the script to trigger. For convenience, you can create a folder for them in ~/Library/Application Scripts/com.devon-technologies.think3/Menus.
-- Triggered script - Create transclusion for current group's supported records

-- Setup: Click "script" in a group's info inspecton . Select path

property selectTransclusionRecord : true -- if false Transclusion record opens in a new window

on triggered(theGroup)
	tell application id "DNtp"
		try
			set theResults to search "kind:markdown comment:\"Transcluded group contents\"" in theGroup
			if theResults ≠ {} then
				set theTransclusionRecord to item 1 of theResults
			else
				set theTransclusionRecord to create record with {name:"_Transclusion - " & (name of theGroup), type:markdown, source:"", comment:"Transcluded group contents"} in theGroup
			end if
			
			set theChildren_ReferenceURLs to reference URL of children of theGroup whose type = markdown or type = txt or type = html or type = rtf or type = rtfd or type = formatted note or type = webarchive or type = sheet and comment is not "Transcluded group contents"
			if theChildren_ReferenceURLs = {} then error "This group doesn't contain Markdown records"
			set theTransclusionRecord_Source to "{{" & my tid(theChildren_ReferenceURLs, "}}" & linefeed & "{{") & "}}"
			set plain text of theTransclusionRecord to theTransclusionRecord_Source
			
			if selectTransclusionRecord then
				set selection of think window 1 to {theTransclusionRecord}
			else
				open window for record theTransclusionRecord
				activate
			end if
			
		on error error_message number error_number
			if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
			return
		end try
	end tell
end triggered

on tid(theInput, theDelimiter)
	set d to AppleScript's text item delimiters
	set AppleScript's text item delimiters to theDelimiter
	if class of theInput = text then
		set theOutput to text items of theInput
	else if class of theInput = list then
		set theOutput to theInput as text
	end if
	set AppleScript's text item delimiters to d
	return theOutput
end tid

Wow - very cool - will try when I can get the work day to settle down. It’s off with a vengeance this morning.

Hey @pete31, quick question:
I was wondering if it’s theoretically possible to modify the smart rule script such that it also transcludes items contained in subgroups of the group, on which the script is run?
Any hints in this direction would be greatly appreciated.

What are you trying to accomplish?

I’m using the smart rule script from this post to automatically transclude Markdown items contained in a set of groups. For each group, the script creates a transclusion document.

While the script does this perfectly for those items contained directly in the group being processed, it doesn’t transclude items contained in subgroups of the group.

If it did, this would further increase the script’s value for my use case. This is because the transclusions serve as a review document and logically, items in subgroups also need to be reviewed.