I have been absorbing lots of information from this forum over the years and I am very grateful for the time and passion shown by all the contributors. Before today, most, if not all of my posts have been in search of answers. Today, I want to give a little back and hopefully others will find some information, or even a full script - a faster version of the built-in script: Scripts > Export > Listing…, below.
Background
The back story is that I have been losing some data from my databases. I have been reorganizing and making some major changes and suspect that these were all operator errors.
I have been able to recover everything from Arq backups following instructions provided in this forum Partial Restore from Arq or Time Machine - DEVONthink - DEVONtechnologies Community. I first identified a missing file or group from the current database, searched the Arq backups to see when that file disappeared, then recovered the database from just before that time. The nagging question was what else might be missing, so I used the built-in script: Scripts > Export > Listing… which produces a text file that lists the contents of the current database, preserving the hierarchical structure with tabs, to help identify the missing groups.
I generated two listings: one for the live database; and one for the earlier version of the database. Then I compared those listings and found that I had deleted a single group high up in the database taking a lot of files with it.
To compare two listings that may not be in the same order, you can use one of these two terminal commands:
comm -3 <(sort filepath/filename1) <(sort filepath/filename2)
diff -a <(sort filepath/filename1) <(sort filepath/filename2)
The Project
Now that the emergency has passed, I wondered how I might stay on top of my data and be made aware of any similar issues moving forward.
That gave rise to a little project to try and speed up the built-in Listing... script which ran very slowly (measured in 10+ minutes) on my database. The reason for the lack of speed is the multitude of AppleEvents used since each record is processed, one by one.
I wanted to use JXA because of the strengths of the JavaScript language and I have become more familiar with it than AppleScript recently. I am very grateful to @chrillek for sharing his insights in Scripting with JXA which opened the door, and his many contributions to this forum.
The good news is that my new JXA script, running against that same database, completes in mere seconds.
The speed ups that I implemented consist of minimizing AppleEvents which could be translated to AppleScript equally well.
I am happy with the performance, but I wouldn’t be surprised if members of this community were able to speed it up some more.
The script is included at the end of this post, but first I would like to present the approach that I took, and then a simpler script to illustrate some of the principles, before a detailed description of how the final script works.
Approach
Rather than recursively traversing the DEVONthink data hierarchically, which requires AppleEvents at each step, we need to batch process arrays of DEVONthink records with one query to drastically reduce the number of AppleEvents that are required.
There are three key techniques to reducing the number of AppleEvents:
1. defer () as late as possible with Object Specifier
@/houthakker explained, in JXA / get specific properties of all database records, that to accomplish fast batch fetching of the same property for all records in a collection depends on preserving the reference and delaying the function call.
The Slow Way (Multiple AppleEvents)
const allRecords = db.contents(); // Fetches array of proxies,
// not a JS array of objects
const uuids = allRecords.map(record => record.uuid()); // Multiple
// AppleEvents, one for each record!
In JXA, allRecords is not a standard JavaScript array of objects; it is an array of proxy pointers to an external Mac application’s automation layer via AppleEvents.
The JXA Fast Way (1 Single AppleEvent)
Do not extract contents() first. Remove the parentheses from your collection and call the plural property directly:
// Remove () from contents so you hold the reference to the collection,
// then call the property method directly on that collection reference.
const uuids = db.contents.uuid();
2. whose instead of JavaScript filter
JavaScript has powerful processing for arrays including filter(), but @mdbraber showed, in Most efficient way to get all nested tags (JXA) - #12 by mdbraber, that the clumsy whose is much faster.
3. synchronized lists for multiple properties
It isn’t currently possible to return multiple properties with a single query, unless you return the whole enchilada (properties). That was shown to be slower than two queries for name and location in the same Most efficient way to get all nested tags (JXA) - #10 by mdbraber.
However, it doesn’t seem to be a problem since, in my experience, the order of different queries is maintained between individual query calls. @cgrunenberg indicated that DEVONthink preserves the order, but couldn’t vouch for AppleScript Most efficient way to get all nested tags (JXA) - #9 by cgrunenberg. He also suggested that future updates may support returning a tuple like (name, location) in Most efficient way to get all nested tags (JXA) - #15 by cgrunenberg. If so, I would like to see (name, uuid) to support this script.
————————
Simple Example of faster processing
Here is a simple example using these techniques to generate a list of markdown formatted links for all the groups (except the tag groups) in a database.
function createListing(db) { // database
// exclude tag groups - interestingly "/Tags" itself is an ordinary group.
const myParents = db.parents.whose({_match:
[ObjectSpecifier().tagType, 'no tag']});
const pNames = myParents.name();
const pUuids = myParents.uuid();
let theListing = '';
pNames.forEach((name, index) => {
// escape opening `[` and closing `]` brackets
const escapedName =
name.replaceAll("[", "\\[").replaceAll("]", "\\]");
theListing += `[${escapedName}](${pUuids[index]})\n`;
});
return(theListing);
}
This is a flat example, we will need to process the hierarchy for the final script below.
- a list of ObjectSpecifiers is created to reference all ordinary groups. Notice there is no trailing () at this stage, so it’s not a function, and this is not a JavaScript array as explained in Working with Objects | JavaScript for Automation (JXA) by @chrillek.
const myParents = db.parents.whose({_match:
[ObjectSpecifier().tagType, 'no tag']});
-
note that
const myParents = db.parents.whose({tagType, 'no tag'});didn’t work . -
the Object Specifiers are then used as a basis to populate two JavaScript arrays with the names and uuids for each group in the same order:
const pNames = myParents.name(); // 1 AppleEvent
const pUuids = myParents.uuid(); // 1 AppleEvent
- these are then used to build the listing of Markdown links in the script above.
————————
Nature of the Arrays returned by the query
The top level array can contain holes, depending on any “filtering” accomplished earlier with the whose clause in the Object Specifier. In other words, it may be a sparse array.
const allParents = db.parents; // an array with a value at each index
const myParents = db.parents.whose({_match:
[ObjectSpecifier().kind, 'Smart Group']}); // definitely sparse
Note: the DEVONthink scripting dictionary stresses that the property name: type, not kind should be used, but that doesn’t work for me.
To get the kids (documents and subgroups) contained in the parents:
const myKidsNames = myParents.children.name();
const myKidsUuids = myParents.children.uuid();
Unlike the parents arrays, these arrays are nested with each entry containing a (possible empty) array containing the children of each parent.
console.log(myKidsnames); // produces a simple list of 90 items for my test database with 14 parents.
Note: Although these arrays are nested, when you log them, the array is flattened.
————————
Building the Listing script
After efficiently filling the key arrays:
myParents: parents who are not tag groups (pNames,pUuids)myKids: children ofmyParents: documents or subgroups(myKidsNames,myKidsUuids)myGroups: subgroups frommyKids(myGroupsNames,myGroupsUuids)
The next step is to build a tree with a node for each element of MyParents containing info for that parent and the items it contains (the kid(s)):
const nodeMap = new Map();
myKidsNames.forEach((kid,index) => {
let AllDocs = [];
if (kid[0] ) { // there are documents or subgroups in this parent
for (let j=0; j<kid.length; j++) {
let thisDoc = {};
thisDoc.name = kid[j];
thisDoc.uuid = myKidsUuids[index][j];
AllDocs.push(thisDoc);
}
}
nodeMap.set(pUuids[index], {
uuid: pUuids[index],
name: pNames[index],
documents: AllDocs,
children: []
});
});
We start by building a documents array containing all the kids of the parent, even if a kid is a subgroup and not a document. The children array starts off empty. Subsequent processing will move items from documents to children to complete the nodeMap which will then be a valid tree structure.
Again, we identify all those subgroups in one query:
const myGroups = myParents.children.whose({kind: "group"});
const myGroupsNames = myGroups.name();
const myGroupsUuids = myGroups.uuid();
Note: these are sparse arrays (of arrays) with empty slots where kind is not group.
Next, we link children to parents, and clean up documents by removing the subgroups.
myGroupsNames.forEach((subgroup,index) => {
const parentNode = nodeMap.get(pUuids[index]);
subgroup.forEach((sub, index2) => {
const childNode = nodeMap.get(myGroupsUuids[index][index2]);
parentNode.children.push(childNode);
parentNode.documents = parentNode.documents.filter(doc => doc.uuid !== childNode.uuid); // remove subgroup from documents
});
});
Apart from some special processing to add the documents from the root into the nodeMap, I also choose to remove the documents from the Smart Groups in the root to reduce clutter in the final listing.
We can then generate the listing recursively from the nodeMap without any AppleEvents being required.
The script
The processing of interest is in the function createListing(). The main processing just provides a file picker, before calling createListing(), and traverseDFS() is a helper function to traverse the tree of nodes, depth first.
Edit: 2026-06-30
See below for the changes. As a result the snippets above may no longer match the code exactly.
// exportListing (fast)
// pmf: 2026-06-30
// ---------
// a. updated to use `recordType` instead of `kind` in `whose` queries
// three edits =>
// 1. groups in kids
// 2. rootDocs
// 3. smart groups
// b. fixed progress indicator
// ---------
// variables in the script:
// myParents: parents who are not tag groups (pNames, pUuids)
// myKids: children of myParents = documents or subgroups (myKidsNames, myKidsUuids)
// myGroups: subgroups from myKids (myGroupsNames, myGroupsUuids)
// ---------
// rootDocs: documents in root (rootDocsNames, rootDocsUuids)
// smartGroups: smart groups in root (smartGroupsNames, smartGroupsUuids)
// (rootNames, rootUuids): all children of root
// ---------
(() => {
"use strict";
// traverse the nodeMap depth first
function traverseDFS(node, theTabs) {
let thisListing = "";
if (!node) return;
thisListing += theTabs + node.name + "\n"; // Process the current node
node.children.forEach(child => { // Recursively visit each child
thisListing += traverseDFS(child, theTabs + "\t");
});
// now log documents at this level
if (node.documents.length !== 0) {
node.documents.forEach(r => {
thisListing += theTabs + "\t" + r.name + "\n";
})
}
return thisListing
}
// create a hierarchical listing of all groups and documents (not tags)
function createListing(dt, db) {
// 1. get all groups in db, except tags
// (interestingly "/Tags" itself is an ordinary group)
const myParents = db.parents.whose({_match: [ObjectSpecifier().tagType, 'no tag']});
const pNames = myParents.name(); // 1 AppleEvent
const pUuids = myParents.uuid(); // 1 AppleEvent
// 2. get all the kids: "nested" groups and documents contained in the Parents
const myKidsNames = myParents.children.name(); // 1 AppleEvent
const myKidsUuids = myParents.children.uuid(); // 1 AppleEvent
// 3. get all groups and documents in the root
const rootNames = db.root.children.name();
const rootUuids = db.root.children.uuid(); // the listing is generated iteratively for each rootUuid, so set totalSteps
// Show progress UI
dt.showProgressIndicator("Creating Listing...", {
cancelButton: true,
steps: rootUuids.length // the listing is developed for each item in the root
});
// 4. build the nodeMap
const nodeMap = new Map();
dt.stepProgressIndicator(`parents: ${myParents.length}`);
// 5. Create a map of all nodes, one node per 'kid':
myKidsNames.forEach((kid,index) => {
let AllDocs = [];
// get all nested items: groups and documents contained in this kid
if (kid[0] ) { // kid contains nested item(s)
for (let j=0; j<kid.length; j++) { // process array of nested item(s)
let thisDoc = {};
thisDoc.name = kid[j];
thisDoc.uuid = myKidsUuids[index][j];
AllDocs.push(thisDoc); // put everything in Documents for now
}
}
// one node per 'kid'
nodeMap.set(pUuids[index], {
uuid: pUuids[index],
name: pNames[index],
documents: AllDocs, // groups will be extracted later
children: [] // groups will be moved here, as nodes, later
});
});
dt.stepProgressIndicator(`built the nodeMap: ${nodeMap.size}`);
// 6. get groups from myKids, not the documents
const myGroups = myParents.children.whose({
_and: [
{_match: [ObjectSpecifier().recordType, "group"]},
{_match: [ObjectSpecifier().tagType, "no tag"]}
]
});
// myGroups is a full length array with one slot for each Parent.
// If a parent has no subgroup(s), then that slot is empty.
// Otherwise, the slot is a secondary array, with one entry for each
// group within that group (kid) and empty slots for non-group items.
// Similarly, myGroupsNames and myGroupsUuids are nested arrays
const myGroupsNames = myGroups.name(); // 1 AppleEvent
const myGroupsUuids = myGroups.uuid(); // 1 AppleEvent
// 7. link children to parents
myGroupsNames.forEach((nGroup,index) => { // nGroup is an array of nested group name(s)
const parentNode = nodeMap.get(pUuids[index]);
for (let j=0; j<nGroup.length; j++ ) { // process array of nested group(s)
const childNode = nodeMap.get(myGroupsUuids[index][j]);
if (childNode) {
parentNode.children.push(childNode);
} else {
throw new Error(`childNode: {${nGroup[j]}, ${myGroupsUuids[index][j]}} not found!`)
}
// remove subgroup from documents
parentNode.documents =
parentNode.documents.filter(doc => doc.uuid !== childNode.uuid);
}
});
dt.stepProgressIndicator('linked the children in the nodeMap');
// 8. Special processing for documents (in the root)
const rootDocs = db.root.children.whose({
_and: [
{ _not: [{ _match: [ObjectSpecifier().recordType, "group"] }] },
{ _not: [{ _match: [ObjectSpecifier().recordType, "smart group"] }] }
]
});
const rootDocsNames = rootDocs.name();
const rootDocsUuids = rootDocs.uuid();
// 8a. add documents to nodeMap
rootDocsUuids.forEach((uuid, index) => {
if (uuid) {
nodeMap.set(uuid, {
uuid: uuid,
name: rootDocsNames[index],
documents:[],
children:[]
});
}
});
dt.stepProgressIndicator('finished processing rootDocs');
// 9. Special processing for Smart Groups (in the root)
//const smartGroups = db.root.children.whose({kind: "Smart Group"}); // 3
// == edit 3 == //
const smartGroups = db.root.children.whose({
_match: [ObjectSpecifier().recordType, "smart group"]
});
const smartGroupsNames = smartGroups.name();
const smartGroupsUuids = smartGroups.uuid();
// 9a. remove documents from nodeMap, and therefore the listing
smartGroupsUuids.forEach((uuid, index) => {
const thisNode = nodeMap.get(uuid); // get node from nodeMap
thisNode.documents = []; // remove documents
})
dt.stepProgressIndicator('finished processing smart groups');
// 10. Generate the listing from the root records
let theListing = '';
// 10a. traverse the nodeMap
rootUuids.forEach((uuid, index) => {
// check for user cancelled, if not step the indicator
if (dt.cancelledProgress()) {
throw new Error("user cancelled");
} else {
dt.stepProgressIndicator(rootNames[index]);
if (uuid) {
// exclude 'Tags' and 'Trash'
if (!['Tags', 'Trash'].includes(rootNames[index])) {
theListing += traverseDFS(nodeMap.get(uuid),'');
}
}
}
});
return(theListing);
}
//=================
// main processing
//=================
const dt = Application("DEVONthink");
dt.includeStandardAdditions = true;
ObjC.import('Foundation'); // Objective-C Foundation framework
const db = dt.currentDatabase;
try {
// 1. Check if a database is open
if (!dt.currentDatabase.exists()) {
throw new Error("No database is open.");
}
// 2. Prompt user for the save file location
let theFile;
try {
theFile = dt.chooseFileName({ defaultName: db.name() });
} catch (e) {
// Handle user cancellation (corresponds to error -128)
throw -128;
}
// 3. Generate the listing
const theListing = createListing(dt, db);
// 4. Write the output to a text file
if (theListing !== null) {
let thePath = theFile.toString();
if (!thePath.endsWith(".txt")) {
thePath += ".txt";
}
// write UTF8-encoded text
const str = $.NSString.alloc.initWithUTF8String(theListing + '\n');
const error = Ref();
str.writeToFileAtomicallyEncodingError(thePath, true, $.NSUTF8StringEncoding, error);
}
dt.hideProgressIndicator();
} catch (error) {
dt.hideProgressIndicator();
// If the user cancelled, exit silently. Otherwise, show the alert.
if (error !== -128) {
const msg = error.message || error.toString();
dt.displayAlert("DEVONthink", { message: msg, as: "warning" });
}
}
})();
Note: The progress indicator doesn’t show a x cancel button, and I’m not sure why.
My script is functionally equivalent to the built-in script. It differs in detail if you are comparing the outputs of the two as I did to verify my code.
- my script uses linefeeds
\n, while the original uses carriage returns\r - my script outputs UTF-8, instead of UTF-16
- my listing outputs groups, then documents at each level, the original seems to intermingle groups and documents, based on an alphabetical sorting.
