Something nicer to interact with/search DEVONthink than AppleScript?

I have an Alfred extension that I use for quickly querying DEVONthink databases, but it’s currently using AppleScript/Javascript to interface with DEVONthink.

Limitations are that it’s pretty slow and I have to run the search for each database individually, then merge the results manually and sort by score before displaying. So something like

  const dbs = devonthink.databases();
  const found = dbs.map((db) => doSearch(db, query));
  let results = [];
  for (const db of dbs) {
    results = results.concat(doSearch(db, query));
  }

  results.sort(function(a, b) {
      return b.score - a.score;
  })

It works, but it’s not very performant.

Any ideas on how to make it faster? Any other ways to pull information out of DEVONthink for scripting?

Running the search for each database via AppleScript should be probably more or less as fast as running the same search across all databases in DEVONthink. But sorting of the results is of course a lot slower, therefore one workaround might be to limit the results to n items for each database.

Is it possible to do a direct query against all databases?

The sorting is heavy because each database query gives me the results sorted by score for that database only, and I need to combine those results after the fact. If DT can give me a full list that is already sorted by score and includes results from all databases, I could skip the sorting part

I remember faintly that DEVONthink exposes the metadata files somewhere that were previously able to be indexed with spotlight. I wonder if it’s possible to use that?

No.

You would have to query Spotlight if the Spotlight index (see File > Database Properties) is enabled.

How does one do that, Chris?

The search command doesn’t support this directly but you could check the count of the results and remove the desired ones at the end.

Ok, that was precisely what I needed to know :wink: Thanks!

It seems that you run dbSearch twice. Is that intentional? It might be more economical to save the results of the first search and work with them.

Edit: Whole post was nonsense :disappointed:


How do you sort by score?

If you do it in JavaScript then using AppleScriptObjC might be faster. As it seems it’s not possible to sort DEVONthink records directly by a record’s score property, I’ve first build a new AppleScript record with some properties, e.g. the name to display in Alfred, the score for sorting and the reference URL or uuid to get the choosen result afterwards from DEVONthink.

I just started to look into AppleScriptObjC, so there might be a faster way to do it … If someone (@houthakker) knows if/how DEVONthink records can be sorted without first building an AppleScript record please share.

-- Search across all databases and sort results

use AppleScript version "2.7"
use framework "Foundation"
use scripting additions

tell application id "DNtp"
	try
		set theQuery to "name:test"
		
		set theDatabases to databases
		set theResults_record to {}
		
		repeat with thisDatabase in theDatabases
			set theResults to search theQuery in thisDatabase
			repeat with thisResult in theResults
				set end of theResults_record to {name_:(name of thisResult), refurl_:(reference URL of thisResult), score_:(score of thisResult)}
			end repeat
		end repeat
		
		set theResults_record_sorted to my sort(theResults_record)
		
		-- [format results for Alfred] -- don't know what's needed for Alfred		
		
	on error error_message number error_number
		if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
		return
	end try
end tell

on sort(TheList)
	set anArray to current application's NSArray's arrayWithArray:TheList
	set theDesc to current application's NSSortDescriptor's sortDescriptorWithKey:"score_" ascending:false selector:"compare:"
	set newList to (anArray's sortedArrayUsingDescriptors:{theDesc}) as list
end sort

Forgive me I missed this – not always regularly checking here, as I always prefer JavaScript now, and there has sometimes seemed to be an active anxiety to preserve this as an AppleScript-only shop :slight_smile:

But since you ask …

JS native sort is very fast (competitive development between browser suppliers) so no need at all for the baroque syntax-mix and slightly tricky type conversions of learning to call ObjC methods through the ‘ASObjC’ foreign function interface.

TL;DR
(fuller code at foot of post)

const devon = Application('DEVONthink 3');
return sortOn(
    x => x.score()
)(
    devon.search(queryString)
);

where sortOn fetches record.score() only once for each record – a standard technique sometimes connected with Perl and the name of Randal Schwartz, but more helpfully called the decorate-sort-undecorate pattern.

// sortOn :: Ord b => (a -> b) -> [a] -> [a]
const sortOn = f =>
    // Equivalent to sortBy(comparing(f)), but with f(x)
    // evaluated only once for each x in xs.
    // ('Schwartzian' decorate-sort-undecorate).
    xs => xs.map(
        x => Tuple(f(x))(x)
    )
    .sort(comparing(fst))
    .map(snd);

A few points:

  • DEVONthink’s search method searches across all databases by default, (if we supply no explicit ‘in’ value) so we don’t even have to iterate over the collection of databases.
  • The ready-made sortOn, and related functions, can be found at: [JS Prelude](https://github.com/RobTrew/prelude-jxa)
  • For a descending sort, we can flip (a, b) comparison to (b, a). I’ve provided both a sortOn and a sortDescendingOn function below:

Full sample script, descending sort on `score`:
(() => {
    'use strict';

    // ---------------------- MAIN ----------------------
    const main = () => {
        const queryString = 'name:test';

        const devon = Application('DEVONthink 3');
        return sortDescendingOn(
            x => x.score()
        )(
            devon.search(queryString)
        )
        // If we wanted to list scores and names, 
        // otherwise, comment out the following lines:
        // .map(
        //     x => [
        //         x.score(), x.name()
        //     ]
        //     .map(str)
        //     .join(' -> '))
        // .join('\n');
    };


    // ------------------- JS PRELUDE -------------------
    // https://github.com/RobTrew/prelude-jxa

    // Tuple (,) :: a -> b -> (a, b)
    const Tuple = a =>
        b => ({
            type: 'Tuple',
            '0': a,
            '1': b,
            length: 2
        });


    // comparing :: (a -> b) -> (a -> a -> Ordering)
    const comparing = f =>
        (x, y) => {
            const
                a = f(x),
                b = f(y);
            return a < b ? -1 : (a > b ? 1 : 0);
        };

    // flip :: (a -> b -> c) -> b -> a -> c
    const flip = op =>
        // The binary function op with 
        // its arguments reversed.
        1 < op.length ? (
            (a, b) => op(b, a)
        ) : (x => y => op(y)(x));


    // fst :: (a, b) -> a
    const fst = tpl =>
        // First member of a pair.
        tpl[0];


    // snd :: (a, b) -> b
    const snd = tpl =>
        // Second member of a pair.
        tpl[1];


    // sortOn :: Ord b => (a -> b) -> [a] -> [a]
    const sortOn = f =>
        // Equivalent to sortBy(comparing(f)), but with f(x)
        // evaluated only once for each x in xs.
        // ('Schwartzian' decorate-sort-undecorate).
        xs => xs.map(
            x => Tuple(f(x))(x)
        )
        .sort(comparing(fst))
        .map(snd);


    // sortDescendingOn :: Ord b => (a -> b) -> [a] -> [a]
    const sortDescendingOn = f =>
        // Equivalent to sortBy(comparing(f)), but with f(x)
        // evaluated only once for each x in xs.
        // ('Schwartzian' decorate-sort-undecorate).
        xs => xs.map(
            x => Tuple(f(x))(x)
        )
        .sort(flip(comparing(fst)))
        .map(snd);


    // str :: a -> String
    const str = x =>
        Array.isArray(x) && x.every(
            v => ('string' === typeof v) && (1 === v.length)
        ) ? (
            x.join('')
        ) : x.toString();

    // MAIN --
    return main();
})();
2 Likes

Didn’t know that. Unfortunately I don’t know JavaScript, but your awesome answer sure will help the OP and other users :slight_smile:

Yes, of course. I’ve mixed up compare content (which I used the day before and which needs a repeat to get all databases) and search. I wonder why @syntagm uses a repeat … maybe to only get results of some databases but not all?

Do you know if it’s possible at all in AppleScriptObjC to sort DEVONthink records by any property without the “workaround” of first building an AppleScript record (the one I used in the previous post)?

Not that I’m aware of.

1 Like

Thanks, saved me a lot of time.

1 Like

Hah, you’re right. I didn’t even notice until now