How to run Javascript on Markdown "summarize highlights" output?

Great. Thanks for this mini tutorial!

You’re welcome. I hope it learns you something :wink: :stuck_out_tongue:

That’s true :slight_smile:

1 Like

Very much agree on avoiding Regex where possible ( not quite write-only, but getting there, and they add a second layer of complexity ).

But the ‘readability’ of AppleScript is not a function of its inherent properties – it just expresses how much time we’ve personally spent with it. (I can remember finding it very opaque, and hard to write, for quite a while, despite a 20 year background at the time, in languages like C, Pascal, Prolog and Lisp).

We probably shouldn’t kid ourselves, or ask others to drink the ‘English-like’ Kool-Aid, on that one :slight_smile:

Now that AppleScript’s development is sunsetted, and iOS is a bigger platform than macOS, perhaps it’s getting rather late in the day to push users back from JavaScript towards AppleScript ?

(I understand the impulse, we are all attached to the tools which happen to be familiar to us, but that doesn’t automatically make them the best choice – or the most responsible recommendation – for beginners).

1 Like

(I understand the impulse, we are all attached to the tools which happen to be familiar to us, but that doesn’t automatically make them the best choice – or the most responsible recommendation – for beginners).

JavaScript is familiar to you but terse and inexplicable to most laypeople.
And I would not advocate people move into JXA as a beginner, especially since it’s very poorly documented and makes a ton of assumptions about a person’s prior knowledge.

There are quite a few people who have taken a shine to AppleScript, just from posts on these forums, our built-in Help, and the scripts we’ve included with DEVONthink. And the fact that Script Debugger continues to sell and flourish speaks to the general health and interest in AppleScript as a primary automation language for Macs.

Interestingly, you are another example of people coming from higher-level languages who said they struggled with AppleScript’s syntax.

I, having no CS degree, taught myself out of necessity in the graphic arts/printing field, then later professionally AppleScripting in the corporate world (no, not a typo or exaggeration) and have always found AppleScript easy to grasp, easy to teach, and charming (despite it’s occasional idiosyncracies).

1 Like

I disagree (and incidentally, neither of us has any formal training in CS), I was very familiar with AppleScript long before I knew any JS at all.

I now find it:

  • Faster and easier to get things working in JS than in AppleScript,
  • much easier than AppleScript to get help with.

“very poorly documented”

Simply not true.

“it …makes a ton of assumptions about a person’s prior knowledge”

Not clear what the “it” refers to in that sentence … JS ? sorry – not clear what you have in mind …

people coming from higher-level languages who said they struggled with AppleScript’s syntax

I came from both lower level languages and higher level languages, and still found AppleScript precious and opaque – in particular, much harder to write than to read.

have “always found” AppleScript … XYZ

I think that’s the problem – long exposure to a language is very easily confused with the properties of the language itself. English speakers think English is easy and charming. Adult learners of it disagree.

I now find AppleScript perfectly legible and writable (I’ve contributed a fair amount of open source AppleScript in the past), but I don’t kid myself that it’s AppleScript itself which “is” easy. Like anything, it can become easier. (If you invest a very significant amount of time).

Meanwhile, even on macOS:

  • The Omni Applications have moved scripting development decisively away from AppleScript to JavaScript. See the discussion in their 2020 Road Map.

    • In particular where they point out that:

… AppleScript had some big caveats: it was generally easy to read, but it was a fairly esoteric language that many developers found less easy to write. It was able to accomplish some amazing things, much faster and more accurately than working by hand—but it was generally slow enough that you could watch it work, and if it had a lot of work to do you could be watching and waiting for a while. And it was only available for the Mac platform—which was totally fine in the ’00s when that was the only place our apps ran, but was a lot less useful in the ’10s when our customers were increasingly spending their time on mobile devices.

Perhaps we need to think carefully before telling customers that AppleScript “is” easy, as if learning to use it confidently and productively (without wasting hours of frustration) doesn’t need to represent a major investment of time.

The reality is that all languages require time to learn, and we have no evidence at all to show that JavaScript would require a different amount of time from AppleScript.

What we do have evidence of, however, is that JS is bound to be a better investment of that time.

  • AppleScript has a small and shrinking pool of applications, and no prospect of further development.
  • iOS is JavaScript-only (there can and will never be any AppleScript on it)
  • macOS has no AppleScript-only applications, but it does have JavaScript-only applications.
  • OmniGroup is moving away from AppleScript development to JS scripting development.
  • Web pages run on JavaScript – if you’re only going to learn one language, JS is the language to learn, and it’s the one with the richest range of learning materials.

I do understand the reflex of defending a technology that one has personally invested time in, and the vertiginous avoidance of stepping into a new one – I felt it myself at the transition from familiar use of AppleScript to first steps in JavaScript (which I originally made simply in order to be able to write scripts for TaskPaper).

But I think it’s only responsible to allow oneself to pause and reflect that the picture is now very different for beginners.

Compared to other languages it definitely is - just try to learn Czech :smiley:

1 Like

( Tho I understand CZ was a bit divided on that one … )

Having spent the best part of ten years teaching English to adult learners in another country, I have my doubts about that proposition (though I concede that Czech is not easy – I have friends who live there). The fact that huge numbers of native English speakers appear to be incapable of speaking or writing the language correctly only adds weight to the idea that it is not an easy language. But I have taken us off topic …

I’m tempted to dig into the 800+ pages of Sal Soghoian’s book on Apple Script, but I’m guessing someone in this thread could tweak BLUEFROG’s script for me so that I do something very similar with my Kindle highlights. I’d like to turn this…

Yellow highlight | Location: 1,121
For a time, we had come to believe that civilization moved in the other direction—making the impossible first possible and then stable and routine. With climate change, we are moving instead toward nature, and chaos, into a new realm unbounded by the analogy of any human experience.

Yellow highlight | Location: 1,131
On local golf courses, the West Coast’s wealthy still showed up for their tee times, swinging their clubs just yards from blazing fires in photographs that could not have been more perfectly staged to skewer the country’s indifferent plutocracy. The following year, Americans watched the Kardashians evacuate via Instagram stories, then read about the private firefighting forces they employed, the rest of the state reliant on conscripted convicts earning as little as a dollar a day.

…into this…

For a time, we had come to believe that civilization moved in the other direction—making the impossible first possible and then stable and routine. With climate change, we are moving instead toward nature, and chaos, into a new realm unbounded by the analogy of any human experience. (Location: 1,121)

On local golf courses, the West Coast’s wealthy still showed up for their tee times, swinging their clubs just yards from blazing fires in photographs that could not have been more perfectly staged to skewer the country’s indifferent plutocracy. The following year, Americans watched the Kardashians evacuate via Instagram stories, then read about the private firefighting forces they employed, the rest of the state reliant on conscripted convicts earning as little as a dollar a day. (Location: 1,131)

In other words, I’d like to
1. get rid of “Yellow highlight |” at the top of each passage,
2. create parentheses at the end of each passage,
3. relocate “Location: xxxx” from the top of each passage into the parentheses at the end of the passage.

I don’t need this to happen within DEVONthink, though no problem if that’s what you think is easiest. I was thinking I could copy and paste my highlights into a regular text file and then run the script on each file individually. Oh, and I have Keyboard Maestro, so if you think that’d be even better to use, please oh please send some instructions my way.

Thank you in advance for whatever help you can provide!

@BLUEFROG By chance, do you have a nice learning example for formatting RTF files?

I’m looking for an easy way to remove the indents from the Summarize Highlights RTF output (i.e., so that all the text is left-aligned and indent free). When compared to your markdown examples above, it would look more like the version on the right-hand side. However, I don’t really care if the links are moved to parentheses after the quote; they can stay above the text, like the default output, or move - either is fine. I just don’t like the indents because they’re more difficult to fix manually afterward.

Thanks for your help!

1 Like

Hey Jim, @BLUEFROG, Could you be so kind as to modify this code so that:

Instead of Page, it writes the name of the file in the paranthesis in the following format(semicolon)?

Right Now:

How I would love it to be:

1 Like

Coming a bit late to the party, but here’s a version in JavaScript:

(() => {
	const app = Application("DEVONthink 3");
	/* get all MD records from current selection */
	const records = app.selectedRecords.whose({ _match: [ObjectSpecifier().type, "markdown"] })();
	/* Loop over records */
	records.forEach(r => {
	    /* Split plaintext in paragraphs */
		const txtArray = r.plainText().split("\n");
		const newText = [];
		let heading = "";
		/* Loop over paragraphs */
		txtArray.forEach(txt => {
		    /* Check for level 2 heading */
			let isHeading = txt.match(/^##\s+(.*)/);
			/* Check for bullet list block */
			let isList = txt.match(/^\*\s+(.*)/);
			if (isHeading) {
			    /* Save heading text in temp variable */
				heading = isHeading[1];
			} else if (isList) {
			    /* save new paragraph: 
				  Bullet list block without the bullet and
				  corresponding heading appended in parenthesis 
				 */
				newText.push(`${isList[1]} (${heading})`)
			} else {
			    /* If neither bullet nor heading, save the paragraph as is */
				newText.push(txt);
			}
		})
		r.plainText = newText.join('\n');
	})
})()

If someone needs not the heading but only a part of it, this should be done in the if (isHeading) part. Just set heading to whatever you want to appear in the parenthesis after the (ex-)bullet blocks.

2 Likes

Thanks for the reply @chrillek.

How do we use JavaScript? Not the way Bluefrog explains above for the Applescript i guess. Tried that. Didn’t work.

Copy/paste the code into script editor and save it from there in DT‘s script folder. Then you can run it from the script menu.

1 Like

Oh it did the trick! Thanks a lot!

I managed to solve this both in Apple Script and Javascript.
Now I get the Heading 1 in the parentheses.

This is the Applescript version:

on performSmartRule(theRecords)
	tell application id "DNtp"
		repeat with theRecord in theRecords
			if (type of theRecord as string) = "markdown" then
				-- Only process Markdown docs
				
				set newText to {}
				-- Set up an empty list to push the paragraphs into)
				
				set currentText to (plain text of theRecord)
				-- Get the current text of the document
				
				set titleText to (characters 3 thru -1 of paragraph 1 of currentText as string) -- Cache all but the control characters in a variable     
				
				repeat with thisLine in (paragraphs of currentText)
					-- Loop through the document, paragraph by paragraph
					
					if thisLine begins with "##" then
						-- If this is an H2 page link
						set cachedText to (characters 4 thru -1 of thisLine as string) -- Cache all but the control characters in a variable
						
					else if thisLine begins with "* " then
						-- If it's just a bulleted paragraph
						
						copy ((characters 3 thru -1 of thisLine as string) & " (" & (titleText) & ": " & (cachedText) & ") " & return) to end of newText
						-- Get all but the control characters and concatenate the cachedText.
						-- Then push it to the newText list
					else
						copy thisLine & return to end of newText
						-- Anything else, including the H1 line at the top, just push into the newText list
					end if
				end repeat
				
				set plain text of theRecord to (newText as string)
				-- Set the text of the document to the new text
				
			end if
		end repeat
	end tell
end performSmartRule

And this is the Javascript version:

(() => {
	const app = Application("DEVONthink 3");
	/* get all MD records from current selection */
	const records = app.selectedRecords.whose({ _match: [ObjectSpecifier().type, "markdown"] })();
	/* Loop over records */
	records.forEach(r => {
	    /* Split plaintext in paragraphs */
		const txtArray = r.plainText().split("\n");
		const newText = [];
		let heading = "";
		let heading1 = "";

		txtArray.every(txt => {
			/* Loop over paragraphs */
			let isHeading1 = txt.match(/^#\s+(.*)/);
			if (isHeading1) {
				heading1 = isHeading1[1];
				return false;
			}
			return true;
		})
		
		txtArray.forEach(txt => {
			/* Check for level 2 heading */
			let isHeading = txt.match(/^##\s+(.*)/);
			/* Check fior bullet list block */
			let isList = txt.match(/^\*\s+(.*)/);
			if (isHeading) {
			    /* Save heading text in temp variable */
				heading = isHeading[1];
			} else if (isList) {
			    /* save new paragraph: 
				  Bullet list block without the bullet and
				  corresponding heading appended in parenthesis 
				 */
				newText.push(`${isList[1]} (${heading1}: ${heading})`)
			} else {
			    /* If neither bullet nor heading, save the paragraph as is */
				newText.push(txt);
			}
		})
		r.plainText = newText.join('\n');
	})
})()

The AppleScript is for use in a smart rule, the JavaScript is not (yet). It’s not too difficult to make it work as script in a smart rule, too.

In the JavaScript, there’s no need for let. unless you modify the variable later on. Otherwise, use const. Also, you seem to have two loops over the same array – why?

If anyone does want to tidy the JS up a little and adapt it for a smart rule, then a comment I might add to @chrillek’s good points is that the idiom:

  • declare a []
  • .forEach.push

may be making life a bit complicated, especially for new users, when you can just define the new array that you want directly, in terms of .map or, even more flexibly: .flatMap, or .reduce

1 Like