Video Playback and Annotations

ChristinWhite · February 21, 2021, 12:36am

I’m moving away from using Eagle for video notes because there’s no linking nor an easy way to export the notes after you generate them and I stumbled on DEVONthink’s Insert Back Link in the Annotations and that it even does timestamps. This is nearly perfect!

What I really liked about Eagle is that I could hit the note hotkey and it would pause, pop up a note panel for me to write and then resume playback after I close the note UI. I’m trying to automate doing roughly the same thing in DEVONthink.

I don’t see a direct way to play and pause the video in DEVONthink’s dictionary but I’ve been able to get it to work with UI scripting, it’s brittle but I don’t change around my UI configurations much so other than needing to add an option for playback depending on whether it’s a document window or a viewer window that’s not the end of the world.

Since I can’t really dismiss the annotations panel I’ll just run it again to toggle playback. I generally avoid AppleScript unless I absolutely have to use it so forgive me if my inexperience with it is making this more complicated than it should be.

tell application "DEVONthink 3"
	think window 1
	
	# Check if item content type is a video
	if (exists MIME type of content record) and (MIME type of content record contains "video") then
		
		# Check if playing, if so pause and add annotation, if not hit play.
		tell application "System Events"
			tell its UI element "DEVONthink 3"
				if (get value of checkbox 1 of group 2 of splitter group 2 of splitter group 1 of window 1) is 1 then
					# Pause
					click checkbox 1 of group 2 of splitter group 2 of splitter group 1 of window 1
				else
					# Play
					click checkbox 1 of group 2 of splitter group 2 of splitter group 1 of window 1
				end if
			end tell
		end tell
	else
		display notification "You aren't playing a media item." subtitle "Playback Invalid"
	end if
end tell

Here’s where I get stuck and I have a couple of questions.

First, is there a reason to use recordType rather than MIME type as recommended here?
Since my annotations will ultimately end up in my Obsidian/Taio/DEVONthink indexed zettelkasten I’d like to use markdown and preferably format it slightly. Is there a way to programmatically create a Markdown annotation note without having to UI script menu button 3 of splitter group 1 of window 1 of application process "DEVONthink 3" then select New from Template/*template name*?
Once the annotation type is set I’d like to be able to insert a - to make it a list, it seems like trying to activate the annotations panel, insert the string *new line*- then hit ^⌘ L is probably not ideal. I could get the value of the panel, manipulate the string in the script and set it again may be better but not great. Is there a more elegant way to do all of this that I’m missing? Can I use the AppleScript Dictionary to find the annotation file and set it that way?
If I was to manipulate the annotation directly instead of using ^⌘ L it looks like the recommended way to get the current timestamp would be to grab current movie frame for the current window and generate it from that but that’s returning some kind of encoded string I don’t recognize or know what to do with, is current time the better property?

CleanShot 2021-02-20 at 19.22.26@2x838×664 72.7 KB
This isn’t actually automation related but it doesn’t seem like there’s a way to change the playback speed unless I’m missing something, though someone mentioned doing it with the mouse here. I’m not just missing some way am I? This is something I can live without but it would be nice.

Thanks for the help, I absolutely love DEVONthink and this is so close to the perfect tool I’ve been looking for to do video annotations. I am also looking at external video players that let me get the current frame via AppleScript or copy a timestamp that I could just paste into DEVONthink too but so far I haven’t found one with

ChristinWhite · February 21, 2021, 1:19am

Doing some more reading and searching, it seems like it would be best to not UI script the annotation but instead create or update the annotation file programmatically, looks like there are plenty of examples for figuring out how to do this so I guess my remaining questions are 1., 4., & 5.

Apologies for still figuring things out, since most of what I do has been webarchives and bookmarks in DEVONthink I’ve really never touched the annotations system before and a little prerequisite knowledge would probably have been more helpful before asking questions.

BLUEFROG · February 21, 2021, 3:04am

No worries and yes, UI scripting us fragile and not advocated by us in almost every case.

Playback speed can’t be changed in DEVONthink.
If you look at the AppleScript dictionary for DEVONthink, you’ll see a record has an annotation file property.

I’m in bed with a bad head cold so I’ll try to look at this tomorrow. (Yeah and still working. Bad work/life balance. I know. I know. )

ChristinWhite · February 21, 2021, 3:17am

Thanks, that was the key attribute I missed earlier, that’s really helpful!

Get feeling better!

BLUEFROG · February 21, 2021, 3:22am

You’re welcome and thanks

joshgibson · February 23, 2021, 7:10am

If you’re willing, I’d love to hear about the solution you landed on while researching this. I, too, just discovered the back link and would love to take better advantage of it when annotating files.

ChristinWhite · February 24, 2021, 2:19am

Absolutely, I’ll post whatever I end up with.

ChristinWhite · March 3, 2021, 9:31pm

So I haven’t yet worked on the annotation component or even playback in DEVONthink (I will though) but I did discover that VLC has AppleScript support and includes a current time properly (to the second). So I created a script that will take the current time from VLC, get the Reference URL for the current file in DEVONthink and copy a markdown link with it and the ?time parameter to the clipboard.

I’ve got a bunch more I want to do, adapt a rich text version. Do a pair of matching scripts for QuickTime (and see if any other players have the necessary AppleScript options). I’d also like to do a more complete annotations workflow and probably adapt them for Keyboard Maestro as well but I thought you might like to see or be able to adapt what I have now.

use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions

# # Copy Markdown Link for DEVONthink Video Open in VLC at Current Time
#
# Since DEVONthink's video player is a bit limited (it doesn't support multiple playback speeds for instance) I wanted to be able to use a more flexible player but link back to the video file in DEVONthink at the current timecode. 
#
# One minor issue, VLC only lets you get the current time to the second, not the exact frame or decimal.
# 
# This script will copy that URL as a markdown link in the format: [13:10:72](x-devonthink-item://45151EC1-8A15-4109-AFE9-1B4EA8C6DE2B?time=352)
#
# Created by [Christin White](http://christindwhite.com)
#

# # Set Options
# 
# - `DatabaseName`     *text*: The name for the database to use, if a valid database is not found the current database will be used.
# - `PausePlayback` *boolean*: If true, pause VLC when script is run.
#
set DatabaseName to "Collection"
set PausePlayback to true

try
	# Get Reference URL from DEVONthink and Assemble Link
	set TheVideo to GetVideoFromVLC(PausePlayback)
	
	# Get Reference URL
	set TheReferenceURL to GetReferenceURLFromPath(TheVideoPath of TheVideo, DatabaseName)
	
	# Assemble Link
	set TimeCodeURL to TheReferenceURL & "?time=" & (TheCurrentTime of TheVideo)
	set TimeCode to FormatTimeCode(TheCurrentTime of TheVideo)
	set MarkdownLink to "[" & TimeCode & "](" & TimeCodeURL & ")"
	
	# Copy to Clipboard
	set the clipboard to MarkdownLink
on error ErrorMessage number ErrorNumber
	if the ErrorNumber is not -128 then display alert "DEVONthink" message ErrorMessage as warning
end try


# # Functions
#
# These will be moved to a script library.
#


# # AddLeadingZeros
#
# Add leading zeros to a number to reach the specified number of digits. Based on [Apple's handler](https://developer.apple.com/library/archive/documentation/LanguagesUtilities/Conceptual/MacAutomationScriptingGuide/ManipulateNumbers.html).
#
# Parameters:
# - `TheNumber`         *number* : The number to add leading zeros to.
# - `TheNumberOfDigits` *integer*: The total number of digits.
#
# Returns:
# - *text*: TheNumber with the specified number of leading zeros.
#
on AddLeadingZeros(TheNumber, TheNumberOfDigits)
	set IsNegative to TheNumber is less than 0
	
	# Determine the maximum number from the number of digits.
	set TheNumberOfDigits to (TheNumberOfDigits - 1)
	set TheThreshold to (10 ^ TheNumberOfDigits) as integer
	
	if TheNumber is less than TheThreshold then
		# If the number is negative, convert it to positive
		if IsNegative = true then set TheNumber to -TheNumber
		
		set TheLeadingZeros to ""
		set TheDigitCount to length of ((TheNumber div 1) as text)
		set TheCharacterCount to (TheNumberOfDigits + 1) - TheDigitCount
		repeat TheCharacterCount times
			set TheLeadingZeros to (TheLeadingZeros & "0") as text
		end repeat
		
		# Make the number negative, if it was previously negative
		if IsNegative = true then set TheLeadingZeros to "-" & TheLeadingZeros
		
		return (TheLeadingZeros & (TheNumber as text)) as text
		# If the number is greater than or equal to the maximum number of digits
	else
		# Return the original number
		return TheNumber as text
	end if
end AddLeadingZeros


# # FormatTimeCode
#
# Converts seconds into a timecode format matching `00:00:00`
#
# Parameters:
# - InSeconds *number*: Number of seconds to set timecode to. Accepts float but is converted to Integer
#
# Returns:
# - *text*: Timecode
#
on FormatTimeCode(InSeconds)
	set InSeconds to InSeconds as integer
	# Calculate Hours
	set CalculatedHours to (InSeconds div hours)
	
	# Calculate Minutes
	set RemainderSeconds to (InSeconds mod hours)
	set CalculatedMinutes to (RemainderSeconds div minutes)
	
	# Calculate Seconds
	set CalculatedSeconds to (RemainderSeconds mod minutes)
	
	# Convert to Two Digit Strings
	set CalculatedHours to AddLeadingZeros(CalculatedHours, 2)
	set CalculatedMinutes to AddLeadingZeros(CalculatedMinutes, 2)
	set CalculatedSeconds to AddLeadingZeros(CalculatedSeconds, 2)
	
	set TimeCode to CalculatedHours & ":" & CalculatedMinutes & ":" & CalculatedSeconds as text
	return TimeCode
end FormatTimeCode


# # GetReferenceURLFromPath
#
# Get the reference URL for a file in a specified or active database.
#
# Parameters:
# - `TheFilepath`  *text*: The path to the file.
# - `DatabaseName` *text*: That database to look in.
#
# Returns:
# - *text*: The Reference URL.
#
on GetReferenceURLFromPath(TheFilePath, DatabaseName)
	tell application id "DNtp"
		# Check if the database is valid, if not use the active database.
		if exists database DatabaseName then
			set TheRecordList to lookup records with path TheFilePath in database DatabaseName
		else
			display notification "The specified database wasn't found, trying the active database instead" subtitle "Invalid Database"
			set TheRecordList to lookup records with path TheFilePath
		end if
		
		# Make sure we found a matching record.
		if (count of TheRecordList) is greater than 0 then
			# A list could return more than one record such as if a file is indexed to more than one group. Use the first item but notify the user.
			if (count of TheRecordList) is greater than 1 then
				display notification "More than one record was found, using the first record" subtitle "Multiple Records"
			end if
			set TheReferenceURL to reference URL of item 1 of TheRecordList
			return TheReferenceURL
		else
			error "File not found in database."
		end if
	end tell
end GetReferenceURLFromPath


# # GetVideoFromVLC
#
# Gets metadata about the current video open in VLC
#
# Parameters:
# - `PausePlayback` *boolean*: Pause video when called.
#
# Returns:
# - *list* (indexed)
#   - `TheVideoPath`     *text*: The path for current video.
#   - `TheCurrentTime` *number*: The current second of playback.
on GetVideoFromVLC(PausePlayback)
	tell application "VLC"
		if (exists path of current item) then
			set TheVideoPath to path of current item
			set TheCurrentTime to current time
		else
			error "VLC does not have an open video."
		end if
		
		if PausePlayback and playing then
			play
		end if
		
		return {TheVideoPath:TheVideoPath, TheCurrentTime:TheCurrentTime}
	end tell
end GetVideoFromVLC

(My website is in-progress and not actually there yet. I’m also a monster who Pascal cases, soft-wraps and prefers tabs.)

Edit: Converted VLC related functions to a handler. Fixed hard-coded test value.

ChristinWhite · March 3, 2021, 5:16pm

One other note I forgot to mention, DEVONthink will use the format [00\:00] for markdown annotations, that works but seems unnecessary, in testing [00:00] worked fine in DEVONthink, Obsidian, iA Writer and Drafts while [00\:00] correctly interpreted the backslash as an escape character in DEVONthink, Obsidian and Drafts but iA Writer doesn’t.

joshgibson · March 3, 2021, 10:44pm

Wow, this is amazing work! Unfortunately, I’m not much of a developer, but I love the direction of what you’re doing. I am curious though, why would you prefer this method over DEVONthink’s built-in “Insert Back Link” option in the given file’s Annotations section in the Inspector?
CleanShot 2021-03-03 at 15.43.22

ChristinWhite · March 4, 2021, 5:26am

A couple of reasons:

As I mentioned, I wanted to use an external player with more options such as different playback speeds.
The annotation panel in the sidebar is really small amd it’s finicky making sure it’s going to be in the format I want (Markdown, not rich text). Right now I haven’t fully implemented a create annotations file (if necessary and insert back link yet but it’s on the list.
This lets me use other apps to take notes such as Obsidian for Markdown notes or if it’s a really complex video or set of videos I may create my annotations in MindNode instead and then export the outline back to Markdown. This depends on getting a rich text link though for MindNode or other apps that accept RTF instead of Markdown, that’s what I’m working on now.
Perhaps most importantly, selecting the options in that right click annotations menu and working with the annotations panel programmatically isn’t possible without using UI scripting which is inadvisable because all kinds of slightly different change to the app stare and updates can brake it, it’s very fragile.

pete31 · March 4, 2021, 6:01am

Not sure I understood what you want to do but maybe the handler in this post could be useful. It copies a Markdown and a RTF link. Depending on the receiving app’s preferred clipboard type one of both will be inserted.

joshgibson · March 4, 2021, 2:29pm

Awesome, thanks for all that info! I’m excited to see what you end up with for all of these needs. Thanks for sharing.

ChristinWhite · March 4, 2021, 5:31pm

That’s a really interesting solution, I’d been playing with textutil as the solution but wasn’t sure how to get it where I was going. I love the idea of having both formats on the clipboard though, that’s perfect!

The only thing I was really hoping to figure out is a way to copy the rich text with the link set correctly but with no font/styling so that it just matches the format of whatever I paste it into. Something along the lines of this.

I’m not sure whether it’s even possible to have rich text without styling though and if it is I’m not sure it’s something I could use textutil to generate.

I’m wondering if it might be possible using the other technique mentioned in the thread your handler linked to using ASObjC, do you have any thoughts on that?

pete31 · March 4, 2021, 9:44pm

No idea whether that’s possible but I don’t think that one can create an attributed string without style attributes. If we don’t set them there’s probably a default used.

This script copies both types, RTF and Markdown, however I just realized that it doesn’t work with e.g. DEVONthink’s “Paste and match style” menu. No idea why, might be that it’s necessary to also add another type to the clipboard.

Here’s what happens, first line using “Paste”, second “Paste and match style”

But I overread your hint with Shane’s HTML technique, I’ll try that later.

-- Copy RTF and Markdown link

use AppleScript version "2.4"
use framework "Foundation"
use scripting additions

tell application id "DNtp"
	try
		set theRecords to selected records
		if theRecords = {} or (count theRecords) > 1 then error "Please select one record"
		set theRecord to item 1 of theRecords
		set theName to name of theRecord
		set theURL to reference URL of theRecord
		
	on error error_message number error_number
		if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
		return
	end try
end tell

set theTypeArray to current application's NSMutableArray's arrayWithArray:{}
set theDataArray to current application's NSMutableArray's arrayWithArray:{}

set theAttributtedString to current application's NSMutableAttributedString's alloc()'s initWithString:((current application's NSString's stringWithString:theName))
theAttributtedString's addAttribute:(current application's NSLinkAttributeName) value:theURL range:(current application's NSMakeRange(0, theAttributtedString's |length|()))
set theType to current application's NSPasteboardTypeRTF
set theData to theAttributtedString's pasteboardPropertyListForType:theType
theTypeArray's addObject:theType
theDataArray's addObject:theData

set theMarkdownString to (current application's NSString's stringWithFormat_("[%@](%@)", theName, theURL))
set theType to current application's NSPasteboardTypeString
set theData to theMarkdownString's dataUsingEncoding:(current application's NSUTF8StringEncoding)
theTypeArray's addObject:theType
theDataArray's addObject:theData

set thePasteboard to current application's NSPasteboard's generalPasteboard()
thePasteboard's clearContents()

repeat with i from 0 to (theDataArray's |count|()) - 1
	(thePasteboard's setData:(theDataArray's objectAtIndex:i) forType:(theTypeArray's objectAtIndex:i))
end repeat

ChristinWhite · March 4, 2021, 10:11pm

I’m not sure about that either, I actually just started a new thread over at Late Night Software’s forum to see if anyone else may have more insight into this, there seem to be a number of ways to get a rich text link but all of them include the styling, my guess is that I’m chasing something that’s not valid RTF but it sure would be nice to paste a link into a Rich Text field and not have to manually reset the styling or paste it as plaintext and add the link myself.

I’m not too concerned about getting it to paste correctly into DEVONthink itself, just about anything I put in DEVONthink is going to be Markdown, really, MindNode is my personal goal here, it’s smart enough to reformat something pasted in as a node but not in the node’s notes.

I’ll take a look at your script there in a bit.

pete31 · March 4, 2021, 10:20pm

I now read your thread, you could try to get data from your RTF string with another handler from Shane which I posted here some days ago.

pete31 · March 4, 2021, 11:42pm

Ahmm, the same happens with a original DEVONthink link: “Paste” gives a properly named RTF link, “Paste and match style” gives a strange hybrid. Same in Tinderbox, so what you want won’t work, I’m afraid.

ChristinWhite · March 5, 2021, 12:19am

Hmm, interesting, that does take my string and encode it as rdat data okay but I’m not sure how to modify it to encode it as RTF.

Switching it to RTF this way

**set** theCode **to** *current application's* NSHFSTypeCodeFromFileType("'RTF'")

Doesn’t do it (rtf also fails). I’m definitely not understanding what’s actually going on with this stuff, it seems like I need to spend some time learning Apple’s Mac APIs and AppleScriptObjC — just a slightly larger scope than I was hoping to need for this project .

Thank you for all your help and advise, I really appreciate it!

pete31 · March 5, 2021, 12:51am

“Some time” AppleScriptObjC is the greatest rabbit hole ever and there’s almost no documentation. You’ll find answers for Objective-C but once you think you’ve figured how something works BOOOM!, sorry, you can’t do that in AppleScriptObjC … If you get Shane’s book start with chapter 29. “Out of Bounds” - and keep it close