Use of DT for Research and Note Linking - What works for you?

Hi Kevin,

As a medical researcher I am in search of a solution of the same problem. Can you elaborate a bit more on your process? What script do you use to tidy up the notes? How do you tidy them? Are you willing to share?

All the best,

Johannes

Johannes

Sure. I can write this up in more detail at the weekend (and don’t hesitate to bump this thread if I haven’t). My AppleScript skills are very amateur but I’ll share what I have so far.

Kind regards

Kevin

Cheers, Kevin.

Looking forward to it! Thanks in advance and happy new year!

With apologies for the slight delay, here you go.

Background

As background, this workflow assumes:

  • that original materials are stored in a (for example) “/Sources” group and its sub-groups, for example:
    /Sources/Articles/[ArticleName]/[ArticleName].pdf

This means that there is one canonical place to look for any article.

  • that feeding DT with short, one or two paragraph, extracts is useful (eg for the see also / auto-classify functionality, and more generally for classifying extracts into multiple groups )

  • that these extracts are stored in:
    /Sources/Articles/[ArticleName]/Notes/

  • that each of these extracts is then replicated to one or more groups in a separate tree that is organised by themes / concepts / ideas.

Also note that this is currently only for extracting original materials from PDFs, rather than directly adding my own comments to PDFs. (It would be relatively easily extendable to that, but that’s now how I prefer to work, at least for now.)

PDFs > Bookends

If a citation manager isn’t important to your work, then this step is not necessary and doesn’t affect the rest of this workflow.

Other citation managers are available but Bookends is a good, dedicated MacOS, app, with (now) very good AppleScript support.

Highlighting PDFs

In any PDF editor that highlights.

I use either native Bookends support - which has the advantage of being able to highlight and sync from an iPad - or Highlights.app.

Different colours of highlights can have different meanings which can be useful once you start to analyse your notes in DT.

For me:

Yellow: interesting extract
Blue: interesting extract that I think I disagree with
Green: To do (typically a reference to another article that I want to read)

(Note: Again, I don’t tend to add my own comments / annotations at this stage.)

Bookends reference > DT3 including custom metadata

I then use a lightly modified version of a script I found here to create a DT3 record from a Bookends reference.

DT3 structure

I have a standard DT3 structure for articles and their notes. All are filed under:
/Sources/Articles/

With the structure:
/Sources/Articles/[ArticleName]/
/Sources/Articles/[ArticleName]/ArticleName.pdf
/Sources/Articles/[ArticleName]/Notes/ <-- Highlighted passages from the PDF live here

/Sources/Articles/[ArticleName]/ArticleName.pdf has several pieces of DT3 custom metadata such as:
CiteKey (the Bookends-generated BiBTex compatible citekey)
StartPage (eg 90 if the article runs from pages 90 - 120)
and so on.

I don’t use DT3 as a citation manager, but having the citekey in the DT3 record means that if I want to use that record in what I write, I can put the citekey in a footnote directly, without having to go back to the citation manager.

Export highlighted passages

First, set the Highlights app export settings in preferences:

  • Color: give the highlight colours appropriate names (for me: Extract, Query, ToDo);
  • Customize the annotation header - I use Color Category and Page Number;
  • Export - MarkDown and Export as Folder of Files

Open the highlighted PDF in Highlights.app then:

File > Export to > Devon-think

This creates a subfolder in the DT inbox with each highlighted passage as a separate note.

Manually move highlighted passages from subfolder in inbox to:

/Sources/Articles]/[ArticleName]/Notes/

“Tidy” the highlighted passages in “/Notes/”

In DT3, select:

/Sources/[ArticleName]/ArticleName.pdf

Then run this script. The script does the following:

  • it takes every note in the /Notes/ subfolder
  • gives each note some of the same metadata as the parent PDF, in particular the CiteKey
  • gives each note a metadata record type of “Extract”, “Query” or “To Do”.
  • removes unnecessary information from the note, leaving only the quoted text
  • If the record type is “To Do” then create an Omnifocus task with a rough and ready attempt at a Google Scholar URL.

The script is not neat and tidy, and has only minimal comments/documentation. Tidying it up is only my list of things to do… sometime…

use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions



set theTaskList to {}

tell application id "DNtp"
	
	-- the original article / case should be selected
	
	set theOriginalRecord to the selection
	
	-- Get the metadata of the original article
	
	set theMetaData to the custom meta data of item 1 of theOriginalRecord
	
	set theAuthor to mdauthor of theMetaData
	set theBookendsURL to mdbookendsurl of theMetaData
	set theCiteKey to mdcitekey of theMetaData
	set theDOI to mddoi of theMetaData
	set theOffset to mdoffset of theMetaData
	set theStartPage to mdstartpage of theMetaData
	set theTitle to mdtitle of theMetaData
	set theYear to mdyear of theMetaData
	set theDTPReference to the reference URL of item 1 of theOriginalRecord
	
	set thePath to the location of item 1 of theOriginalRecord
	
	set theNotesPath to thePath & "Notes/"
	
	set theNotesGroup to get record at theNotesPath
	
	set theNotes to the children of theNotesGroup
	
	set theStartPage to theStartPage - 1
	repeat with theNote in theNotes
		
		set theContent to the rich text of theNote
		
		set theParagraphs to the paragraphs of theContent
		
		set theParagraph to the fifth paragraph of theContent -- 5th para begins with "Extract" "Query" "To Do"
		
		set theType to the characters 1 thru 4 of theParagraph as string
		
		if theType is "Task" then add custom meta data "To Do" for "recordtype" to theNote
		if theType is "Note" then add custom meta data "Extract" for "recordtype" to theNote
		if theType is "Quer" then add custom meta data "Query" for "recordtype" to theNote
		
		set thePage to paragraph 4 of theContent
		set thePage to characters 6 thru -1 of thePage as string
		
		-- Create DTP url to precise page. Need to do this before we mess up the page with the offset etc
		set theDTPPageURL to theDTPReference & "?page=" & (thePage - 1)
		
		set theExtract to paragraphs 6 thru -1 of theContent as string
		
		
		set thePage to thePage + theOffset
		set theExtractPage to theStartPage + thePage
		
		
		
		--
		add custom meta data theAuthor for "author" to theNote
		add custom meta data theTitle for "title" to theNote
		add custom meta data theYear for "year" to theNote
		
		add custom meta data theExtractPage for "startpage" to theNote
		add custom meta data theOffset for "offset" to 0
		
		add custom meta data theCiteKey for "citekey" to theNote
		
		add custom meta data theBookendsURL for "bookendsurl" to theNote
		add custom meta data theDOI for "doi" to theNote
		add custom meta data theDTPPageURL for "devonthinkurl" to theNote
		
		
		if theType is "Note" then
			set theRevisedNote to return & return & "Extract: " & return & return & theExtract & return & return & "Comment:" & return & return
		else if theType is "Quer" then
			set theRevisedNote to return & return & "Extract: " & return & return & theExtract & return & return & "Query:" & return & return
		else if theType is "Task" then
			set theRevisedNote to return & return & "To Do: " & return & return & theExtract & return & return
			set theTaskList to theTaskList & theExtract
		end if
		
		set the plain text of theNote to theRevisedNote
		
		set theEnd to the length of theExtract
		if theEnd is greater than 100 then set theEnd to 100
		set theNewName to characters 1 thru theEnd of theExtract as string
		set theNewName to theCiteKey & " p." & theExtractPage & ": " & theNewName
		set the name of theNote to theNewName
		
		
	end repeat
	
end tell

tell application "OmniFocus"
	
	repeat with theTask in theTaskList
		
		set theQuery to "https://scholar.google.co.uk/scholar?hl=en&as_sdt=0%2C5&q=" & theTask
		
		set the clipboard to theQuery
		
		tell application "TextSoap"
			
			cleanClipboard with "Clean text for use in websearch"
			
		end tell
		
		set theQuery to the clipboard
		
		set the clipboard to theTask
		
		tell application "TextSoap"
			
			cleanClipboard with "Extract URLs by replacing"
			
		end tell
		
		set theURL to the clipboard
		
		set theQuery to theQuery & return & theURL
		
		set theTask to "Find: " & theTask
		
		tell its document "OmniFocus"
			tell its flattened project "[PROJECT NAME]"
				
				tell its task "Articles to find and review"
					
					make new task with properties {name:theTask, note:theQuery}
					
				end tell
				
			end tell
			
		end tell
		
	end repeat
	
end tell

Classify the notes

I then review the notes and replicate them into a separate folder tree of “Issues”.

Hope the above helps and happy to answer any questions.

And Happy New Year to all.

K.

3 Likes

wow. that is abslutely excellent and inspiring. will need a bit of time to work through that. cheers and happy new year!

It looks more intimidating than it is, really ! And once it’s set up, actually using it day to day is very simple (which was important to me).

Happy to discuss and help - as best I can - tweak this to suit your needs.