Exporting from Day One and importing into DEVONthink Pro: the procedure in detail

[Stephen_C]

Introduction

This thread is about the manner in which I export diary entries from Day One, import them into DEVONthink and enhance and use them. It effectively replaces my contributions on this old thread (which I’ll supplement to refer to this newer thread).

Why bother bringing Day One diary entries into DEVONthink?

Day One search is very poor and there is no way to construct any complex search query. It is also slow if you have thousands of entries. In addition, you can do things with records in DEVONthink (like transclusion) which you cannot do in Day One. I also add many more photos to my records in DEVONthink than I ever did when the entries were in Day One.

Day One is pleasant enough for writing daily diary entries—automatically adding to each the time, location and weather data. DEVONthink is perfect for enhancing and using those valuable records. For example, when you can so easily search records it encourages use of cross-reference links in Day One which makes the diary entries much more interesting (and, of course, more of a “rabbit hole”!).

Important warnings

When I started importing my Day One entries into DEVONthink I settled on a record naming format of Saturday 30 July 2022. That is not the best format for automatic sorting of the records (in my case, within their yearly group). It’s more sensible to use the format 2022-07-30.md (which enables sensible automatic sorting of the records by name) and I have now renamed all my diary records accordingly. However, Day One does not export the name of the weekday in the heading of each exported entry so I’ve made use of a custom date metadata field in DEVONthink (available only in the PRO edition) in which I store the full date. That affects how I use a Hazel renaming rule and also a script processing newly imported diary records (the rule and script will be explained later in this thread).

The result of that is this important warning: the procedure I’ll outline in this thread, and most of the scripts I’ll include, rely on the record naming convention mentioned in the previous paragraph. If you wish to adopt a different record naming convention you’ll probably need to amend the relevant Hazel rule and certainly many of the scripts accordingly.

The second important warning concerns Day One tags. The export procedure I use does not export tags you may have used in Day One. I simply initially re-created all of my Day One tags in DEVONthink (a lengthy, one-time process) but now use hashtags in my Day One diary entries. If you do that make sure that you have checked the DEVONthink preference Convert hashtags to tags (under Preferences > Files).

Finally, I am far from the world’s best programmer and there are undoubtedly other (and probably better) ways to do what I describe but what I have described works for me.

What you need (i.e., what I use)

• The outstandingly useful Hazel app.
• In Finder, folder dedicated to receive the entries exported from Day One. I’ll call that the Journal Folder.
• In DEVONthink PRO a Temp group where you manipulate the imported records before adding them to the appropriate year group in your database.

What follows

I’ll post separately in this thread, following this post:

• The export procedure and my Hazel rules for automating much of it.
• The import procedure and the DEVONthink smart rule I use to automate that.
• The scripts I use to enhance imported records (adding the name of the weekday, recreating the Day One links as item links in DEVONthink, converting hashtags to YAML front matter).
• Bonus scripts for working with imported Day One records.

Stephen

[Stephen_C]

Export and the Hazel rules

Exporting from Day One

I export using the Timeline view. Simply select in the timeline the entries you wish to export and from the context menu choose Export > Plain Text/Markdown and direct the output (which will be named journal.txt)to the Journal folder (see previous post).

The Hazel Rules

All of the following Hazel rules operate on the Journal folder, of course.

1. The first rule provides that if the file extension is zip the matched file should be unarchived and then Hazel should Continue matching rules.

2. The second rule looks for journal.txt and then runs the following shell script:

   cd journal split -p '^ Date:' $1 out

   (Note there is a tab before the word “Date:”. You enter that by pressing ⌃V and then pressing the tab key.) That script splits the file into each individual entry. The same Hazel rule then renames the journal.txt file delete and continues matching rules.

3. The third Hazel rule looks for files starting with out* and renames them out*.md.

4. The fourth Hazel rule looks for all the out* files and checks for something I have called Diary entry date (but with Automatically detect date format checked in the attribute in the Hazel rule). It then renames each out* file with the pattern Diary entry date extension. However. note that in that case the Diary entry date pattern is edited to this format:

​ After renaming the files the Hazel rule moves them to the DEVONthink Inbox.

5. The final Hazel rule deletes the delete file (i.e., the renamed original journal.txt file).

Stephen

[Stephen_C]

Import into DEVONthink

If you have followed the preceding procedure you will have ended up with a number of files called something like Monday 8 August 2022 in your DEVONthink Inbox. This is how I deal with those.

I have a DEVONthink smart rule which looks like this:

The embedded script is this:

on performSmartRule(theRecords)
	tell application id "DNtp"
		try
			
			repeat with theRecord in theRecords
				-- first we remove the extension from the record name
				set theLongDateText to (name without extension of theRecord) as string
				-- now we simply populate the custom metadata with the date text
				add custom meta data theLongDateText for "diaryentrydate" to theRecord
				-- ensure we have a blank line before the metadata
				set theText to plain text of theRecord
				set amendedText to return & theText
				set plain text of theRecord to amendedText
			end repeat
			
		on error error_message number error_number
			if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
		end try
	end tell
end performSmartRule

The rule:

1. populates the Diary entry date custom metadata field (which is a date type field) with the date name of the record (which, following use of the Hazel rules previously described, is the date of the diary entry); and
2. adds a blank line at the start of the record (which preserves the heading formatted as a block and prevents it being treated as metadata because of the presence of the colon on what would otherwise be the first line).

You now have some diary records in your Temp folder, locked and marked as read and with any Day One hashtags converted to DEVONthink tags. What more could you want? Actually, quite a lot…. I need to take a break but the next post will deal with the initial DEVONthink processing script adding the name of the weekday to each record header and renaming each record to name-sortable format.

Stephen

[DTLow]

Thanks for the detailed processing notes

It’s more sensible to use the format 2022-07-30.md (which enables sensible automatic sorting of the records by name)

My naming standard is
yyyy-mm-dd type [details] keywords
2022-07-30 Journal [2022.211 Saturday] !Type-Journal !Journal-NoteDaily
No custom metadata
No group structure

Day One is pleasant enough for writing daily diary entries—automatically adding to each the time, location and weather data.

I write my journal notes directly to Devonthink, or DTTG on my iPad
Time is automatic; manual entry for location and weather

tags

Could you give examples of your tag use?
My standard tags are !Type-Journal, !Journal-Note<Daily/Health/…>

[Stephen_C]

@DTLow

Could you give examples of your tag use?

My diary records are tagged with things like:

• Photo for entries with a photo or photos attached
• Holiday - self-explanatory but with child tags for the location of the holiday
• Significant for key personal dates
• Humour for records that…well, you get it!
• Coronavirus
• DairyHistory (or, even, DiaryHistory!) for entries where I have chronicled the evolution of my diaries…

and others…(some of which may be regarded as politically inflamatory so I shall refrain from mentioning them here ).

I have smart groups for the various tags which enable me to see the relevant collated records at a glance. It’s always good to click on the Humour smart group if I feel I need a laugh!

Stephen

Edit: corrected typos.

[Stephen_C]

The initial DEVONthink processing script

You have in your Temp folder in your DEVONthink database some diary records with names like this Monday 8 August 2022. You can use the custom date field to sort them in date order of course. But you may wish to export some to Finder one day and have them sorted by date order there. That is one of the purposes of the following script.

When I have imported diary records in my Temp folder (following the procedure outlined in the preceding posts in this thread) I select all of them and run the following script:

(* This script adds the name of the day to the date of diary entries imported from Day One
and better aligns the date field. It renames each record in the format 2022-08-30 for
better sorting. It should be run on any entries when they are imported from Day One.
*)


tell application id "DNtp"
	try
		-- You'd better know what's about to happen so you can opt out if you wish		
		display dialog "Please ensure you have selected one or more records in the Diaries database. This will add the name of the day to the date at the start of each selected record and rename each record in the 2022-08-30 format." with title "Process imported diary entries." default button "OK" with icon 1
		set theRecords to the selection
		if theRecords = {} then error "Please select some diary entries."
		set n to 0
		show progress indicator "Processing records…" cancel button 1 steps count of theRecords
		repeat with theRecord in theRecords
			if cancelled progress then error number -128
			step progress indicator (name of theRecord) as string
			-- First we remove the extension from the record name
			set theLongDateText to (name without extension of theRecord) as string
			-- and get the name of the day
			set theWeekday to word 1 of theLongDateText
			-- Now we get the rest of the date (i.e., without the name of the day)
			set RestOfTheDate to words 2 through 4 of theLongDateText
			-- Store the current settings for the tids
			set tid to AppleScript's text item delimiters
			-- Set the tids to space
			set AppleScript's text item delimiters to " "
			-- Now assemble the full date, including the name of the day and two leading tabs
			set theNewFullDate to "		" & theWeekday & " " & RestOfTheDate & " " as string
			-- Reset tids to whatever they were previously
			set AppleScript's text item delimiters to tid
			-- Get the plain text of the entry
			set theText to plain text of theRecord
			-- and extract the old date (i.e., without the name of the weekday)
			set theOldDiaryDate to my extractBetween(theText, "Date:", "at")
			-- Now we need to replace, in the diary text, theOldDiaryDate
			-- with theNewFullDate
			set theText to my replaceText(theText, theOldDiaryDate, theNewFullDate)
			set plain text of theRecord to theText
			-- now we split the date into the required constituent parts
			set theOldDate to word 2 of theLongDateText & " " & word 3 of theLongDateText & " " & word 4 of theLongDateText as string
			-- format it as needed
			set theNewName to my formatDate(theOldDate)
			-- and rename the record accordingly
			set name of theRecord to theNewName
			set n to n + 1
		end repeat
		hide progress indicator
		-- It's good to know when it's finished, so…
		set theMessage to "Finished. Processed " & n & " records out of a total of " & (count of theRecords) & "."
		display alert theMessage
	on error error_message number error_number
		if the error_number is not -128 then log message "DEVONthink:" & error_message
		hide progress indicator
	end try
end tell

on extractBetween(SearchText, startText, endText)
	set tid to AppleScript's text item delimiters -- save them for later.
	set AppleScript's text item delimiters to startText -- find the first one.
	set endItems to text of text item 2 of SearchText -- everything after the first.
	set AppleScript's text item delimiters to endText -- find the end one.
	set beginningToEnd to text of text item 1 of endItems -- get the first part.
	set AppleScript's text item delimiters to tid -- back to original values.
	return beginningToEnd -- pass back the piece.
end extractBetween

on replaceText(theString, find, replace)
	-- with acknowledgment and thanks to @cgruenberg
	-- https://discourse.devontechnologies.com/t/search-replace-text-in-rtf-document/65757/10
	local od
	set {od, text item delimiters of AppleScript} to {text item delimiters of AppleScript, find}
	set theString to text items of theString
	set text item delimiters of AppleScript to replace
	set theString to "" & theString
	set text item delimiters of AppleScript to od
	return theString
end replaceText

on formatDate(LongDateStr)
	set {dd, mm, yyyy} to words of (short date string of (date LongDateStr))
	set formattedDate to (yyyy & "-" & mm & "-" & dd)
	return formattedDate
end formatDate

When you run that script you will have diary records in your Temp folder with:

1. the name of the weekday to the date at the start of each selected diary record and a better aligned date field; and
2. each record named in the format 2022-08-30.md (making the records sortable by name).

Now…, what about those pesky Day One links (i.e., cross references to other Day One entries)? We’ll deal with those in the next post.

Stephen

[Stephen_C]

Converting Day One links to DEVONthink links

How it works

If, like me, you cross-refer to other Day One entries from the day’s entry you’ll want to ensure those links work within DEVONthink: that’s what the following script does. Note that it doesn’t involve querying the Day One database so it does involve a little manual work obtaining the date(s) of the link(s) from Day One but I find it an efficient way to work.

Run this script on one selected diary record at a time. It will loop through the Day One links in the record, prompt you for the date of the link and then replace the Day One link with a DEVONthink item link. It’s very helpful while running the script to have open the Links inspector in DEVONthink so you can see the links with which you have to deal.

Warnings

1. If you follow, with your diary records, a naming convention other than that referred to in this thread (i.e., and for example, 2022-06-30.md) the script will be remarkably unhappy and will fail miserably. In that case you might wish to tackle amendment of the on getName handler.
2. If you have two identical Day One links in one entry, both of which refer to the same second Day One entry, running the script will replace both Day One links at the same time. That’s entirely logical, of course, but don’t be surprised when you appear to have replaced more links in fewer steps than you might have expected.
3. You need to insert the path to your database at the beginning of the script.

The script

(* This script replaces Day One links in a single selected diary record
with DEVONThink links. It assumes each diary record is markdown
with the name of the record in the format "2022-08-30".
The name format can be changed in the getName handler. You are invited to
input the date of the linked record - obtained manually from checking
the Day One link in Day One - and the script then substitutes
the DEVONThink link for the Day One link in the selected
diary record, looping through the record until all Day One links
have been replaced. *)

-- Set the database we want to use
property pDatabase : "[path to your database]"

tell application id "DNtp"
	try
		set theDatabase to open database pDatabase
		-- Test for selection of single markdown record
		set theSelection to the selection
		if (count of theSelection) is 1 then
			set theRecord to the content record
			if (type of theRecord) is markdown then
				-- Set the current date as the initial default date for the input dialog
				set theDate to short date string of (current date)
				set dAnswer to theDate
				-- Set up a repeat loop to replace Day One links with DEVONThink links
				repeat
					set theText to plain text of theRecord
					if theText contains ("dayone://") then
						-- Ask for the date of the target record in the Diaries database
						-- and provide clear guidance as to the format required
						set userEntry to display dialog ¬
							"Enter the link date as DD/MM/YYYY" default answer dAnswer with title "Replace Day One link with DEVONThink link"
						-- Hold over the first response for the next loop
						-- if there is more than one link to be processed
						set dAnswer to text returned of userEntry as string
						-- Now reformat it to match the name of a diary entry record
						-- Here we use the getName handler to construct
						-- a name like "2022-08-30"
						set theTarget to my getName(dAnswer)
						-- Search for the named record in the database
						set foundRecord to (search "name:" & theTarget & " kind:markdown") in theDatabase
						if foundRecord = {} then
							error theTarget & " was not found in " & pDatabase
						end if
						-- Get the reference URL of the record
						set theDTLink to reference URL of item 1 of foundRecord
						-- Extract part of the Day One link…
						set theDOurl to my extractBetween(theText, "dayone://", ")")
						-- …and assemble it
						set theDOLink to "dayone://" & theDOurl
						-- Now replace the Day One link with the DT link
						set theText to my replaceText(theText, theDOLink, theDTLink)
					else
						error "There are no more Day One links in the selected record."
					end if
					-- Set the record to the amended text
					set plain text of theRecord to theText
				end repeat
			else
				error "Please select a single diary markdown record."
			end if
		else
			error "Please select a single diary record."
		end if
		
	on error error_message number error_number
		if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
	end try
end tell

on getName(dateString)
	set theDay to characters 1 through 2 of dateString
	set theMonth to characters 3 through 5 of dateString
	set theYear to characters -4 through -1 of dateString
	set formattedDate to (theYear & "-" & theMonth & "-" & theDay)
	return formattedDate
end getName

on extractBetween(SearchText, startText, endText)
	set tid to AppleScript's text item delimiters -- save them for later.
	set AppleScript's text item delimiters to startText -- find the first one.
	set endItems to text of text item 2 of SearchText -- everything after the first.
	set AppleScript's text item delimiters to endText -- find the end one.
	set beginningToEnd to text of text item 1 of endItems -- get the first part.
	set AppleScript's text item delimiters to tid -- back to original values.
	return beginningToEnd -- pass back the piece.
end extractBetween

on replaceText(theString, find, replace)
	-- with acknowledgment and thanks to @cgruenberg
	-- https://discourse.devontechnologies.com/t/search-replace-text-in-rtf-document/65757/10
	local od
	set {od, text item delimiters of AppleScript} to {text item delimiters of AppleScript, find}
	set theString to text items of theString
	set text item delimiters of AppleScript to replace
	set theString to "" & theString
	set text item delimiters of AppleScript to od
	return theString
end replaceText

Now enjoy clicking on the links in DEVONthink and flipping through your diary records. Remember you can use the keyboard shortcuts Cmd [ and Cmd ] to move backwards and forwards as you click on the links.

Next…how to deal with those pesky Day One hashtags.

Stephen

[Stephen_C]

Dealing in DEVONthink with Day One hashtags

Summary

1. My method of exporting Day One entries does not export Day One tags (see post one in this thread). To overcome that problem I use hashtags at the end of any Day One diary entry that is to be tagged. In order to ensure the tags are consistent I call them up using an abbreviation stored in Typinator (but, of course, you could use any other text expander app—or even add them manually so long as you retain consistency of naming each tag).
2. If, in DEVONthink preferences, you check Convert hashtags to tags under General > Files the Day One hashtags will be converted to DEVONthink tags on import. However, to my mind, there is a small problem with that.
3. I don’t like to have hashtags cluttering the bottom of my diary records but if you remove the hashtag in DEVONthink the DEVONthink tag also gets removed.
4. This script circumvents that problem by removing the hashtag and creating in the record YAML front matter containing the tag. The tag thus remains in DEVONthink but does not appear in the rendered view of the record.
5. As a bonus, the script also strips any blank lines remaining at the end of the record.

Warnings

1. Like the previous script, if you follow, with your diary records, a naming convention other than that referred to in this thread (i.e., and for example, 2022-06-30.md) this script will fail miserably. Again you might wish to tackle amendment of the on getName handler.
2. As before, you’ll need to insert the path to your database at the beginning of the script.
3. Run the script on one record at a time.

The script

(* This script effectively moves hashtags, typed
at the end of a Day One diary entry (each on a separate line), to
a metadata entry at the start of the DEVONThink diary entry record
and deletes the corresponding hashtag(s) from the end of the record.

It is designed to overcome the fact that export of
Day One entries as markdown fails to export hashtags. It
also ensures that the hashtags do not display in unsightly
fashion in the Day One record.

It assumes each diary record is markdown.  *)

-- Set the database we want to use
property pDatabase : "[path to your database]"

tell application id "DNtp"
	try
		set theDatabase to open database pDatabase
		
		-- Test for selection of single markdown record
		set theSelection to the selection
		if (count of theSelection) is not equal to 1 then
			error "Please select a single markdown diary record"
		end if
		
		set theRecord to the content record
		-- we are going to check for the existence of a hashtag
		set tagItem to "#"
		set existingMetadata to ""
		set theText to plain text of theRecord
		set amendedText to theText
		
		if type of theRecord is markdown then
			-- we do all this so long as the text contains "#"
			if theText contains tagItem then
				repeat while amendedText contains tagItem
					-- we extract the text of the tag minus the #
					set theTag to my extractBetween(amendedText, tagItem, linefeed)
					-- check if we need to add to existing metadata
					if existingMetadata is "" then
						-- there's no existing metadata so we set it up
						set theMetadata to "tags: " & theTag
						set existingMetadata to theMetadata
					else
						-- there is existing metadata so this is what we add to it
						set theMetadata to ", " & theTag
						set existingMetadata to existingMetadata & theMetadata
					end if
					-- reassemble the hashtag
					set theHashTag to "#" & theTag
					-- delete the hashtag from the text (noting that we then
					-- work with the amended text in the repeat loop)
					set amendedText to my replaceText(amendedText, theHashTag, "")
				end repeat
				set existingMetadata to "---" & return & existingMetadata & return & "---" & return
			else
				error "There is no hashtag at the end of this record"
			end if
			-- we add the metadata to the text
			set theText to existingMetadata & "  " & linefeed & linefeed & amendedText
			-- remove any unwanted returns from the end of the record
			set theText to my stripEndBlankLines(theText)
			set plain text of theRecord to theText
			display notification "Hashtags converted to metadata" with title "Convert hashtags to tags"
		else
			error "Please check you have selected a markdown diary record"
		end if
		
	on error error_message number error_number
		if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
	end try
end tell

on extractBetween(SearchText, startText, endText)
	set tid to AppleScript's text item delimiters -- save them for later.
	set AppleScript's text item delimiters to startText -- find the first one.
	set endItems to text of text item 2 of SearchText -- everything after the first.
	set AppleScript's text item delimiters to endText -- find the end one.
	set beginningToEnd to text of text item 1 of endItems -- get the first part.
	set AppleScript's text item delimiters to tid -- back to original values.
	return beginningToEnd -- pass back the piece.
end extractBetween

on replaceText(theString, find, replace)
	-- with acknowledgment and thanks to @cgruenberg
	-- https://discourse.devontechnologies.com/t/search-replace-text-in-rtf-document/65757/10
	local od
	set {od, text item delimiters of AppleScript} to {text item delimiters of AppleScript, find}
	set theString to text items of theString
	set text item delimiters of AppleScript to replace
	set theString to "" & theString
	set text item delimiters of AppleScript to od
	return theString
end replaceText

on stripEndBlankLines(someText)
	-- split the text into a list of paragraphs
	set theParas to paragraphs of someText
	-- keep checking the list until the last item is not a blank line
	repeat until last item of theParas is not ""
		-- check whether we have a trailing blank line
		if last item of theParas is "" then
			-- we do, so strip the last item
			set theParas to items 1 thru -2 of theParas
		end if
	end repeat
	set astid to AppleScript's text item delimiters
	set AppleScript's text item delimiters to linefeed
	set someText to theParas as text
	set AppleScript's text item delimiters to astid
	return someText
end stripEndBlankLines

You will be hugely relived to know that, for the moment, that’s it! However, I do have a couple of bonus scripts (one for quickly finding a diary record and one for quickly retrieving the UUID of a diary record) and can post them if anyone is still alive and interested after getting this far.

Stephen

[BLUEFROG]

Thanks for the thorough posting of your scripts and workflow. You’ve come a very long way from your first tentative steps at AppleScripting. Very nice to see and I’m sure will be useful and inspirational to others thinking about trying their hand at automation.

[Stephen_C]

@BLUEFROG

You’ve come a very long way from your first tentative steps at AppleScripting.

Thanks so much—and that’s in no small part to the considerate mentoring of a number of people on this forum, including you!

Stephen

[Hens]

wow thats a lot of processing but very interesting. wanna give this a go. Thank you. pretty sure to get you where you want it to, took a lot of time.

[dsh1705]

Anyway you could export your Hazel rule? I am having a bit of a struggle trying to figure out the fourth rule you have here.

[dsh1705]

This does not handle any Day One entries that have media attached, right? So even if DayOne exports the photos in the Plain Text/Markdown export, the media is not referenced anywhere?

[Stephen_C]

@dsh1705

Anyway you could export your Hazel rule? I am having a bit of a struggle trying to figure out the fourth rule you have here.

I’m afraid the forum won’t allow me to add the Hazel rule to a post, sorry! If you can tell me the exact problem you have with the rule I’ll try to help.

@dsh1705

even if DayOne exports the photos in the Plain Text/Markdown export, the media is not referenced anywhere?

I suspect that would be the case. However, I now never add photos to Day One entries. What I do is to add a #photo hashtag which will then remind me to add the photo to the entry once it’s imported into DEVONthink. DEVONthink seems to handle the attachments far more efficiently than does Day One. I now have 19,248 diary entries in DEVONthink with 417 photos and the database does not skip a beat when I’m browsing or searching it.

Stephen

[Stephen_C]

I’ve discovered I can add the Hazel rule you wanted as a .zip file so here it is:

Journal.hazelrules.zip (4.4 KB)

Stephen

[dsh1705]

Thanks for this! I think I figured it out (new to Hazel, just got it to make this happen!) but this will help. Thanks so much for sharing this in so much detail.

[Stephen_C]

Glad to have helped.

Stephen