If the program exports JSON, you could write a JavaScript script to import it (the JSON data) into DT. DT can execute such scripts (although not in smart rules).
To be honest I haven’t had (taken?) the time to write a Javascript import procedure. I’m not too experienced in Script so it would take me too long. I am therefore currently still keeping TheBrain on my computer, just to access my data.
Regarding the Folders export route, yes it seems to keep the hierarchy, but in my case I need the links too. And for 3000+ “thoughts” and even more links, I just don’t feel like doing this by hand.
So either some magic happens, or I find the time to write a proper routine, or it will stay as is for the moment.
As a side thought, it probably means it doesn’t “hurt” enough to have such an inefficient setup…
Where would you store these links in the DEVONthink records? Appended to the actual text? Or as custom metadata? If so, which metadata type?
Writing an AppleScript should be doable but I‘ve no idea where to store the links.
I use links in TheBrain in 2 ways : between thoughts, and in-text links (I mean a link from within the text of a thought to another thought).
Migrating to DT, I would seek to replicate this, probably by using a wikilink in the text of a DTnote. Maybe with a "see [[linkednote]] " line at the top -or at the bottom- of a DTnote.
But once again, I haven’t thought this through, as I haven’t got the time right now to code.
Yes, I got that. The reason I asked is that I have time to code and if you don’t insist on doing it yourself I would try to write it.
@shiiko unfortunately this would skip a lot of folders as TheBrain creates aliases and DEVONthink doesn’t import aliases. (Edit: But it’s possible with this script).
I’m writing a script that works around this situation, but I just downloaded TheBrain last week, so I have no idea about the features and how they can be used, also I’ve of course no good test data. I’ve included as much data (that I’m aware of …) in the import as possible, but I’m quite sure I missed things that would be obvious for experienced TheBrain users. I don’t even know how brains are structured, if more than one top thought is possible etc.
My intention is to only use features that can be somehow exported, but without knowing the app it’s quite hard to figure if I’m on a good way.
Would you or other TheBrain users try the script on a copy of real data?
@pete31 I would be willing to try a test. But TheBrain website has some demo Brains that can be downloaded that come pre-filled. Also note that TheBrain is going through a major update and I am using their new v12 which introduces backlinks, among other things.
Great! Didn’t know of demo Brains, I’ll take a look.
Where are backlinks visible? In the plex or in the thoughts? I guess they didn’t change the whole linking system so it’s probably easy to include backlinks. Ah, moment! The script already includes backlinks
I downloaded an update a few days ago but my TheBrain is still 11.0.131.0. Are you beta tester?
In some hours I’ll post a first version.
I think your best bet would be to use the Export Brain Data feature to export a “brain” to JSON and then parse that export file. TheBrain does not support AppleScript, JXA, etc.
The “Folders” and “Text Outline” export options are of limited usefulness.
I made a quick-start brain for you and exported it to JSON. Here’s the data:
Link will self-destruct 20201101
Hi korm, the trick is to use both, “JSON” and “Folders” export. This way we’ll get both, the links and the folder structure
However I didn’t find a way to convert in-text links, so clicking them in DEVONthink will open the thought in TheBrain.
I think it could be possible to replace them afterwards by querying the SQL database, but I didn’t try it yet (and that’s something I’ve never done so I’ll do the basics first).
Thanks!
Thanks! Could you also upload a “Folders” export of this Brain? Without it I can only test a part of the script.
Looks like there’s a bug in the current v12 beta – it crashes when it tries to export as folders.
If you ever have questions re querying the sqlite db, let me know.
May I ask, what is the point of importing the data from The Brain?
Are you transitioning from it to DEVONthink?
Unlike DEVONthink, TheBrain is a locked proprietary platform. Though working with TheBrain is interesting and useful, getting off of it is chore. So, many people have collections of notes, references, and attachments in TheBrain that could reside just as well in DEVONthink. I think that’s a good enough motivation to continue the discussion and thank Pete31 for what he’s working on.
This script should
- Import TheBrain’s JSON export …
- … structured like TheBrain’s Folders export …
- … create outgoing and incoming links in markdown records …
- … create links to attachments …
- … and optionally import external attachments.
It works reliably with my very small test data, I’m of course not sure whether it works as expected with real data.
Please test it even if you don’t want to use TheBrain’s export in DEVONthink, any feedback would be appreciated. I’m not satisfied with the look of the markdown records, however fine looking markdown doesn’t make sense if it doesn’t work …
Create a new folder and export your Brain as “JSON” and as “Folders” into two subfolders named “JSON” and “Folders”.
Hope it works
Edit: There’s a new version, down in this thread.
-- Import TheBrain export
-- Create a new folder and export your Brain as "JSON" and as "Folders" into two subfolders named "JSON" and "Folders".
use AppleScript version "2.4"
use framework "Foundation"
use scripting additions
property useNameAsHeading : true
property theHeadingLevel : "###"
property theLinkSectionDelimiter : "---"
property theLinkDelimiter : " "
property importExternalAttachments : false
global thePlist
tell application "System Events"
try
activate
set theFolderPath to POSIX path of (choose folder with prompt "Choose folder that holds \"JSON\" and \"Folders\" subfolders:" default location (path to desktop folder) without invisibles)
set theExportAsFolderPath to (theFolderPath & "/" & "Folders") as string
my existsPath(theExportAsFolderPath)
if folders of folder theExportAsFolderPath = {} then error "Error: Missing \"Folders\" export"
set theExportAsJSONPath to (theFolderPath & "/" & "JSON") as string
my existsPath(theExportAsJSONPath)
set theExportAsJSONSubfolderPaths to POSIX path of folders of folder theExportAsJSONPath
if theExportAsJSONSubfolderPaths = {} then error "Error: Missing \"JSON\" export"
set theThoughtsJSON_Path to (theExportAsJSONPath & "/" & "thoughts.json") as string
my existsPath(theThoughtsJSON_Path)
set theLinksJSON_Path to (theExportAsJSONPath & "/" & "links.json") as string
my existsPath(theLinksJSON_Path)
set theAttachmentsJSON_Path to (theExportAsJSONPath & "/" & "attachments.json") as string
my existsPath(theAttachmentsJSON_Path)
set thePlist to (theFolderPath & "/" & "Brain.plist") as string
try
exists POSIX file thePlist as alias
on error
make new property list file with properties {name:thePlist}
end try
on error error_message number error_number
activate
if the error_number is not -128 then display alert "System Events" message error_message as warning
return
end try
end tell
tell application id "DNtp"
try
activate
set theGroup to display group selector
set theDatabase to database of theGroup
set theLocationAndName to ((location of theGroup) & (name of theGroup)) as string
show progress indicator "Importing Brain ..." steps 4
on error error_message number error_number
activate
if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
return
end try
end tell
try
my stepProgress("Reading JSON")
set theJSON_LinksDictionary to my readJSON(theLinksJSON_Path)
repeat with thisItem in theJSON_LinksDictionary
set thisThoughtIdA to (|ThoughtIdA| of thisItem) as string
set thisThoughtIdB to (|ThoughtIdB| of thisItem) as string
my writePlist(thisThoughtIdA, missing value, thisThoughtIdB, missing value, missing value)
my writePlist(thisThoughtIdB, missing value, missing value, thisThoughtIdA, missing value)
end repeat
set theJSON_ThoughtsDictionary to my readJSON(theThoughtsJSON_Path)
repeat with thisItem in theJSON_ThoughtsDictionary
my writePlist((|Id| of thisItem) as string, (|Name| of thisItem) as string, missing value, missing value, missing value)
end repeat
set theJSON_AttachmentsDictionary to my readJSON(theAttachmentsJSON_Path)
repeat with thisItem in theJSON_AttachmentsDictionary
set thisName to (|Name| of thisItem) as string
set thisLocation to (|Location| of thisItem) as string
if thisName ≠ thisLocation then my writePlist((|SourceId| of thisItem) as string, missing value, missing value, missing value, {|name|:thisName, location:thisLocation})
end repeat
on error error_message number error_number
tell application id "DNtp" to hide progress indicator
activate
if the error_number is not -128 then display alert "JSON" message error_message as warning
return
end try
tell application "System Events"
try
my stepProgress("Creating Groups")
set thePaths to POSIX path of (disk items of folder theExportAsFolderPath whose visible is true)
my createLocations(thePaths, theExportAsFolderPath, theDatabase, theLocationAndName)
on error error_message number error_number
tell application id "DNtp" to hide progress indicator
if the error_number is not -128 then display alert "System Events" message error_message as warning
return
end try
end tell
tell application id "DNtp"
try
set theJSONExportGroup to create record with {name:"JSON Export", type:group} in theGroup
set theSmartGroup_Group to create record with {type:smart group, search predicates:"kind:group", search group:theGroup, name:"kind:group"} in theGroup
set theSmartGroup_Any to create record with {type:smart group, search predicates:"kind:any", search group:theGroup, name:"kind:any"} in theGroup
repeat with thisPath in theExportAsJSONSubfolderPaths
import thisPath to theJSONExportGroup
end repeat
my stepProgress("Creating Links ...")
set theIDs to my getPlistIDs(thePlist)
set theRecordNames_record to {}
repeat with thisID in theIDs
set theThoughtGroups to (children of theJSONExportGroup whose name = thisID)
if theThoughtGroups ≠ {} then
set theThoughtGroup to item 1 of theThoughtGroups
set theThoughtGroupChildren to (children of theThoughtGroup whose name = "Notes.md" or name = "Notes")
if theThoughtGroupChildren ≠ {} then
set theMarkdownRecord to item 1 of theThoughtGroupChildren
set {theName, theLinkIDs_outgoing, theLinkIDs_incoming, theAttachments} to {item 1, item 2, item 3, item 4} of my readPlist(thePlist, thisID)
step progress indicator ("Creating Links ... " & theName) as string
set end of theRecordNames_record to {|record|:theMarkdownRecord, |name|:theName}
if useNameAsHeading = true then set plain text of theMarkdownRecord to theHeadingLevel & space & theName & linefeed & linefeed & (plain text of theMarkdownRecord)
if theLinkIDs_outgoing ≠ {} then my writeMarkdownLinks(theLinkIDs_outgoing, theJSONExportGroup, theMarkdownRecord, "Outgoing")
if theLinkIDs_incoming ≠ {} then my writeMarkdownLinks(theLinkIDs_incoming, theJSONExportGroup, theMarkdownRecord, "Incoming")
my writeMarkdownLinks_Attachments(theAttachments, theMarkdownRecord)
end if
else
error "Can't get group ID \"" & thisID & "\""
end if
end repeat
my stepProgress("Moving Records")
repeat with this_record in theRecordNames_record
set thisRecord to (|record| of this_record)
set thisName to (|name| of this_record)
set name of thisRecord to thisName
set theGroups to (children of theSmartGroup_Group whose name = thisName)
if theGroups ≠ {} then
set thisGroup to item 1 of theGroups
set theChildren to (children of (parent 1 of thisRecord) whose type ≠ group)
if theChildren ≠ {} then
repeat with thisChild in theChildren
move record thisChild to thisGroup
end repeat
else
error "Can't move children"
end if
end if
end repeat
hide progress indicator
on error error_message number error_number
hide progress indicator
activate
if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
return
end try
end tell
on existsPath(thePath)
try
exists POSIX file thePath as alias
on error
activate
display alert "Error: Handler \"existsPath\"" message "Can't get path: \"" & thePath & "\"" as warning
error number -128
end try
end existsPath
on stepProgress(theInfo)
tell application id "DNtp" to step progress indicator theInfo as string
end stepProgress
on readJSON(thePath)
try
set ndJSON_File to POSIX file thePath
set ndJSON to read ndJSON_File as «class utf8»
set theParagraphs to paragraphs of ndJSON
if (item -1 in theParagraphs) as string = "" then set theParagraphs to items 1 thru -2 in theParagraphs
set theJSON to "[" & linefeed & my tid(theParagraphs, "," & linefeed) & linefeed & "]" -- ndJSON -> JSON
set theJSON_String to current application's NSString's stringWithString:theJSON
set theJSON_Data to theJSON_String's dataUsingEncoding:(current application's NSUTF8StringEncoding) -- https://forum.latenightsw.com/t/reading-json-data-with-nsjsonserialization/958/2
set theJSON_Dictionary to current application's NSJSONSerialization's JSONObjectWithData:theJSON_Data options:0 |error|:(missing value)
on error error_message number error_number
activate
display alert "Error: Handler \"readJSON\"" message error_message as warning
error number -128
end try
end readJSON
on writePlist(theID, theName, theOutgoingLinkID, theIncomingLinkID, theAttachments)
tell application "System Events"
try
tell property list file thePlist
if not (exists property list item theID) then make new property list item at end of property list items of contents with properties {kind:record, name:theID, value:{|name|:"", |linkIDs_outgoing|:{}, |linkIDs_incoming|:{}, attachments:{}}}
if theName ≠ missing value then set (value of property list item "name" of property list item theID) to theName
set theValue_LinkIDs_outgoing to (value of property list item "linkIDs_outgoing" of property list item theID) -- testing
if theOutgoingLinkID ≠ missing value and theValue_LinkIDs_outgoing does not contain theOutgoingLinkID then set (value of property list item "linkIDs_outgoing" of property list item theID) to theValue_LinkIDs_outgoing & {theOutgoingLinkID}
set theValue_LinkIDs_incoming to (value of property list item "linkIDs_incoming" of property list item theID) -- testing
if theIncomingLinkID ≠ missing value and theValue_LinkIDs_incoming does not contain theIncomingLinkID then set (value of property list item "linkIDs_incoming" of property list item theID) to theValue_LinkIDs_incoming & {theIncomingLinkID}
set theValue_Attachments to (value of property list item "attachments" of property list item theID) -- testing
if theAttachments ≠ missing value and theValue_Attachments ≠ theAttachments then set (value of property list item "attachments" of property list item theID) to {theAttachments}
end tell
on error error_message number error_number
activate
display alert "Error: Handler \"writePlist\"" message error_message as warning
error number -128
end try
end tell
end writePlist
on createLocations(thePaths, theExportAsFolderPath, theDatabase, theLocationAndName)
tell application "System Events"
try
repeat with thisPath in thePaths
set thisPath to thisPath as string
set thisResolvedPath to my resolveAlias(thisPath)
if (class of (disk item thisResolvedPath)) = folder then
tell application id "DNtp"
try
if thisPath = thisResolvedPath then
set theLocation to my removeFromString(thisPath, theExportAsFolderPath)
create location ((theLocationAndName & "/" & theLocation) as string) in theDatabase
else
set theAliasDestination_Location to my removeFromString(thisResolvedPath, theExportAsFolderPath)
set theAliasDestination_Group to create location ((theLocationAndName & "/" & theAliasDestination_Location) as string) in theDatabase
set theAliasParent_Location to my removeFromString(my dir(thisPath), theExportAsFolderPath)
set theAliasParent_Group to create location ((theLocationAndName & "/" & theAliasParent_Location) as string) in theDatabase
replicate record theAliasDestination_Group to theAliasParent_Group
end if
on error error_message number error_number
activate
display alert "Error: Handler \"createLocations\" - DEVONthink" message error_message as warning
error number -128
end try
end tell
end if
if (class of disk item thisPath) = folder then
set thesePaths to POSIX path of (disk items of disk item thisPath whose visible is true)
my createLocations(thesePaths, theExportAsFolderPath, theDatabase, theLocationAndName)
end if
end repeat
on error error_message number error_number
activate
display alert "Error: Handler \"createLocations\"" message error_message as warning
error number -128
end try
end tell
end createLocations
on getPlistIDs(thePlist)
try
tell application "System Events"
tell property list file thePlist
set theIDs to name of property list items
end tell
end tell
on error error_message number error_number
activate
display alert "Error: Handler \"getPlistIDs\"" message error_message as warning
error number -128
end try
end getPlistIDs
on writeMarkdownLinks(theLinkIDs, theJSONExportGroup, theMarkdownRecord, theLinkSectionName)
tell application id "DNtp"
try
set theMarkdownLinks to {}
repeat with thisLinkID in theLinkIDs
set theLinkGroups to (children of theJSONExportGroup whose name = thisLinkID)
if theLinkGroups ≠ {} then
set theLinkMarkdownRecords to (children of (item 1 of theLinkGroups) whose name = "Notes.md" or name = "Notes")
if theLinkMarkdownRecords ≠ {} then
set thisLinkMarkdownRecord to item 1 of theLinkMarkdownRecords
set thisLinkMarkdownRecord_Name to item 1 of (my readPlist(thePlist, thisLinkID))
set thisLinkMarkdownRecord_ReferenceURL to reference URL of thisLinkMarkdownRecord
set end of theMarkdownLinks to ("[" & thisLinkMarkdownRecord_Name & "](" & thisLinkMarkdownRecord_ReferenceURL & ")") as string
else
error "Can't get markdown record \"" & thisLinkID & "\""
end if
else
error "Can't get group \"" & thisLinkID & "\""
end if
end repeat
if theMarkdownLinks ≠ {} then
set theMarkdownLinks_string to my tid(my sort_list(theMarkdownLinks), theLinkDelimiter)
set theMarkdownLinksSection to theLinkSectionDelimiter & linefeed & linefeed & theLinkSectionName & ": " & theMarkdownLinks_string & space & space
set theText to plain text of theMarkdownRecord
set newText to (theText & linefeed & linefeed & theMarkdownLinksSection) as string
set plain text of theMarkdownRecord to newText
end if
on error error_message number error_number
activate
display alert "Error: Handler \"writeMarkdownLinks\"" message error_message as warning
error number -128
end try
end tell
end writeMarkdownLinks
on writeMarkdownLinks_Attachments(theAttachments, theMarkdownRecord)
tell application id "DNtp"
try
set theMarkdownLinks to {}
set theAttachmentRecords to (children of (parent 1 of theMarkdownRecord) whose name ≠ "Notes.md" and name ≠ "Notes")
repeat with thisAttachmentRecord in theAttachmentRecords -- Internal
set thisAttachmentRecord_ReferenceURL to reference URL of thisAttachmentRecord
set thisAttachmentRecord_Name to name of thisAttachmentRecord
set end of theMarkdownLinks to ("[" & thisAttachmentRecord_Name & "](" & thisAttachmentRecord_ReferenceURL & ")") as string
end repeat
repeat with this_record in theAttachments -- External
set thisName to |name| of this_record
set thisLocation to |location| of this_record
if thisLocation starts with "/" then
if importExternalAttachments = false then
tell application "System Events"
try
set thisLocation to URL of disk item thisLocation
end try
end tell
set thisMarkdownLink to ("[" & thisName & "](" & thisLocation & ")") as string
set end of theMarkdownLinks to thisMarkdownLink
else
set thisAttachment to import thisLocation to (parent 1 of theMarkdownRecord)
set thisAttachment_ReferenceURL to reference URL of thisAttachment
set thisAttachment_Name to name of thisAttachment
set end of theMarkdownLinks to ("[" & thisAttachment_Name & "](" & thisAttachment_ReferenceURL & ")") as string
end if
else
set thisMarkdownLink to ("[" & thisName & "](" & thisLocation & ")") as string
set end of theMarkdownLinks to thisMarkdownLink
end if
end repeat
if theMarkdownLinks ≠ {} then
set theMarkdownLinks_string to my tid(my sort_list(theMarkdownLinks), theLinkDelimiter)
set theMarkdownLinksSection to theLinkSectionDelimiter & linefeed & linefeed & "Attachments" & ": " & theMarkdownLinks_string & space & space
set theText to plain text of theMarkdownRecord
set newText to (theText & linefeed & linefeed & theMarkdownLinksSection) as string
set plain text of theMarkdownRecord to newText
end if
on error error_message number error_number
activate
display alert "Error: Handler \"writeMarkdownLinks_Attachments\"" message error_message as warning
error number -128
end try
end tell
end writeMarkdownLinks_Attachments
on readPlist(thePlist, theID)
try
tell application "System Events"
tell property list file thePlist
if exists property list item theID then
set theName to (value of property list item "name" of property list item theID)
set theOutgoingLinkIDs to (value of property list item "linkIDs_outgoing" of property list item theID) as list
set theIncomingLinkIDs to (value of property list item "linkIDs_incoming" of property list item theID) as list
set theAttachments to (value of property list item "attachments" of property list item theID) as list
return {theName, theOutgoingLinkIDs, theIncomingLinkIDs, theAttachments}
else
error "Plist: Can't get ID \"" & thisID & "\""
end if
end tell
end tell
on error error_message number error_number
activate
display alert "Error: Handler \"readPlist\"" message error_message as warning
error number -128
end try
end readPlist
on tid(theInput, theDelimiter)
set d to AppleScript's text item delimiters
set AppleScript's text item delimiters to theDelimiter
if class of theInput = text then
set theOutput to text items of theInput
else if class of theInput = list then
set theOutput to theInput as text
end if
set AppleScript's text item delimiters to d
return theOutput
end tid
on resolveAlias(thePath)
try
set thePath_URL to (current application's class "NSURL"'s fileURLWithPath:thePath)
set thePath_Original_URL to (current application's class "NSURL"'s URLByResolvingAliasFileAtURL:thePath_URL options:0 |error|:(missing value))
set thePath_Original to (POSIX path of (thePath_Original_URL as string))
if thePath_Original ends with "/" then set thePath_Original to (characters 1 thru -2 in thePath_Original) as string
return thePath_Original
on error error_message number error_number
activate
display alert "Error: Handler \"resolveAlias\"" message error_message as warning
error number -128
end try
end resolveAlias
on dir(thePath)
try
set revPath to (reverse of characters of thePath) as string
return (reverse of (characters ((offset of "/" in revPath) + 1) thru -1 in revPath)) as string
on error error_message number error_number
activate
display alert "Error: Handler \"dir\"" message error_message as warning
error number -128
end try
end dir
on removeFromString(theText, CharOrString)
local ASTID, theText, CharOrString, lst
set ASTID to AppleScript's text item delimiters
try
considering case
if theText does not contain CharOrString then ¬
return theText
set AppleScript's text item delimiters to CharOrString
set lst to theText's text items
end considering
set AppleScript's text item delimiters to ASTID
return lst as text
on error eMsg number eNum
set AppleScript's text item delimiters to ASTID
error "Can't RemoveFromString: " & eMsg number eNum
end try
end removeFromString
on sort_list(theList)
considering numeric strings
set theIndexList to {}
set theSortedList to {}
repeat (length of theList) times
set theLowItem to ""
repeat with a from 1 to (length of theList)
if a is not in theIndexList then
set theCurrentItem to item a of theList as text
if theLowItem is "" then
set theLowItem to theCurrentItem
set theLowItemIndex to a
else if theCurrentItem comes before theLowItem then
set theLowItem to theCurrentItem
set theLowItemIndex to a
end if
end if
end repeat
set end of theSortedList to theLowItem
set end of theIndexList to theLowItemIndex
end repeat
end considering
return theSortedList
end sort_list
@pete31 thanks, I’ll test this later.
One thing I could change, or make optional by commented code, would be to import the data to a new database. Obviously, we can create a database and do that if we want that option. Some “brains” can be very large and moving the import into a new database for cleanup might be a good idea. With DEVONthink 3, moving the import over to existing database(s), post-cleanup, is no big deal.
Unfortunately JSON export in TheBrain is all of nothing – you cannot export a subset of a database.
Thought of it while writing but finally skipped it as it didn’t make sense in my case with this little test brain, for large brains this is of course a good idea.
It’s possible to export the desired subset as “Folders” export and then match the JSON against it. That’s how records are moved in the script too, so exporting subsets should be doable.
Sure, for someone who wants to muck around in different exports, munge them together, and figure it out. Ordinary folk don’t need that annoyance. But, my point is that TheBrain is not designed to merely select several “thoughts” (notes) and export just that selection to JSON. IMO, it’s bad design, but the product manager of TheBrain does not agree with me.