Journal script: a relational notes system that works like Roam Research

[ngan]

This script creates a relational notes system in DT3 in a way that is similar to Roam Research. Another way to describe this script is a complex script that creates the wikilinks of a special type of views in which each view is a page that merges the text content of all notes under a tag. The view can be refreshed to reflect the change in files and their contents in a tag, and the files can be edited directly from within the view.

I have re-written this post to reflect all latest changes up to version v1b16.

There are three nonexclusive approaches to relate notes and documents by brute-force: by links and by groups and tags. For example, wikilink and DT’s item-link connect notes by backlinks; the Hook app creates a hidden group for each document and files are hooked when they are being placed into the same hidden groups. The Roam Research mainly uses tags to connect its text entries.

My Stack V5, Stacklink, scripts create and manage notes somewhat similar to the Hook app and strengthen the article-notes connections by DT-links. I use Tagger V5 extensively to maintain the lateral relationship among the notes.

I develop MergeView and MVoutline scripts to access info within each tag and the connection among tags. MergeView script merges the content of files under a tag and MVOutline consolidates the merge views of tags into an outline. But the workflows are too linear to be effective in exploring the connection of info. Journal script takes a different approach.

What the script does:

What a journal view does: A journal view keeps track of notes or random thoughts or tasks in chronological order.

New text entries in a journal view are converted into timestamped notes. When the user enters specially formatted links (view links) in a note, Journal will convert those view links into tags assignment. The links point to pages (tag views) that merge the text content of notes within the same tags. For entry of new view link, the script will create a new tag and a new tag view. User can surf among the links of journal views and tag views effortlessly.

What a tag view does: A tag view is a page that keeps track of the components and contents of files under a tag, and allows user to edit each file [almost]directly within the view. Existing tags in a database are integrated into the system of Journal automatically.

While writing a note, the user can type the view link of an existing tag to review their previous notes instantly. The user can also type a new view link to start collecting notes under a new tag. By integrating writing, review, and tagging into a single process, less time is spent on searching for keywords or setting up smart rules. Furthermore, the user can begin the review from a tag view of interests and surf among other view links in the tag view to explore the connections of their previous knowledge.

To really understand what Journal can achieve, you will need to: (1) be comfortable with the markdown editing environment of DT; (2) know briefly about Roam Research (a rather good overview series here); (3) know briefly about what Mergeview and MVOutline scripts do.

DEMO 1: Setting up a journal of personal investment in a new database**

During the first time set up, the script will ask the user for a few preferences and will create two folders in the database: “Journal resources” and “Journal Notes”.

The main elements of a journal view (the tag view has the same elements)

1. Journal view: There are three journal views: Month Journal, Year Journal, and Full Journal. Month Journal shows the notes for the current month. Year Journal shows the view link of twelve calendar months. Full Journal shows the view links of all calendar years and months. For example, clicking on [[tt2020]] in a Full Journal (the lower-right window in the image below) will bring the user to a tag view that shows all notes in 2020. I will explain about what is [[ttSomeName]] later.

2. Text Commands: Text command is like a menu system. Journal will perform a specific action when the user selects a text command and invoke the script. There are now seven text commands in journal view and tag view in version v1b16.

3. Text entry area: Any texts that are entered within the delimiters [[]] are converted into a single journal note after the script is invoked. The user can create multiple notes using multiple sets of [[]]. Notes that are created in Journal use timestamp as the name. But any files in the database can be added to the journal view.

4. Info bar: A info bar that shows the refresh time of the view, number of notes in the view, and which database is Journal running upon. Each database has its own Journal.

First note entry in the investment and personal diary

My first note to create is the earnings reporting of Apple. In this example, I enter four view links in the text: [[ttAAPL]], [[ttEarnings Reports]], [[tt2020-07-28]], and [[tt2020-10-29]]. There are no existing tags and tag views in the database at the time of entry, but the script will take care of it.

What is view link? Typing a new view link [[ttAAPL]] means that I want to create and assign a tag of “AAPL” to the note, and create a tag view “ttAAPL” to merge the text context of all notes under the tag “AAPL” now and in future.

Journal will create tag views for existing tags in a database at the time of set up. For example, if “Earnings Report” is an existing tag, typing the view link of [[ttEarnings Report]] means that I want the assign the existing tag of “Earnings Report” to the note, and use [[ttEarnings Report]] to link to the tag view of “Earnings Report”.

Four things happened after the note is typed within [[[ ]]] and the script is invoked.

1. The text entry becomes a note, is saved under “Journal Notes”, and has a timestamp of 20200701.1931.4555 as the name. The user can click on the name in the view to edit the note directly (see how MergeView works).

2. and 3. View links are converted into tags. New tags are created under “New Tags”. The date tags are converted into hierarchical date-tags. E.g. The script creates the tags of 2020, 2020-07, and 2020-07-28 for [[tt2020-07-28]].

4. Tag views are created corresponding to the newly created tags under “Tag views”.

Note: An existing note or file can be added to the journal view by adding a hierarchical date-tag. The user can assign a date-tag to a note/file by invoking the script while it is in the active window, or type a view link of date in the note.

What is tag view: Tag view is almost the same as journal view except for two differences: First, notes are automatically in a tag view when they have the same tag, but notes are in journal view only when they have a date tag. Second, a new note created in journal view will be given a today date tag as default, a new note created in tag view will be given the tag of the tag view.

The user can create multiple notes at once in journal view and tag view. In this example below, two new notes are created at the same time.

For user who prefers to use render view (like me), there is a way to create new notes without using the source view.

Select the text command “EditPane” and invoke the script, a markdown file will be shown. The user can create multiple new notes within the EditPane file. Notes will be created after invoking the script. All notes that are created in the EditPane file will be given a date-tag of today.

Summary: The relational notes system of journal views and tag views.

How to surf to tag views that don’t show their links in the current view.

Select the text command “TagView” and invoke the script. User can select and open one or multiple tag views from the list. The tag views will be refreshed when they are opened.

If the user has a long list of tag views, the list can be filtered by using the “1.Set/Reset list filer” to enter multiple partial search strings.

Clean View: A reading-focused view of tag view and journal view.

Each note in the tag view and journal view always have its name-link and a list of view links at the bottom of the note’s section. Those are the links to facilitate surfing among views. But the user can also switch to a more condensed view by selecting the text command [CleanView] and invoke the script. In the long run, if the user always use type-in view links for tagging, the clean view is the way to go for.

DEMO 2: How I surf from one concept/topic to another by using tagview in a current database:

The user doesn’t need journal view to take advantage of the relational note system of Journal. Existing notes and literature with tags are embedded in Journal.

With Journal, I can now surf from one tag view to another to explore the connection of my previous notes and reference articles to the one I am writing, or to the different hubs of ideas.

How to write, review previous notes, and create a new collection of an idea in one step.

The user can type in a link of exisiting tag view and click on the link to recall the merged content of their previous notes instantly. While reviewing those contents, the user can add new thoughts to the old notes within the tag view by clicking on the name of the note to start editing it.

The user can continue to add existing links of views or create new view link for new ideas while writing a note. The flow of writing, review of ideas, and the creation of new ideas are integrated into the process of writing.

Text command “[[”.

• When the user selects the double brackets and invoke the script in a markdown or rich text file, the list of tag views will pop up. When the user selects one/more tags from the list, the view links will be pasted into the note.
• The matching of [[ in Journal uses case-insensitive wildcard matching. Which means that the search letters doesn’t have to match from the beginning of the tags name.
  
  

Note: Even If the user has forgotten to type a view link to create new tags and has instead created new tags by the standard DT ways, as long as the user invokes the script while in the note the script will scan and create the new tag views for the newly created tags.

The next post will explain the rather complex but logical arrangement of the contextual actions performed by a single short-cut key of Journal script.

[ngan]

The contextual actions of Journal script

When the group “Journal resources” is selected: the action from the script is run set up preferences and scan and update all tag views in the database.

When any group/s or no item is selected: the action from the script is open the list of journal views and other favourite tag views. See post#3.

Alternatively: the user can go to group “Tag views” and “Journal Views” to open any views directly. Don’t worry that the existing notes will be damaged if these views are deleted or altered. These views are just the containers of info from other notes with the ability to extract texts from those notes dynamically. If views or even the two groups are deleted entirely, running setup again will create them again.

When the frontmost document is a journal view: the actions are (1) scan and convert all texts in each [[[ ]]] into journal notes, add a date-tag for today to the notes, scan all view links in note and to assign and create the tags and tag views, (2) clear the text entry area after the notes are created, and (3) refresh the journal view.

When the frontmost document is a tag view: the actions are the same as journal view except that date tag for today won’t be added. Instead, the underlying tag of the tagview will be assigned to the note. For example, a note created under the tag view “ttAAPL” will be assigned the tag of “AAPL” even when the view link “[[ttAAPL]]” is not typed in the text.

When a text command is selected:, the actions of the script is to act according to the text command. V1b16 now has [EditPane] to add note more easily, [TagView] to open tag views directly, [IncTags] [ExcTags] menu for tags filter, [Favorites] to show the list journal views and the frequently used tag views, [Template] to open a file that saves the commonly used snippets for note taking, and [CleanView] for reading-focused view. See post#3.

When the “EditPane” is the frontmost document: the actions are (1) scan and create notes from the texts in each block of [[]] (2) scan, assign and create the tags and tag views for all view links in the note (3) assign a date tag of today to the note (4) clear the EditPane file.

When a markdown and the rich text file is the frontmost document:, the action is (1) to scan all view links in [] that have a prefix of “tt” in the note, convert and assign the tags to the notes, and create a new tag and tag view for newly entered view links. (2) to scan the existing tags of the file, if tags are found to not having the corresponding tag views, the script will create them.

When any other file formats are selected: the action is to ask the user whether they want to add a hierarchical date tag in order to show the file at a specific date in the journal view.

The script

The refresh time is faster than MergeView and MVOutline scripts by 3 times. In my computer (quite a fast one), it takes two secs to sort and merge 200 text files and 360 pdf list in a tag view. I have to give big thanks to MacScripter.net, particularly to Nigel Garvey for his two super-fast sorting handlers (with multiple sort keys!) for list of lists that contain reference objects, and the MacScripter.net members who taught me how to extract text entries from within the [[]] efficiently, and to @cgrunenberg who kindly updates the get preferences in DT dictionary and mentioned the ASObjC handler for sorting a plain list in another post and tips from @BLUEFROG in how to save a DT file programmatically.

You need to download Dialog Toolkit Plus from the Late Night Software website and save it under “/Library/Script Libraries”.

The script doesn’t require the user to create a custom metadata field nor groups or tags in advance. There are no internal preferences to be changed. But if you know script you’ll know where and how to tweak the parameters.

V1b16 posted on 2020.07.01. Additional features of all updates are described in post #3
JournalV1b16.scpt.zip (210.3 KB)

[ngan]

I am consolidating the changes of all updates, up to the v1b16, of Journal script here:

1. Add text command of [IncTags] and [ExcTags].

• [IncTags] and [ExcTags] filter the notes in tag view by tags are and tags are not respectively.

Example: There are 561 notes+articles that are under the tag of C.China. I only want to see those notes that also have the tag of T.Institutional_Logics. I select “IncTags” ( without the square bracket! ) and invoke the script. The list will only shows the tags of the notes that are in tag view C.China.

• The symbol of means that Tags are filter is on.
  
  

• Both filters can support multiple selections of tags, but they cannot be used together. Personally, I find IncTags more useful.
• The symbol of means that the filter of Tags are not is on.
  
  

2. Add text command [Favorites] to journal view and tag view
   The user can replicate the more frequently used tag views into the group “Journal Views”.

• When the user selects the text command [Favorites], or using the contextual action of selecting any group or select no item, a list will be showing the three journal views (always on top) and the tag views in the “Journal Views” group.
• The view will be refreshed before it’s opened.

3. Add the text command of [Template].

• When the text command is activated, a blank markdown file with the name of “NoteTemplate-Database name” is opened. The user can save the commonly used snippets to be copied and paste into different views while creating notes.

4. The EditPane now include the relevant text commands at the top.

5. Journal v1b14 now allows case-insensitive wildcard matching for view tags lookup when invoking the contextual action of [[ .

• For example, select “[[pol” and invoke the script will bring up a list of tag views that contains “pol” anywhere in the name. Double-click or OK button will paste the view links into the note.
• If only “[[” is selected, the list of all tag views will be shown.

6. A small change in Month Journal. The view now separates the notes into two sections: Notes that are created today and notes that are created in the rest of the month.

7. Add one more text command [CleanView]/[FullView]. When the user wants to focus on reading, they can select [CleanView] for a more condensed presentation of notes.

FullView is the default mode of view. If the user selects [ClearView] and invokes the script.

CleanView: Once CleanView is chosen, the choice will apply to all databases when the views are refreshed, until the user changes the setting to FullView again, or until they quit DT.

How to create new note and edit existing note directly in CleanView, and how to surf among view links

1. The surfing from view to view can only be performed through the in-text view link.
2. While in Clearview, the user can still edit each note directly by clicking on the square dot. The square dot is a DT-link to the note. See the first image here:
3. If the user needs to add note/s while in CleanView, select EditPane and invoke the script and create notes from there. See the second image here:
   
   
   
   
   

[ngan]

This is a rather interesting update v1b19.

Two problems with using the Journal script:
Problem 1: What if the notes are only tagged in broad categories, or not tagged at all?
Problem 2: When there are many notes within a tag view, it is getting harder to read all of them.

Update 1. A possible solution to problem 1 by adding the third type of view: Search View.

For example, I am reading the tag view of ‘C.China’. I come across the word ‘capitalism’. I haven’t defined a tag for this idea but I’d like to check what have I written about it. I select the word “capitalism” and invoke the script.

A search view for the word ‘capitalism’ is shown. A search view has the prefix of “ss” instead of “tt”.

Note: The user can also insert the link of an existing search view or to create a new search view by typing e.g. [[ssThe new search Words]] into the note. When the script is invoked, it will scan the note and create the search view when it is applicable.

Within the view, I continue the search on ‘developing country’. I select the phrase and invoke the script.

Another search view is created. If the search view for ‘developing country’ already exists, the view will be refreshed and opened.

Search view returns more focused results by performing a two-stage search behind-the-scene:

The standard DT’s function of ‘search name’ while selecting the words in a document will return all documents that contain the search words. But the Journal script is focused on notes.

How search view works:

• First, the script conducts a DT search on ‘capitalism’ but only for rich text and markdown files.
• Second, the script extracts all paragraphs that contain the word ‘capitalism’ from each note that is found in the first-stage search (the extraction is performed by ‘grep -i’).
• Therefore, within the search view only the paragraphs that contain the search words will be shown under the section of each note, but the user can click on the note’s link to read/edit the complete original note.

Search view can be created or opened as long as the user selects the words and invokes the script while in journal view, tag view, search view, markdown or rich text file, and pdf file.

---

Update 2. A possible solution to problem 2 by adding the text command of [WordFilter]

There are 561 notes + files under ‘C.China’. I only want to read the notes, within the current tag view, that contains the word ‘CSR’.

The number of notes under the tag view is reduced from 561 to 52.

The script

JournalV1b19.scpt.zip (221.6 KB)

Technical discussion:

For the two-stage search in search view, the script will make internal adjustments to the search predicate. Let’s use the search words “developing country” as an example:

the DT query used by the script internally is:
text: "developing country*" {any: kind:markdown kind:rtf}
the regex term feed to the grep - i by the script is:
grep -i 'developing*.country*'