I would really find this functionality useful. Would it be possible to write a script that generates a graph—perhaps using D3JS—that displays as a static HTML page within DT3? It might then be possible to include links from the nodes in the graph to the resources in the DT3 database (e.g., click the node to open the resource), as well as metadata of each resource.
1 Like
That is awesome. Would be really nice to have obsidian-like support in Devonthink. That was a huge news a few days ago at HN which shows demand for this functionality on the market.
Roam research plans a 15 usd subscription. Obviously, Devonthink could offer already more out of the box provided they would be able to serve that new emerging market of “graph-based knowledge management systems”.
1 Like
There is a major difference between a graph of a handful of documents and the thousands of documents in a typical DEVONthink database. @benoit.pointet’s example clearly shows the problem.
2 Likes
As a static graph of an entire database it can have readability issues, but certainly what would help is having the graph update based on where the user is focusing their attention.
For example, if the graph focuses on the currently active document and highlights only the connections of up to X degrees of separation (ideally user adjustable) then that would help a lot to contextualise the current document. If one could then click on one of the related nodes to “walk” through the graph and have the graph then dynamically update to centre on that node then this would allow one to follow a chain of related thought.
This could therefore be an additional way of navigating through one’s notes rather than browsing through lists of items in a given Group or Tag. Furthermore, it complements the AI functionality in helping expose related notes to a user.
One other thing to note is that even in a database of thousands of documents, not all documents would have a x-devonthink-item:// link in them - only the notes where a user has explicitly made the connection.
3 Likes
Interesting path of discussion.
To clarify: what I coded the other night is a dirty tech proof of concept (and a very poor representation).
It did prove (to myself)
- that generating a graph on a decently large DB was technically feasible in a decent amount of time (in a script language).
- that in-browser graph representations nowadays handle 1000+ nodes in a breeze.
Decent graph libs (with a broad palette of interaction and layout/representation possibilities) are a rare breed. Most graph libs are produced by academia and dropped once the lead dev is through with academia life. So the web stack is of interest in this sense: a broad and live ecosystem. With very mature low-level frameworks like D3.js or more specialized ones like sigma.js.
One could then imagine that DT “serves to itself” live graph data which a web-based view asks for, consumes an provides to the user.
So tech solutions exist.
Now the hardest part: network graphs are ungrateful. To most eyes, it will remain “blobs”. At best “exciting blobs”. Because data visualization implies elaborate visual language(s), and graphs are complex structures. Most of us haven’t properly learned how to read graphs. And most of us won’t take the time to do so.
Meaning: the true challenge does not lie, imo, in coding it, but in finding what representation and interactions to provide, (sub-graph span, types of nodes, node properties, types of relations and their properties, selection, hovering, edition, …), and, of utmost importance, to support what purpose.
Some examples:
I read in above comments : “sensing the shape of a wiki”, “following thought paths”, “visually grasping the context of an item”.
The purpose that initially triggered my explorations is a bit different: “seeing how my 1000+ topics (groups) interconnect, being able to re-arrange and link/unlink them” directly on the graph.
So here’s probably where this discussion could continue: if there was a graph representation of a DB, what would it allow you to do that the current DT interface doesn’t? What would it ease? What possibilities would it unlock?
I do agree that as a community our role is not to code a core UI element.
But our role might well be to contribute shaping the purposeful dream of it.
4 Likes
Much as I love (noisy!) force-directed diagrams as much as the next person, I find they become an exercise in filtering and setting up mapping and relationships to render them - not that this is not a useful thing in and of itself, but would require a fair whack of UI work to build the mapping interface as much as the rendering of it.
I’ve enjoyed the graph plots that DEVONagent can produce when searching, and the fact that I can direct this search at DEVONthink as a data source to extract some of the high-level maps it produces. Perhaps there’s some manner of half-way house there at some point in the future?
But I’m in the same boat as others here; my 4,000+ item database will just produce noise unless there are tools and UI mechanisms to trim the data being plotted, and that demands an interface to be able to built those trimming controls and criteria, and also a mechanism to be able to save and edit them later for recall. Is this a new type of document that can fit in existing groups as an ‘overview’ or is it something at the UI level alongside smart groups? There’s a huge amount to consider to not make a hash of this funcitonality, however tactile and visually compelling it looks.
I’ve found a good manual alternative is to create mindmaps (I’m using MindNode and Tinderbox) which either link to groups as nodes in the map or to imported notes, as the tool allows. Mindmaps generally impose a single root, and the structure is static (or at least, requires manual maintenance) but both these tools can be previewed natively in DEVONthink and the links within them can link back to different parts of DEVONthink, whilst allowing (manual) interconnections to be overlaid as a superstructure on the main maps.
This is not to pour water on the idea, but to point out ① there’s some complexity there and ② sometimes these things can be achieved quite well manually.
However, thanks @benoit.pointet for the link to Sigma.js, which will be useful for my academic pursuits!
2 Likes
I think some of these concerns are addressed in my previous post:
Being able to adjust the maximum degrees of separation would make the graph less unwieldy and highlight the “local” context of related notes. E.g. at two degrees of separation, only show notes that are directly linked by the current document, and also notes that are directly linked by those notes.
As I envisage it, the purpose is not a visualisation tool for the sake of it but to allow the user to explore the local context of related notes and hopefully make new connections in their thinking. It also would facilitate being able to traverse through a certain line of thought of related notes and discover new connections, by clicking on a node and going to that document, and then having the node graph update centred on that node.
This obviously depends on note taking Workflows similar to the Zettelkasten/“personal wiki” approach where one is explicitly creating links to related notes.
7 Likes
Interesting!!
I literally google and find this page after watching Obsidian’s Youtube video. But disappointed to see DEVONthink don’t have this function and it seems the company won’t want to add this function at all. So disappointed.
I think the graph view is defiantly helpful to analyze the database and linking nodes.
I do not think that we are speaking about connecting and presenting all possible documents in a db. Not all documents have links to each other per definition. So I might have 10000 items in a db but only 100 might be interconnected via wiki-links or devonthink links.
We could envision multiple avenues to explore:
a) folder support. Majority of tools like obsidian are targeting a zettelkasten principle, so all the files are inside one folder. It might be an idea to provide this functionality only for one folder (with nested folders) inside a DB
b) on a db level. In devonthink way of thinking this actually makes 100% sense to me. Create a separate db for all my interconnected zettelkasten-style notes. Go into DB setting and enable “graph view”. Boom magic happens. This graph could be leveraging a combination of a) wiki links b) devonthink links c) aliases d ) combination of either of those. That is actually the power of devonthink with that broad level of tools which other products do not (yet) support on the market.
With very mature search capabilities of devonthink, your product would be able to integrate additional information from the graph into search.
Integration with tags, labels could be considered to overlay this on top of a graph. One could see e.g. which areas of the knowledge graph belong to which “topic”/label.
Level of connectedness between different ideas could be calculated. Let us say i want to find “my ideas which impacted at least 10 other ideas”. Just put into the search - and it will be there on the graph.
Interesting example of this: http://gordonbrander.com/pattern/
A great source of inspiration for network theory and a source of illustration above is this book from Cornell.
The more i think about this the more sense the famous Charlie Munger “Latticework of knowledge” makes sense to me with these tools. Exciting times!
3 Likes
Can you give a practical example of how you can use this information?
just a little remark to throw into the discussion:
for me the visual representation of the knowledge (data)base is also something very important.
but I also think one has to acknowledge there is a real bifurcation in cognitive styles/preferences/cultures at the heart of this. generally DT adheres to a text + list model. personally, like some others, I am in the camp of having both, these text-list-db structures and the visual field as information/knowlege space. but a look at PIMs and other dedicated software in this field clearly shows there are two cultural tribes. – I think acknowledging this is better than to argue what is relevant for/wanted by everybody etc.
having said that I am also feeling the lack as I belong to the second tribe. though this might not be of practical help here in terms of node visualization, I can say that the lack of it drove me to a parallel use of other apps to cover this kind of visual knowledge tool function (alongside DT):
1 - Personal Brain; obviously one of the main alternatives for some people to the DT approach; and in itself actually congenial in terms of the ‘visual/logical representation problem’ that @benoit.pointet pointed to (especially the systemic mixture of hierachy and a ogic of lateral nodes makes this much more digestable (visually/cognitively) than full-blown graphs representations.
2 - Trilium (GitHub); this one I only found recently, and it is actually quite close in some aspects to the DT-db/notebook-aesthetics+logic. but it also implemented a graph alongside the hierarchical note tree. while DT is my intelligent info-dump, Trilium helps me with much more focus-oriented knowledge work – not least by its visual graph supplement
3 - then there is of course also Tinderbox: I think it is also an obvious candidate for the ‘second tribe’, and (like many others) I frequently bounce back to it… just to give up on its idiosyncratic behmoth nature after another half an hour… 
2 Likes
I get it that there are two schools of thought and that’s great to have different models for different people.
But I am curious if the node graph of document links is is theoretical tool hopefully of use someday, or if someone can explain a specific case where that graph is helpful in a practical sense.
I would say this is not simply a visualisation tool for the sake of it, but it allows the user to explore the local context of related notes and therefore serve as a note navigation tool and creativity aid during an active note-taking and reviewing process (e.g. Zettelkasten).
If the node graph is kept updated and centred based on the currently active document then this would provide a local map that one can click on to go to related documents or documents that are tangentially related. The graph would then centre on the newly opened document, which facilitates further exploration.
This allows one’s knowledge to be more accessible for review as these indirect links may not be obvious unless exposed, which facilitates creativity by allowing new connections to be drawn for which notes can be updated and new notes can be made. This could also highlight areas where you may have deficiencies in your knowledge, e.g. notes that are on the ‘frontier’ and do not link to other notes.
Having this built right into DEVONthink would really facilitate the interactive nature of this, as I think it really needs to keep updated around the local context of the currently active document, and should update as you keep adding new notes and establish new connections. Hence, scripts that output a static representation of the graph do not deliver the full potential of this concept.
I don’t think it is a coincidence that the two ‘next generation’ note-taking tools that are causing considerable buzz (Roam Research and Obsidian) have such a node graph as an integral part of the user experience. DEVONthink would have a real competitive edge over this for those who need to work with different filetypes, and the AI functionality is a natural complement.
7 Likes
I am also very skeptical of the information value to be gained from viewing force-directed or other graph types that show related documents – which is what is being requested in this thread. There is declining value as the number of nodes and edges (relationships) increase. There is almost no value if the nodes represent documents especially if the edges are calculated based on an unconstrained concordance. If you based edges on a pre-defined stop list of words then perhaps there is value, but what I’ve seen demonstrated above is a concordance-based blob that has little semantic meaning.
The sort of auto-created graphs described here, suggested above for DEVONthink, present in Obsidian and Roam, etc., are, sorry to say, eye candy. I seriously doubt they have much more than passing value for the preponderance of DEVONthink users and as such have no competitive edge at all.
(FWIW, competitive edge is gained by unique and unreproduceable or proprietary features – not by using off the shelf tools that anyone can use. There’s a reason, I suspect, why DEVONthink has never had a graph feature that has been well-known in hypertext and graph theory communities for decades: no one wants it.)
2 Likes
Certainly there are issues with presenting the whole graph, and some users have made suggestions to make this more manageable. E.g. To only show nodes for a given level of degrees of separation.
One problem with the illustrations in this thread is they have focused too much on showing the whole graph in an unconstrained and static manner, which has limited value. That does not mean there is fundamentally no value in the concept, but like all good software it is a matter of designing a thoughtful experience that meets the needs of a given use case.
This is not a feature that would apply to all workflows, rather only those who use the note taking capabilities of DEVONthink and the item links / wiki links features to explicitly link notes together in a methodology such as Zettelkasten. I would assume that this does not apply to you, hence this thread is of little value to you. In which case, that would explain your obvious passion in shooting down the concept rather than exploring whether there could be benefit to certain users if delivered in a thoughtful manner.
3 Likes
I expressed my free opinion without attacking you personally and demeaning your intentions, @Jun_GL. Please be civil.
1 Like
I have to agree with @korm on the value of force-directed graphical UI. Perhaps I am just old school, I ran away from thebrain app after using it for half an hour. I find self-constructed concept map (works best with a pencil/Apple Pencil and a writing pad/iPad) is perhaps more helpful in organising my ideas coz I am forced to organising my thoughts in my brain first!
I am not trying to vote for or vote against the suggestion. But I wonder if there are many users that have actually used the roam research or whatever similar apps for a sufficient amount of time and have been using the “node-connection” as the major means of productivity - for a few years as a minimum. I think it may be easy to get excited when a new feature is discovered (which present a new possibility) but whether such feature can boost productivity - in what way and on what kind of tasks - has to be time-proved. A lot of features in DT3 are coming from the accumulative feedbacks of longterm users who have been using DT2 as their main productivity tools. I hope those who praise graphical UI and/or node features in other app can share more of their long-term usage experience and that will make a more convincing narrative. Only time will tell which app/feature really matters - for example people won’t stop praising KM, Hanzel (the DT’s answer to that is smart rule - I think) way after the initial excitement cycle.
Having said that, I am still excited to see that node UI can actually be done by other forum members - it sure is an exciting programing job 
.
Thx to @Jun_GL 's and @jooz’s inputs I realize my approach was forcefully unproductive: I started exploring with the graph analysis perspective. I was curious of generating the big picture. But often this does not convey a lot of practical information, only the forementioned blob effect.
So yes, I agree, focusing on the relationships between the items witin a set (eg. a group) seems more meaningful.
Use cases
I now see several use cases where visualizing the relationships within a set of items would add value to the existing « list view modes». Again, I focus on use cases I face to make them as tangible as possible, rather than theorizing.
1. Visualizing wiki notes
I have some wiki about a topic, with 40 notes inside. They’re all enclosed in a same group. In general, each of those notes hold 3-8 wiki links to other notes in the same set. Currently, DT allows me to see the set in mode Icons, List, Column, Coverflow. There’s currently no way to see how, within the set, they relate.
Visualizing this set as a network graph would give me the opportunity to
- identify unlinked notes (orphans)
- identify the note that’s got most connection (centrality)
- identify groups of notes in this wiki that are tighly coupled (clusters)
2. Visualizing a long-read and my stuff about it
When I read ebooks/pdfs, I take quite some excerpts (of text or illustrations) from it, and tag them more precisely that the original document itself. Those excerpts point (through a x-devonthink link in their URL field) to the page they come from. Sometimes in the excerpt I do in-text links to another excerpt, because they relate.
Then I jote down thoughts, questions, etc. as markdown docs which often point to these excerpts. So this is what I end up having in a same group about a long-read:
- notes and thoughts which point to others notes or to …
- textual and visual excerpts which point to other excerpts or to …
- the original document
Having it all in the same group helps me to manage it all.
But there’s no way the current list view modes are going to help me make sense of the structure I web around the original document.
Visualizing this set as a network graph would allow me to see how they relate, distinguish who’s who in this set of (often) more than 50 items.
3. Visualizing my topics network
Even a single item is rich of relationships. It links to some web resource and/or to other items, it is tagged and/or replicated. A concrete use case: my “professional knowledge-base” contains 1000+ topics (represented as group tags). I do heavy use of replication because a concept my pertain to many theories, a tool or a construct to many methodologies. The inspectors and list view are great to show the topics a topic relates to (aka parent and child relationships, but not in a synthetic nor uniform way.
Being able to focus on one of these topics and seeing its multiple parent- and child-relationship would be way easier.
4. Visualizing the context of a set of items
Remember I said that I tag excerpts and notes about long-reads more precisely than the long read itself (use case 2)? That is a point of interest: this graph of « 2nd-degree tagging » (i.e. how I tagged the notes and excerpts that link to that long-read) provides a richer view of the topics covered by the long-read.
In a more general way, a set of items rarely has zero relationship towards its outside, rarely self-contained. It can have tags, links internal to the DB, etc. These links to the context are a very rich source of information.
Outline of a solution
The contributions in this thread opened my eyes that the most meaningful exploration would be to stick to DT information architecture and UI concept. In other words, to dream from within it.
DT IA, summarized
Let me briefly introduce how I describe it to then be able to speak of it.
A DT database contains items, which are either
- documents (PDF, markdown, …) or
-
non-exclusive sets of items, among which
-
static sets
- groups
- tags
-
dynamic sets
- smart groups
- search results (not properly belonging to the DB, but useful to fit search result somewhere in this IA)
-
static sets
The list view pane in the central column of a DT main window is there to represent sets, let use navigate it and select or several items in it. And this regardless of the type of set: group, tag, search results, smart group.
Principles for a network viz within DT
- A network visualization of a set of items (and the relations within it (and potentially to its context)) should happen in this list view pane as well, as an alternative list view mode.
- The concepts of set and selection should also be translated into the visualization (as visual boundary, respectively highlighted nodes).
- Another consideration is the use of colors: colors are used to mark nodes. So the network graph should make a use of colors coherent with its current usage.
How it could look like
Here’s a mockup of how it could look like — on the basis of my use-case 2: the long-read — and what options / widgets it would, imo, require.
I apologize in advance if that act seems pretentious to some. As stated earlier, I intend to contribute to this discussion through exploration and ideation, in the hope that it sparkles a little something somewhere …
15 Likes
Thanks @benoit.pointet. I think you provide a good overview of the current challenges that exist with simply having a list of notes, and what the benefits are from a node graph representation. Certainly you do a good job of describing what the workflow possibilities / UX limitations addressed by this are, which is not obvious.
I would also add that this addresses another issue: DEVONthink currently does not show backlinks, i.e. documents that link to the current document (I know there is an Applescript that adds this to custom metadata but this is a bit of a hack that has its own challenges, as well as the limitation of a list presentation that @benoit.pointet has described).
I think your mockup also does a good job of showing what the value of this approach could be if done in a measured way, centred on the currently active document rather than the whole network. I would suggest that under the visualisation options it ought to be possible to adjust the number of degrees of separation for folders where there are lots of notes (suggested default of 2 but can be increased so the user can set the balance of greater context vs information density). It is fine in your example as there are only a maximum of 2 degrees of separation currently.
I accept not every DEVONthink user will benefit from this approach as not everyone uses the Wikilinks / Item links functionality extensively, but for those who do this could be invaluable. The new book (“Taking Smart Notes with DEVONthink”) certainly shows DEVONthink is adept at handling this kind of workflow / approach (even if it is not fundamentally designed to do so). Also, the ideas explored here would be a complement to existing functionality and hence would not alter the character or function of DEVONthink for those who do not use this workflow.
5 Likes
Thx for pointing those elements I did not highlight.
Worth repeating, thx!
1 Like