Do you have any good examples of a “visual display” of a large set of files? One that was useful for you?
I’ve never seen an auto-generated visualization that is does anything better than create a big pile of work to sort out the mess that someone’s visualizer algorithm created. These things work best and you’ll understand your data better, if you build your own visualization. I usually build my own visualizations ad hoc – using my scripts that export to Tinderbox or TheBrain – two great tools that are excellent for this type of work. Sometimes mind mapping apps are helpful – just select some documents and copy to the clipboard. This copies the names, which can be pasted as nodes in iThoughtsX, which also pastes links back to the documents.
What I know about scripting/programming, could fit on the back of an Apple Watch. Just putting that out there first…
That being said - I’ve used mindmapping tools to help myself along (eg Scapple etc), and it certainly has its place. I cannot really speak directly to what you’re querying - but touching on the automation part, what would be fantastic for me (at least) - would be something that extracts the filenames of all documents inside DTPO, that have a ‘wikilink’ attached, together with the file/document so linked. A step further, would be a list of all those rtf/text files where I have inserted ‘wikilinks’, and what/where they point to…
PDF 1 is linked through the /New /From Template/Annotation Note feature, to a RTF Annotation file. Inside that RTF file, I might have a list of wikilinks to other PDFs, or other RTF notes - which in turn, link to even more. I’ve created, in other words, a series of interconnected documents. If those could be pulled/extracted, even into a simple list, that would be very useful, imo. If it could be used to automagically construct a visual mindmap, displaying the interconnected files (names) - really, really useful.
Not really on point, but something I figured I’d throw in.
I should have mentioned that my scripts, and scripts written by others, can be found in the forum and elsewhere. I’m also intrigued by the suggestion and Cassady’s comments. Not promising, but I might take a stab at coding something basic with Graphviz that portrays links.
I’m not a fan of visual mapping. Such a map of my main database that I use for most research would seem no more enlightening to me than a picture of that biggest ball of twine in the world, which someone has on display somewhere or other.
I do make a lot of use of See Also to see suggestions of other documents that are contextually similar to the one I’m viewing, or of See Related Text for a selection of a segment of a note or a draft to see suggestions of documents that are contextually similar.
Once in a while, however, I do click on one of the “keywords” mapped by DEVONagent Pro in a visual map, to see its relation to another term in the Digest view. Mapping of simple relationships is OK. But my mind boggles when I look at multiple squiggles in different colors scattered all over the screen.
I think its worth posting a backlink to this thread (among many) exploring these ideas, although Cassady’s idea merely to graph the references to other documents is interesting and may provide less visual clutter.
The Hanah Jacobs paper referenced in that thread is perhaps the most useful example of different kinds of visualisation I have seen
Personally after spending many hours with graphviz I haven’t managed a result which is actually useful, as opposed to just looking pretty. Search this forum for what other people have achieved with graphviz and python before to map the connections between tags.
The problem I can see from a scripting perspective with Cassady’s idea is that its quite hard to obtain a clean list of all urls in a rich text document with the current DT applescript. It can be done but its very slow as you have to parse every word of the text and filter out duplicates. Its probably a task better given to an external script.
To clarify my previous remark about visual mapping, I don’t personally find it useful for my main database document collection. In that database the potential number of visual linkages of terms or blocks of text would run at least into the hundreds of thousands. Visual representations get too messy and complex to help me think about them. I’d rather do that more abstractly, instead. That’s when I depend on my familiarity with the literature, bolstered by heavy use of DEVONthink’s tools including searches as well as See Also to explore contextual relationships. Sometimes different writers use different terms for similar meaning. On a few occasions the Similar Words button in the full Search window helped clarify such differences, in one recent case Option-clicking on a couple of terms helped identify differences in word usage for similar concepts. When I hit a block (writer’s block) I find that playing with some of DEVONthink’s rich armory of tools gets me going again. No mind-mapping scheme could be so flexible and powerful (and I’ve never found a substitute for thinking, as hard as that may be at times).
But I’m an enormous fan of data visualization of many kinds of numerical data. Financial trends, absolutely! Variations in something like a cost-benefit ratio affected by a variable, yes!
I’ve had to analyze and communicate data in circumstances such as trying to predict the potential for movement of pollutants in subsurface geology, where I’ve got historical data about the levels of pollutants over time in monitoring wells and considerable information about the permeabilities and structural characteristics of subsurface soil layers – and can evaluate risks of off-site movement of pollutants and threats to groundwater resources. (A sand lens in an otherwise tight clay formation can act like a hole in a water bucket.) I’ve found payoffs using visual representations of such data as not only helpful to me, but great tools for communication with the public in discussing problems. But I’ve never seen a mind-mapping tool that could save me time and effort in this kind of work.
In my work, I often need to trace the evolution of a technical idea over time. Questions like, “what is the current understanding of X topic, and how has that changed over the last 5-10 years?”
The usual method for doing this kind of research is a citation index: X key paper cited references A, B, and C, and was itself cited by P, D, and Q.
Being able to visualize those relationships within my database would be helpful, but doing it automatically might not be possible. In my case, most of the source documents are PDF copies of technical papers, and the relevant information is buried in the References section.