How DT Could Replace $10,000 software

DEVONthink Feedback
1

DT does well as an outliner which works for me in most projects which do not have a highly complex structure. Sometimes, however, I have no idea how things fit together and I rely on a visual organizer such as Omnigraffle or Tinderbox to help. The problem with those programs as that they are woefully inadequate as databases to store large amounts of varied data, relying mostly on “notes” instead of actual web pages or documents.

So, once I realized that I could link an Omnigraffle shape or connector to a DT2 folder or item by way of the new DT2 hyperlinks, I was wondering if I could somehow export a DT2 outline to Omnigraffle. I created a simple database and then exported as an OmniOutliner document and I was pleasantly surprised to get this result:

Wow! This is an incredible start and it could serve as a means for visually organizing a simple database. The problem is that simple databases don’t need this kind of organizing and there are a couple of things which need to happen to make this useful in those cases:

  1. Each exported item needs to have the DT2 URL assigned to the corresponding OG item. I can’t imagine doing this for hundreds or thousands of items.

  2. There needs to be an option to export only groups. Some users might want to look at folder contents in the diagram but with large databases, there might just be too many items inside the folders and I myself would prefer to see only folders in OG and then click on the linked DT2 folder to see the actual items.

However, there is a much larger issue. Let’s say we add a fifth item to the database and one which is linked to both Joe and John. With DT2, one would have to add this to one of the folders and then replicate to the other. This is the ideal result I would expect after exporting to OG:

This, however, is the actual result:

This is not trivial. In addition to the unnecessary duplication and associated logistical issues, somebody relying on this diagram might not realize that John and Joe are “linked” and at this risk of over dramatizing, a criminal or terrorist connection could be overlooked for those who rely on such link charts for those purposes!

It should be clear from previous threads that the problem here is the same one discussed earlier- that DT2 relies on an outdated metaphor for storing items which needs to be replaced with the ITunes/Gmail tagging system. The implications are profound for DT2. Trust me, police and intelligence agencies pay thousands and sometimes hundreds of thousands of dollars to get this kind of capability. I for one worked with a product called Orion Magic which cost about $1000 and tried to do exactly what I have outline here. It had an outliner which was tied to a link chart creator but it never worked well and the product has been abandoned.

There are so many things which could then be implemented. For example, one could set up a filter/alert system so that any item going into the “Library” could be tagged according to a set of criteria and then automatically assigned to the correct groups. That would essentially amount to an ongoing search which is sometimes neglected or overlooked.

None of this will be possible however unless this fundamental flaw in the DT file storage is addressed. It shouldn’t be that difficult-- implementing a tagging system and then allow groups to be created which represent nested tags. As already discussed, items would be stored in a “Library” and then either tagged directly or by dragging into the appropriate folder (not too different from a smart folder I guess).

Trust me, if DT had the capability of creating proper link charts as well as a database to store the information related to the chart, you would get ALOT of interest from government, police, researchers, etc who need this capability. As I said, right now you have to pay thousands to get it and the products are no where near as flexible as DT. I also think the product could take users away from Tinderbox which is an innovative product but is stuck with an antiquated interface.

1 Like
2

This is an interesting program which contains some of the tagging functionality I am talking about:

http://web.me.com/jonstovell/Tag_Folders/Tag_Folders_Home.html

Unfortunately it doesn’t seem to allow the creation of collapsible, indented folders so it makes outlining pretty impossible. I think the problem is relying on Spotlight comments and Smart Searches. Its kind of jerry-rigged.

Still, the idea seems right.

3

Is nobody but me interested in visually organizing information??

1 Like
4

What a nice complement!

Anyway, those of us who would like to see this visual element made possible should make our voices heard and we need the two changes I identified to make it viable. I cannot believe the devs would not be interested because I think this would open up a whole new market for them

5

One of my frequent suggestions (repeatedly nixed by Christian with good but IMHO arguable reasons) has been a plugin API and the ability to hook into certain functions of DTP, like data visualization. My frequent examples are Graphviz and TouchGraph. Or someone bright might make a new mind-mapping plugin or outliner plugin. (If all works out for me, I might be going back to school to get a BS in Computer Science. I’d love to start really getting into Objective-C in my off-hours, and I think a great way to do that would be to code plugins for my favorite app).

Many people will answer a request for data visualization in DEVONthink with “Well, I don’t need it, and 75% or more of the userbase doesn’t need it, so just use some other piece of software.” That’s fine. Now, the power of an application like QuickSilver is that BlackTree didn’t code the hundred or more plugins available for the application. Quicksilver by itself is just a launcher, and even with plugins is limited to basically replicating Spotlight’s functionality with a more powerful interface with the crucial addition of allowing the basic items (files/messages/contacts/applications) to be used as objects in a long chain of instructions.

Yesterday I referred a user with a comment to Path Finder 5. He already used it, natch, but that’s irrelevant. One thing that I’d really like to see DEVONthink take cues from is Path Finder’s UI. I see this conceivably fixing all of the UI issues that trouble me about DEVONthink (tags, hateful info panel, etc), as well as UI issues that will be presented by things like the extensible metadata.

If you’re not a Path Finder user (I’m not), check out this video and note the following:

  • the “breadcrumbs” ( > Leopard > Users > etc). Someone else has requested this, and the widget is available in XCode, so I’m sure the developers have considered this. Why have breadcrumbs on a document display window when viewing a folder simply displays a useless “No Selection” placeholder? Because I’m hoping that DEVONthink will replace that placeholder with something useful – for instance, provided a proper plugin, a mindmap or outline view of that directory – a virtual document stored outside of the internal Files.noindex structure. Being able (again, through no DEVONcode, only through code by plugin developers) to create a network of mind maps or outlines linked to specific folders and displaying the content of those folders would approximate the most powerful and useful (read: non-eyecandy) features of a Zoomable User Interface (ZUI). (It appears the Raskin Center, where I usually refer people for demos approximating a ZUI, has been hacked) See the below note on for how we could make the document pane immensely useful when a folder is selected.

  • Path Finder’s tabs look fantastic. I hope to see many or all of these features appear in DEVONthink, but I’ll be patient :slight_smile:

  • IIRC, something like a bookmarks bar has been suggested for DEVONthink, but I can’t find the thread. Path Finder’s implementation appears quite excellent – especially if you could bookmark “objects” such as a specific part of a document (like a scroll-to position) or bookmark scripts or Automator workflows stored within the database (the Script menu in DT currently does this very, very well for such things stored outside the database). An idea would be to again rely on my oft-suggested capability for an embedded scripting language within DEVONthink that would allow linking to something like ```

x-devonthink-command://RUNCOMMAND( RECORD( “0a79b080-ddb1-11dd-ad8b-0800200c9a66” ) )/



*  the modules (~40% through the video).  Essentially, you click on the document title and the information can be displayed/handled in multiple different ways.  For some file formats (.eml, .rtf, etc), we'd have a "DEVONthink" view as well as a Quicklook view.  If some enterprising chap were to do so, you might have a plugin view for, say, OmniOutliner files, that allows things like conversion to rich/plain text and text copying that the Quicklook view does not currently allow.  Also, this seems to me to be a good place to locate a new version of the "Show Info..." panel with more details and support for things like tags and extensible metadata that the current panel simply cannot do effectively.  If you need to edit a record's metadata and content at the same time, all you have to do is open a new window.  (Have I mentioned before that I would really like to see "collaborative editing" in DTP, so that you can edit two components of a record at the same time without the risk of losing information?)  You could also have a module for version control (important for many people, and might streamline the PDF OCRing process).  At the folder level, you could see a view via a plugin (mindmapping, outlining) or an icon/list/column view of the contents, or the information for that folder and/or its contents (like a combination of "Database Properties..." and the Info panel), etc.  At the smart group level, you could edit the criteria in a larger and friendlier interface that might even incorporate a realtime view of which files are/are not included by the current criteria and does not monopolize the main window.  


*   Also possible through this multiple-view functionality is the ability to quickly and fluidly define the conditions for class inheritance in records -- I could, say, create a new class for all PDF files that says I want them to always be opened in Skim, and that class would be editable when you open up any PDF file simply by selecting the proper module when viewing that document.  This would provide an enormous increase in functionality, fine-grained control of the application's behavior, and could likely rely on a combination of current smart group/saved search technology and user-extensible metadata.  The initial class could be created by making a smart group that applies conditions (additional metadata, associated files, a particular way of handling records) to the items that match those conditions.  This could also be done (as in other programs like Tinderbox, for instance) through a specific item of metadata like "RecordClass=ARRAY(45b5b7b0-ddbc-11dd-ad8b-0800200c9a66)".  A direct link to the applied conditions of the class would be available when viewing that record (as well as when viewing the smart group itself).  This would unlock, as I said above, enormous gains in control over the application.  For instance, let's say that you're a comic book collector and want all comic books of 20 pages or fewer to be viewed in Quicklook and all those with more than 20 pages to be viewed in SimpleComic.  Create a class and you can do that automatically.  Similarly, you could open all text files with a certain label in Word/Mellel/whatever, and others in TextEdit, or have a specific text file updated with your word count and other statistics whenever you save any RTFs with "draft" in the name, or when you add a new Address Book contact and import it into your database, any emails you have to or from that person can be moved out of your general email archive and into a folder created for that specific person.  When I say enormous, I mean [b]enormous[/b].


*   Note also how Path Finder uses drawers -- somewhat similar, I suspect, to how DEVONthink could further improve their drawers.  If you want everything in one window, you could simply add a similar widget controlling the module display for the drawers and have metadata/version control/see also/classify or whatever customarily shown when you open the drawer.  (Conceivably, you could also switch between case-sensitive and case-insensitive versions of "see also" and "classify")  Some people don't like drawers -- my philosophy is that if it means $FEATURE gets implemented sooner rather than later, and with fewer bugs/display glitches, I'm all in favor of drawers.


*   Customizable context menus are just plain neat.  DEVONthink's contextual menus are generally excellent, and of course far more complex than anything a file manager has to deal with, but I just wanted to say that this feature is @#$%ing sweet.


I know this goes considerably off-topic (and I started babbling about things that are completely separate, like class inheritance and metadata and internal scripting support) but the essential point is still valid, IMHO -- that there are two ways to go about this: bugging DEVONtechnologies to implement many complex features for which they don't necessarily have any experience or enthusiasm, and which in fact may be completely counter to where they want the app to go, or bugging DEVONtechnologies to implement a single (admittedly massive) powerful feature that will allow the users themselves to augment the application's functionality.  Everyone benefits from a wider range of functionality for the application, particularly when that functionality is optional and does not bloat the core application.  

DEVONtechnologies (or at least Christian) believe that the plugin API would massively increase their support load (due not just to the difficulties of plugin development but also the risks of running code written by someone like me) and time spent doing documentation.  My suggestion to counteract that is to make the API completely unsupported and to add an option to DEVONthink -- for instance, something like how iTunes handles switching between multiple databases.  Opt+click the icon in the dock to load DEVONthink without loading third-party plugins.  The database is and should remain proprietary, so plugins should not have raw access to the database.  

I'd cheerfully do the documentation for the API.  If nothing else, it'd be something to put in my portfolio.

(What I really want right now is a million bucks so that I could just flat-out [b]pay[/b] DEVONtechnologies to make DEVONthink Pro: Kalisphoenix Special Edition)
6

An Addendum: I’m not particularly opposed to the current file metaphor for general purpose computing and, from what I can see, it doesn’t look like this is going to change any time soon. So far as I know, Apple’s filesystem plans do not extend beyond a probable future move to ZFS which, while cool, does nothing about the metaphor. I do, however, dislike managing my information – my mind, really, in electronic format – according to the restrictions of that metaphor.

I see DEVONthink as being able to transcend the file metaphor by allowing considerable abstraction. This is where the AI comes in – it’s nothing compared to a human brain, natch, but there are things that I can see making it more useful as it continues to develop:

  1. Blur the distinction between “folder” and “file.” This has been suggested by myself and others in the form of a “default document” or document associated with a given folder, or by allowing documents to act as containers for other documents à la Scrivener. This seems to me to be accomplishable through a UI trick and controlled through metadata hooks (Add a "DefaultDocument=RECORD( “5bcee020-163e-4e5b-ba56-27be6b72dc8b”) key-value pair – the default document conceivably would not even have to reside in that folder).

  2. Blur the distinction between “part of file,” “file,” and “series of files.” As with the folder-file distinction, this has been blurred by Scrivener in the form of Scrivenings, or groups of documents merged into a virtual document, and by Tinderbox’s ability to “explode” notes into subnotes given a proper delimiter.

I would like this distinction to be more fundamentally blurred, though – not just in terms of presentation, but in terms of how the AI handles information. This would almost surely require improvements to the AI, and likely would require increases in database size, search execution time, memory usage, and so on. Essentially, this would enable DEVONthink to construct an entire train of thought. Remember Steven Berlin Johnson? What if the examples he cited in his usage of “See Also,” instead of requiring repeated applications of that command, were constructed on-the-fly and merged into a virtual document? Or take a passage out of a book and execute “Build New Database from Document” – and it would create a database with an entirely new document hierarchy, all elements auto-classified, from paragraphs of the documents within your current database? Or if you created a document hierarchy automatically and went through it, deleting uninteresting or unhelpful suggestions, “teaching” the AI’s intuition rather than modifying its initial assumptions (as in the current system)?

An interesting thought experiment, but likely impossible at this point in time… but even without improvements to the AI, this blurring of boundaries would be immensely useful to anyone who has had to explode an encyclopedia or dictionary in DEVONthink, or to anyone who wants a dynamic document, never out-of-date, of the entirety of their novel/address book/hit list/whatever.

  1. Blur the distinction between “metadata” and “data.” This might be controversial, but I think it bears some thought. Part of my thinking regarding the internal scripting language or whatever is that an RTF document like a form letter can use tokens to reduce processing time ("%%CurrentDay%%",etc) and, through the same fundamental mechanism, you can create a powerful HTML record view for sheets ("%%Interests%%", where “Interests=RECORD(CurrentRecord).Interests” or something with two images linked to the next/previous records, etc). This is, in a way, inserting database/spreadsheet functionality into completely normal documents that would remain human-readable.

  2. Blur the distinction between databases. DEVONtechnologies, I believe, is doing this marvelously – searches and smart groups can span multiple databases. There are some UI issues that I’m sure they’ll iron out, but they are largely inconsequential. The ability to virtually merge the AI of the two databases might be an incredible feature, especially considering my AI thoughts from #2. I don’t know what or how much is currently being done in this area, though.

1 Like
7

I guess it’s now my turn to be nonplussed at your lack of response, sgmiller :wink:

8

Sorry :smiley:

I guess I must have missed the notification on this but I will process when I can come up for air!

9

I can quickly reply to this.

When my girlfriend says 'You don’t NEED that", whatever it is, I always reply “you only NEED foot and water, everything else is a luxury.” First of all, if 25% of the user base already already knows they want it, that would seem a pretty big demand but if not, one should remember that the rest might not know they NEED it because they don’t have the opportunity to see its use. I doubt that before DT, the world in general was clamoring for auto group and threatening mass suicide if they didn’t get it.

I suspect that if DT offered what I have suggested, much of the existing user base would explore it and a large percentage come to see that they NEEDED it all long. In addition, the user base would grow as people who know they NEED it jump on board.

In the case of data visualization, we are so close and only require the two steps I outlined earlier to make DT compatible with OmniGraffle. Granted, the URL is the easy one while changing the whole file metaphor vastly harder, but I would settle for just the first and be willing to hand re-do the common parent links if need be.

As to how it is done, plugin, etc, I am indifferent as long as it is done.

10

This is a highly interesting discussion. Either one of you ever checked out Personalbrain? (thebrain.com). Now what if we could combine the structure and storage of DTO with the graphics of Omnigraffle and the visual dynamics of PersonalBrain. I would be in heaven! Point is, sometimes I need structure, sometimes I need graphics and sometimes I need to jumble things around. There’s no either/or.
:mrgreen:

11

I have in fact played around with Personal Brain although it wasn’t too much too my liking as I remember. As I recall, I found it too confusing to navigate.

12

This is a very interesting conversation. I hope it continues.
Nick

13

I have to agree with the concepts here. I’ve been struggling for a way to integrate the DB functionality of DTP with Curio (zengobi.com) which is a visually oriented notebook/data organizer/mind map program.

14

I just migrated my note system from Journler to DEVONthink a week ago, and I found this forum thread by searching the user forum for the terms network visualization. Knowledge visualization is one of my research areas, and I am very interested in visualization of my note system.

Some of what sgmiller wrote in the original post is now out of date, because DEVONthink now has excellent tagging that is well integrated with the operating system ever since OS X 10.9 Mavericks.

One idea in the original post that is still lacking in DEVONthink is visualization of links: not only visualization of categories, as the original poster discussed, but also visualization of direct hyperlinks between notes.

Journler was superior to DEVONthink in this regard: In Journler, whenever you created a hyperlink in (for example) note 1 that linked to note 2, whenever you then opened note 2 you would see a list of all the other notes that linked to note 2 including the backlink from note 2 to note 1, and under that backlink (as a folded outline) you would see all the other notes that linked to note 1. In Journler, this backlink functionality also applied to attached files: when you attached a file (for example, an Excel file or a URL bookmark) to note 1, you would see a list of all other notes that linked to that file. DEVONthink’s see-also list is cool and I hope it will be helpful, but it doesn’t completely substitute for explicit lists of backlinks.

Network visualization of links would be great, but I’m not surprised that DEVONthink doesn’t have such visualization. I hope to find a little time to explore how to visualize my note system outside of DEVONthink. I’m already a big fan of the open-source issue-mapping database software Compendium, but I see no way to integrate Compendium’s database with DEVONthink. Inspired by Douglas Barone’s philosophy of the file system infobase manager, I’m trying to manage my note system so that it is as independent from DEVONthink, or from any other proprietary software, as possible; I’m just indexing external files with DEVONthink.

One method of network visualization, noted by the original poster above, is Graphviz’s DOT format, a plain text format that can also be read by OmniGraffle, for example. Or there is the more flexible, but no more difficult, set of graphdrawing macros for PGF/TikZ 3, which can be used from LaTeX or TeX. There are also some graph drawing libraries for JavaScript and HTML, such as Sigma.js, but the amount of programming required may be beyond my current abilities. Since I already use OmniGraffle to create my own project network diagrams, it may be the easiest tool for now.

Although visualization is nice to have, it’s not necessary for getting things done. I always try to focus on, to use the title of a relevant blog post by Christian Tietze, how to program yourself for productivity and stop searching for the ideal software! The ideal software doesn’t exist (yet)…

15

I’m currently working on integrating DTPO and TheBrain (the former PersonalBrain) for a particular project. This involves indexing folders of files in DTPO and using TB to visualize data using links to connect thoughts in various ways. (A thought in TB can contain a note and any number of files or links to files. The universe of thoughts is a plex.) I would use DTPO’s AI to find relationships between files and TB to manually roam the plex (the visual part of the data). The files are stored on DropBox and TB references files using DropBox specific links (I don’t store files in TB or in DTPO). I subscribe to TB’s web hosting, so I can use both my Mac and iPad/iPhone to access a brain plex. I’m waiting for Devon’s next generation of software to see how it will handle indexed files when using desktop/mobile connections.

I’m still not sold on the DTPO/TB integration since there’s some duplication of effort, but time and use of the “system” will be the final judge. It’s all about the look and feel.

Most of the files used by the system are PDFs. I use PDF Expert in iOS (also on the Mac) since the new iOS version can define “sync” folders in DropBox that automatically download a PDF from DB and save it back after editing. On iOS I can roam the plex and “Open in…” any PDF file in Dropbox to view or edit using PDF Expert. I’m hoping the new DTPO/DTmobile will have similar capabilities. If so, then I’ll be almost there.

16

The thread titled “Topic Maps, Word Clouds, etc. – DocEar” is an interesting discussion on this topic.

The Hannah Jacobs “New Woman” project and the discussion of how it was created remains a very interesting example of using Devonthink as part of a visualisation workflow.

Frederiko

17

Thanks to pvonk and Frederiko for the responses! I like what Frederiko said in the other forum thread that he cited:

I am not an academic researcher, so Hannah Jacobs’ project that Frederiko mentioned is quite far from what I am doing, but I like the diagram of her research workflow in the blog post that Frederiko cited. I find the visualization of her workflow to be more relevant to what I’m doing than the visualizations that are in her final product! The visualizations I am most interested in are closer to project network diagrams: graphs that show how to get things done. My note system is both a personal knowledge base AND an integrated GTD-style project management system.

I recently found a superb paper written ten years ago that provides an excellent overview of these issues (and the paper also mentions DEVONthink). It surprises me how little progress has been made in this area in the ten years since the paper was written. See: Stephen Davies, Javier Velez-Morales, & Roger King (2005). Building the Memex sixty years later: trends and directions in personal knowledge bases. Boulder: Department of Computer Science, University of Colorado at Boulder.