DT and the filesystem

DEVONthink Feedback
1

Sorry if this seems like an angry post. I’m just disappointed because after trying so many of the applications discussed in Ted Goranson’s articles for ATPM, DT got close to being perfect for me! But unless I’ve missed something, I won’t be paying for it any time soon.

I’m really struggling to get to grips with DT’s relationship to my computer’s filesystem. It seems to integrate in a completely arbitrary, very complicated way, which is virtually useless to any application except DT itself.

Am I right in thinking this is how it works?

If I have ‘Copy files to database folder’ enabled AND the file I link to is an image or a PDF, a copy is made in ~/Library/Application Support/DEVONThink/Files (by default). That folder is maintained as a flat file, with no relation to the link’s position in the DT database.

This seems to me odd behaviour in two ways. Firtsly, why is a copy only taken for images or PDFs? Surely this is a useful feature for all links! Secondly, why doesn’t the Files folder nest its contents in the same way as the database?

I guess what I’m basically asking, is why isn’t the database completely transparent to the filesystem? Why doesn’t it work like, say, SkinkHunt Notes or iTunes?

When you import a file to these applications, a copy is made in an organized, hierarchy which corresponds to a real folder on the filesystem. When you create a document in either it is not added to some opaque database, but rather to the same filesystem hierarchy imported files are added to.

I can’t see why DT’s very sophisticated and attractive built-in editing features and use of metadata would preclude it working like this. In the case of iTunes it is still possible to index its hierarchy for quick searching and add metadata (play count, playlists, etc.) to an XML file stored separately from the Library. Doing it like this would be at no cost to users who work entirely within DT, and ignore the file system.

As it currently works, DT’s file-handling is virtually useless to me. It doesn’t store half the information I’d add to it in a useful way (by which I mean easily accessible from other applications), and the other half is stored in a big flat folder for no obvious reason.

If I were to use it, I’d just use it as a basic note-taker and outliner, and there are plenty of other applications that do that.It has the potential to be a great Finder replacement–a front-end to my Home folder AND an outliner/editor/manager of arbitrary information–but it’s unnecessarily crippled.

Mike Williams, a very confused potential DT user.

2

I don’t really see why DT should store the included files in an hierarchy inside the DT folder. Look at the DT folder as a document, you don’t bother how Word stores the clipart in a document, why should you bother with how DT stores the information in one of it’s documents (folders)?

3

I don’t use Word ;-> And this is one of the reasons I don’t use it. I want to commit to DT for things other than basic note taking, but through accident or design access to its database by other applications is limited. This makes me very nervous.

Of course Word not doing something is no reason for another application not to do it. I’d like to think development was headed towards something open, transparent and innovative (e.g. iTunes), rather than something baroque and opaque like Word.

Some outliners/snippet managers go quite a long way to being a front-end to the filesystem (Hog Bay Notebook, Skink Hunt Notes), but don’t have all DTs other cool features. Although I’ve never used it, BeOS worked a little like this. It’s not as crazy as it sounds; it’s been done and it works. Refer to the “Multi-level finder integration” section of Ted’s article for more.

Let me explain in more detail why beiing transparent to the file system would be good for DT

  • it would be incredibly useful once your workflow breaks out of DT because all your notes and files are right there in the same hierarchy, accessible to any application that can read them;
  • it allows it to retain information other than PDFs and images in an open database;
  • once you’re done editing the file externally, those changes are reflected in DT at no extra cost;
  • it can work like this, at no cost to people who have no interest in it.

Let me give an example:

I’m currently trying a number of Outliners, making notes on them. Here’s how I’d like DT to help me: I want to drag the application itself onto DT into my Outliners group as I download it, and then delete the original file from my desktop. I want to be able to put my notes on those outliners in the same groups, and attach metadata such as URLs to the applications themselves.  When I settle on one I like, I want deleting an application in DT to at least have the option of deleting the real file (see iTunes).

At the moment, when you drag an application (or anything other than images, PDFs and text files that can be imported) onto DT when ‘copy to database folder’ is checked, it looks through the package, throwing away everything that isn’t text or graphics, and ignoring the hierarchy in its copy.

DT could be, in addition to a note taker, a front end to the filesystem, that indexed and categorized it for you, encouraged you to add notes, and played nice with other applications, respecting their changes to its database. Imagine setting the database root to your home directory! DT then becomes the Finder crossed with an outliner crossed with an editor crossed with a notetaker.

That would be so cool, way more intuitive than how it works presently, so much more flexible. And the crucial thing is that for those who never change database root from ~/Library/Application Support/DEVONThink/ or work with their DT data in applications other than DT, they don’t know anything has changed from how it works now.

4

Hi Mike:

Your example is nicely envisioned.  There’s been some prior discussion on this topic; if you’re curious you might want to check out this link.  (If that link fails, just search the forum for “finder replacement” over the past, say, 365 days).

5

Personally I wouldn’t like DT to work the way described above. I want to keep the file system separate from my “information database”.

6

Hi Mike.  You’ve pretty throughly described, explained, and questioned many issues I encountered when looking at DEVONthink from the “Finder replacement” perspective.

At first I had “Copy files to database folder” enabled but that was more confusing than helpful.  After disabling that it became easier making the database/filesystem distinctions, even if I’d prefer tighter integration between the two.  I’m anticipating some upcoming features in DT Pro will improve on that, but I don’t expect it’ll be as transparent as I’d (we’d) like.  And that’s okay with me for now.

Some of your comparison with iTunes doesn’t seem quite accurate but I’m late for dinner so I won’t get into that now.

In quick summary, I discovered how DT could be useful for me as long as I didn’t try forcing it to be more than it currently is.  And I understand folks who may not want to make “compromises” if it won’t fulfill their needs and expectations.

7

Mike, I came to DEVONthink having a very different set of expectations about what it could do for me. I neither expected nor wanted it to replace or emulate the Finder. Rather, I had been looking for knowledge mining tools that would help me to constructively use the enormous amount of information I already had on my TiBook’s hard drive.

I had thousands of PDF reference documents covering environmental science and technology topics. A fair number of those PDF documents are over 500 pages long. I also had a couple of thousand files of Word, Excel, text, RTF and HTML file types that contained information useful to me.

My problems were [1] finding and correlating information that I already had, and [2] facilitating addition of new information, including online resources from the Web. My expectations were "large" as I once managed a scientific literature computerized information system based on a university mainframe, and knew what might be possible.

DEVONthink is simply awesome as a solution to my problems. And it keeps getting better. By the way, don’t put down flat files! Many years ago I programed a database that could handle statistical analysis of data for monitoring wells (including seasonal reversals of ground water flow). I used Barney Stone’s marvelous programmable database tools for the Apple II. Every field was indexed, and I could use algorithms that would be tricky to work out in a modern relational database (I never could figure out how to make a couple of the tricks work in FileMaker Pro). And it was amazingly fast, as well. Of course, I had to program every pixel for screen and printed reports – I never want to have to do that again.

I don’t put my PDF and Word files into my DEVONthink database. I manage their storage using the Finder. I import their text content into DEVONthink or, for large files, use DEVONthink’s Index feature. This approach lets me use DEVONthink’s search and contextual recognition features (knowledge mining), while preserving the original files’ integrity outside the database (where they are also accessible to other programs, as well).

I do incorporate Web surfing HTML and RTFD files into DEVONthink. DEVONthink has become the premier tool for grabbing information from the Web, especially in conjunction with DEVONagent, which is great for batch search grabs.

DEVONthink is not the premier tool for outlining, word processing or to-do lists. I don’t use it for those things, myself. Those features, while they are available, are not what DEVONthink is about. But if you want to access information quickly and reliably, and without worrying about database size limitations, DEVONthink is the only contender in the Mac world (and I haven’t found a consumer Windows application that comes close to meeting my needs). StickyBrain, NoteTaker, CP NoteBook, HogBay and other Mac programs have their niches and meet real needs, but none of them could manage my knowledge mining needs; they weren’t designed to do what DEVONthink can do.

There are lots of criticisms of the OS X Finder, including the fact that it doesn’t have database functions and metadata like the Be finder, for example. Beyond the most simplistic of approaches, it would not be a trivial matter for DEVONthink to become a “Finder replacement” and I don’t think the effort would be the best use of DEVONtechnologies’ resources. (Even Microsoft seems to be giving up on plans to make the Longhorn OS database-based.)

So I use DEVONthink for its virtues, and NoteTaker for its virtues. They complement rather than compete with each other.

Here are some things I hope to see soon in DEVONthink:
[1] Synchronization with files in Finder folders. If I edit a file externally, I want to see the edit changes automatically reflected in DT’s database.
[2] Metadata - additional useful information about files and their contents.
[3] Tables, tables, tables (and spreadsheets).

8

From what I’ve read Microsoft WinFS won’t be in the first Longhorn release.

If Apple introduces a similar “database filesystem” in the future I’m very interested in how DEVONthink (et al) would utilize it.  Would a database filesystem make the DEVONtechnology data/storage solution “obsolete”, or at least require significant re-architecting? There certainly appears to be overlap there, hopefully to DEVONtechnology’s advantage.

There are many “information management” apps on different platforms, yet it seems they’re missing some underlying, ubiquitous OS support that would better integrate them with everything else.  Oracle, FileMaker, and other traditional databases provide the necessary backends for certain kinds of solutions (which I’m not particularly familiar with) but those databases quite often aren’t optimal and practical, especially on a smaller scale.  My general impression is that you’re fine as long as you stay within their corner of the universe but it gets uncomfortable going anywhere else.

As I’ve mentioned before, I’m quite curious what Bruce Horn’s iFile project (also see comments in this interview) brings to the table.  IIRC he expects to have a preview release available sometime this summer.

I admit ignorance of some technologies currently evolving that may be relevant (e.g. .Net, web services) but take a different strategy.  The boundaries can get fuzzy.

9

I think you would have loved the Newton, it seems to be what you would like to have.

10

Folks,

thanks for all the feedback and suggestions. However, although the integration with the filesystem will be improved (e.g. by copying linked files on demand into the database folder/package or by adding a synchronization command), DT will never be a Finder replacement.

But a filesystem using metadata (let’s wait & see what Tiger will offer :slight_smile:) could be really useful for DEVONthink because it would be quite easy to im/export this metadata (currently one has to implement such things on your own, e.g. read image, IPTC, EXIF, HTML, Word, PDF, BibTeX, MP3, Quicktime metadata etc.) and to use this information e.g. to find, compare or classify contents.

Finally, the "Copy files to database folder" option will be much more useful for DT Pro users as databases will be document packages instead of folders. If you copy your documents into these packages, you can easily copy and move those packages (e.g. to different volumes or computers) without having to worry about missing files or wrong paths. And after adding support for read-only volumes, you could even burn packages ("Knowledge Bases") to CD/DVD-ROMs and distribute them for example.

11

Yep, I’m curious about “as well as upcoming enhancements to the file system” in File System Best Practices for Application Developers session description. :slight_smile:

Thanks for the informative followup.

12

All right! That could certainly induce me to change my file management practices. :slight_smile:

13

Perhaps I am dense in my reading of the manual.  I’m still pretty new to DevonThink.  Some of the authors of the above posts mention DT’s use of meta-data.  Where is this data stored, and how could it be edited.  

14