Backups in same package

Program level backups are kept in the same package in Pro. I think this is not such a smart move. I keep three levels of backup and this means my package file is four times the size of my base data - so when I burn that to CD or copy to iDisk I have to open the package and delete the backup “folders”; which seems to work, but feels dodgy (I always do this in a copy of the database, which then gets trashed just in case).

So - the backups need to be in a separate package. Judging by the prefs dialogue, this is coming - but I just thought I’d remind you.

Thanks for the comment. Your workaround with the current database structure seems appropriate.

When the revised database structure is put into place, some of the issues relating to backups may be rethought.

When the revised database structure is implemented, all documents, including text, rtf and HTML files will be held in the Finder, so that Spotlight can index them. It will be especially interesting to see if a metadata tag can be supplied to Spotlight so that the user has the option of opening documents listed by Spotlight either under their native application, or under DEVONthink/DEVONthink Pro. The latter case would be very powerful for many purposes.

I don’t know if I like this or not. I do have quite a few things present in both the finder and the DT database but I’m constantly grabbing stuff and adding it to DT without storing it separately in the Finder now. Part of my concern is the level of clutter that will end up somewhere in the finder. I can’t break my old habit of organizing things by topic and subtopic in finder folders and subfolders, no matter what Spotlight is capable of doing. I just can’t stand looking at it that way (ok, I’m old fashioned but I’ve been doing it this way on Macs since 1984); and I don’t have time to do all the things I’m collecting for DT that way anymore. Anyway, it would be great to have a little more information about how this is going to work.

BTW, I’ve found some great GTD tools I’ve (partially) integrated with DT lately that I’ll try to find time to write about later, but I’m flying off to San Francisco tomorrow for 8 days and I don’t know when I’ll get to it (writing about it is in my someday/maybe stuff right now).

ChemBob

My bit of followup to Bill’s and ChemBob’s offtopicness …

I’ve also had concerns/questions about the aforementioned one-item-per-file DT database structure. But I probably won’t care (too much?) if it’s still possible to encapsulate a database under a single folder hierarchy since that currently has some distinct benefits such as being easy to copy and back up (caveats ignored for brevity’s sake).

I’m not sure I’d always want that encapsulated DT folder hierarchy mirroring DT’s group hierarchy although I’d want some type of hierarchy to avoid any single folder containing “too many” items. I can imagine it being similar to how iTunes and iPhoto automatically manage their library folders on disk (which some people hate) and still allow it to be Finder-manageable (which iTunes/iPhoto unfortunately don’t).

Even though DT’s document/group structure mimics traditional file/folder hierarchies it’s been a powerful tool for making a certain kinds of “content clutter” more accessible and useful to me than through Finder and supporting apps. It’s compensated for some of Finder’s weaknesses and limitations without needing to be a Finder replacement (yet :slight_smile:).

And my interests are quite modest for DT providing functionality that other apps (e.g. outliners) are better specialized for. DT serves me best as a general purpose content management tool, not as a content production tool.

If DT’s revised database retains its value for me, performs well, and enhances integration with the system (e.g. by using Spotlight) then I don’t foresee having problems with it. Every major/minor revision of DT has brought improvements I’ve made use of. So, I’m optimistic that future changes will continue that way unless there’s some dramatic shift of direction or other unexpected circumstances intervene. And I appreciate how DEVTech shares parts of their roadmap, giving us opportunities for feedback like this.

Most users won’t notice that there will be a new database structure in v2.0 as databases will be still packages hiding all the contents. But inside the package there will be a file for every content and DT will create a folder structure representing the structure of the groups (except indexed material which will of course still be located outside of the package and just referenced)

This has at least five advantages:

  1. It’s easier to support Spotlight
  2. A lot of people don’t like databases as they’re afraid that they won’t be able to access their data anymore in the future
  3. DT will use less memory, should work faster and therefore there will be room for improvements in other areas
  4. Improved integration with Mac OS X and other applications, e.g. the ability to edit files externally
  5. All those confusing import/index/link/copy preferences & commands will be reduced to import and index.

The only minor disadvantage will be that copying a database package to another volume or computer will need more time.

The new structure seems to make synchronizing two versions of a database simpler (based on comparison of the modification dates of the individual files). Is this right? Will this feature be built in?

I’ve just spend 40 minutes copying DEVONthink documents between my laptop and desktop, so I’m very interested in being able to automate this synchronization process.

Yours,

David.

Just want to add my support to divine_tar’s request for an option to store backups in a location other than the main database package. I have to go through the same process as he does to backup to CD-RW. In addition for extra security I would like my backups to be stored on my second drive while my main database is stored on my primary drive. To achieve this at the moment I have to manually copy the database to the second drive.

It’s more secure to store both original and backup on your first drive, and back up BOTH to a second drive. I recommend doing that for all of the main user files (Documents, Movies, Music, Pictures, Public, Sites) at the end of each work day. I do that with Synchronize! X Plus, which copies only the new, altered files, and it takes less than three minutes.

I left out Library because some of its files are locked and won’t copy properly.

Happy Independence Day!

Good advice!

I’m running my iMac G5 20" 2 GHz 2 GB RAM on a 500 GB external FireWire drive that I cloned from my 250 GB internal drive. (The iMac runs cooler that way.) I keep all my externally linked (Indexed PDFs and Word files) in one of two folders. Every day I manually copy the DT Pro database and my PDF folders (if changed) over to the iMac’s internal drive. (With DT Pro closed on both drives, of course.) That takes only minutes and happens in the background. So I’ve got two bootable drives, each up to date, if one should go wonky. (In the many years I’ve been using Macs, I’ve never had a drive failure.)

I’ve got a TiBook that I’m keeping in OS X 10.3.9 for support purposes. If I’m planning to use the TiBook, I update the files from the iMac to the TiBook over my wireless network. That also happens in the background. (If I add new content to DT Pro while using the TiBook, I use History to identify the new content, export it via File > Export > Files & Folders, send the export to the iMac and import it into DT Pro.)

And I’ve got another external FireWire 400 HD that I periodically update with important files and store in a fireproof safe in a reasonably nonflammable area of the house. (When I go on an extended trip, I drop that drive off in my bank box and take the TiBook with me.)

Paranoid? Time-consuming? Not really. The value of the information stored on my computers is worth (to me) many times the value of the computers. Bad things happen. I’ve never had a Mac or HD go bad, but it could happen. My little procedures take only seconds to set up, and the routines don’t take much thought. In the few minutes they take to complete, I can answer email or just go get a cup of coffee. :slight_smile:

Bill, that is indeed an elaborate backup strategy but clearly one that works for you. I’ve also used Macs a long time (and Apple II before that) and never had a hard-drive failure…UNTIL…I got my iMac 20" G5, etc, and that drive went belly up in the first 6 weeks of usage.

Fortunately, I had made my nightly backup just a few minutes before the failure. I lost no data at all. Apple replaced the bad drive promptly. Techs at my university shop said they had heard of similar failures, in the iMac 20" G5 models.

So the moral to everyone is trust but verify: if you have precious data on a computer, back it up every night to another drive. They are relatively cheap and quite reliable.

PS, I am fascinated by the practice of running from an external drive and copying to an internal. My machine does run hot, and I worry about that leading to another hard drive blow-out. --Will

Will:

I’m pleased with the iMac G5 20" – 2 GHz and 2 GB RAM – but my 250 GB HD was often running at 55 C or above, which is max or above the manufacturer’s recommended ambient temperature range. (The SMART HD temps were well into the 60s C, usually hotter than the CPU.)

Saw a couple of tips on the Apple Support discussion site about people running iMacs on external drives for cooler operation. As I had just gotten a new FireWire drive, I decided to do that. My internal drive now hovers around 42 C. So at least during the long, hot Louisiana summer, I’ll boot from the external drive. As the iMac is limited to FireWire 400, I’m taking a little I/O hit, but not enough to really notice. The big aluminum case of the LaCie d2 500 GB feels barely warm to the touch.

howarth said:

Sorry howarth but I don’t agree. It IS more secure to have the database on one drive and backups on a separate drive (I’m including a backup of the main database here). Having extra backups on the same drive as the working database is no more secure although it may feel that way. If that drive goes down then all those extra copies are no use.

Bill_DeVille said:

Bill, you’ve just been lucky. I had significant experience with Macs since the ‘Fat Mac’ and I’ve personally used dozens of Macs of various models between work and home. I worked as a Macintosh Technical Specialist for Trinity College Dublin after which I worked for an Apple Dealer. Drive failures are unfortunately common enough and I’m not just talking about pure hardware failures and I’ve seen and experienced plenty of them. It is true that modern drives are generally more reliable but it is still perfectly possible for them to fail in various ways. Thankfully the Mac OS has improved considerably as well. There have been versions of the Mac OS which required running Norton or other utilities weekly to avoid creeping directory structure problems and corruption.

Er… guys and guyettes… I’ve just read all the above and I don’t understand. My original post was about Devon’s new strategy in Pro of keeping the backups in the same package, and about how if you’re backing up to iDisk, CD or Flashstick this was inconvenient; yet other than a post in support from Psmyth and a single line from Bill sanctioning the workaround that I had felt was clumsy and dangerous, we’ve had nothing but discussions on the new file format, on HD running temperatures, on booting from external drives, on backup strategies, drive failures… Hey - it’s my thread! Does that sound petulant? The thing is, it’s exactly because everyone agrees we should be totally over-the-top paranoid about backups that this admittedly minor step backwards in Pro should be reversed. Sure, the file format and drive failures and other stuff is probably more important, and certainly sexier; but don’t forget little old me and my flashstick or Psmyth and his CD.

Hmmm … I like the accessibility of storing all files separately, BUT based on my experience with Hog Bay Notebook (excellent product by the way), the time taken to copy files or databases to another volume or computer is NOT a MINOR disadvantage … it is a major one. When I have 50 million words in my DTPro Database in the not too distant future, it will be a nightmare!!!

Is there another way?

Rollo

Sorry if I wavered off your topic, but I try never to lose an opportunity to encourage backups, or to point out that there are many ways of doing that.

Back to your original post:

[1] The only difference between a DT PE database (contained in a folder) and a DT Pro database is that the latter has become a double-clickable package file with a distinctive suffix. The storage of a rotating set of internal Backup folders inside the DT Pro database package follows the DT PE model. The package file difference is already important, as it allow quick changes of DT Pro databases from the Finder – double-click on one to launch DT Pro and open it, or to close a currently open database and replace it. It will become even more important in the future, with reference to Spotlight.

[2] Because a revision of the DT Pro database structure is coming “pretty soon, now” I don’t think the developers will change the location of the Backup folders in the current structure, as you requested. That would be a very temporary fix requiring development time, and wouldn’t be appropriate for the coming new db structure. So your current practice of making a copy of your db and removing duplicative or outdated Backup folders is appropriate to save space, and isn’t dangerous with reasonable care.

[3] Yes, the new database structure will take somewhat longer to copy over to an external storage device. The internal Backup folders (or their equivalent) will be much smaller, as they will hold only glossaries, metadata, etc. about the actual file contents, which will be stored in the Files (or equivalent) folder inside the package. Those file contents will be stored in the Finder with an organizational structure approximating the groups structure of your DT Pro database, which should delight the more hierarchically inclined of us. (Details, of course, are still pending during the development period.) A number of things users have been requesting, such as synchronization of edited files with the database, will be much easier and simpler to handle. Spotlight will be able to index the file contents of your database. How about clicking on a Spotlight list and instantly opening that document inside DT Pro, with the ability to edit it, search for similar content (much more powerfully than Spotlight), etc.? Dt Pro searches will become still faster, of course, and DT Pro will have lower memory requirements.

Yes, rollo, the time taken to copy your 50 million-word DT Pro database to another drive will take a while longer – the first time. (Fortunately, OS X is multithreaded, so you won’t have to sit and watch it take place. Answer an email while the copy happens.) But the revised database structure will be conducive to using appropriate synchronization software so that subsequent backups will only have to update changes in your database. So I anticipate that external backups will become faster, not slower. It’s even possible to schedule backups, so that you can maintain a good backup routine without much effort.

If you were to decide for some reason (I can’t imagine why :slight_smile: ) to stop using DT Pro, just open your database package and pull out the file contents; they will all be there in the Finder. No export from DT Pro required, as would be required in the current database structure. DEVONtechnologies won’t ‘capture’ your files and keep them from you (unlike a number of other database programs, especially in the Windows world).

When a version of DT Pro arrives that incorporates the new database structure, it will be transparent to you. No action by you will be required during the structure revision.

But 50 million words != 50 million files. :slight_smile:

I’d already thought about using rsync since it has logic to only copy differences between files. In some cases I wonder if that could turn out being faster than current methods of copying every DEVONthink-*.database file in a db package. That could be simulated right now doing something like:

Export a database, copy that exported hierarchy somewhere, make changes in the original hierarchy and rsync it to the copy. Compare the time that takes with current methods for copying a database.

Thanks for the feedback, Christian.

It would be chaotic to do otherwise. :slight_smile:

Do you know at what point HFS+ performance degrades because of too many files in a single directory? And will that degradation then occur in DT (Pro) 2.0 by saving too many items in a single group?

DT primarily uses a hierarchical group organization that mimics the filesystem hierarchy, which can often be more of a limitation than a benefit for me. I try overcoming that by thinking of groups as “virtual” containers, all stored inside the monolithic *.database files. With the revised 2.0 database I suspect that sense of “virtual similarity” will diminish since the database will be a strict reminder of the identical underlying filesystem structure. But I’m less pessimistic about it than that may sound.

I hope future DT development offers more capabilities for creating non-hierarchical relationships between items, regardless of how and where they’re “physically” located. And I’d (still) eventually like to use an iTunes library-like storage model for a DT database, then correlate items into virtual groups/views/hierarchies. A replication-like mechanism could be the default behavior, in contrast to physically moving and copying items as one of the tedious tasks of organizing them. Any number of instances of items could be trivially added to and removed from anywhere in the virtual hierarchy. Deleting items from the master library would physically remove them from the database. With the right GUI that’s pretty much my Finder replacement. :slight_smile:

I use the the “Backup” application that Apple gives with .mac to back up my files. You list the files that you want backed up, and click the button, and it will backup to your CD, iDisk, iPod, etc., as the case may be.

Can anyone clarify what I should be listing to backup ? In my Application Support folder I have two folders called “Devonthink” and “Devonthink Pro”. Does “Devonthink” still hold the backups even after you’ve switched to Pro ? Or should I just list the regular “DEVONthink.dtBase” ? I realize I would have to make sure DT was closed if I did it this way.

Yes. Back up your DT Pro databases, which have the suffix “.dtBase”.

It would be a good idea to run Tools > Verify & Repair and Tools > Backup & Optimize before quitting DEVONthink Pro and performing the external backup.

Once again, Thanks Bill !