Lost changes

DEVONthink Tips
1

I have just lost a substantial amount of work for the second time while using DT Pro (DTP). I am not sure if the problem is with DTP, another program, or with something I am doing. Here is what I was doing:

I have been attempting to find the most convenient way to work with project in DTP. Since I don’t like indexing, I decided to import all files and folders into my DT data bases. Since DTP seems backup only to within DTP on the same volume, I decided to export the database to another volume as files and folders; then I thought I would just have to keep the two in sync manually. However, when I backed up a file from the database to the files and folders using export, the changes to the files were not saved! This prompted me backup the database to asecond folder in the second volume using Personal Backup by Intego. Since it backups incrementally, I thought it might work with DTP, thought I had my doubts. The first backup did not show the changed files within DTP, which is strange since the changes had been made and presumably saved. At this point, I wasn’t able to save the database to its original location (by Cmd. + S)! The original database kept persisting. Since I had some other tasks I needed to divert to, I then exported the whole folder that contained my changes to the second volume, checked to see that the changes were saves (they were), and I shut DTP down. When I returned to the project, my changed files were nto to be found anywhere!

2

Hi, Jerry. You didn’t say what file types you were editing, or how you opened or saved them. That can be important, especially for MS Word files.

Word .doc files captured by Import are the only file type that isn’t physically copied into the database. Only the rich text content of a .doc file is captured, and that will lack images and some formatting features of the original. But the actual .doc file remains external to the database and the database document links (but without synchronization) to the external file.

If your only purpose is to capture the text of the .doc file into your database for searching and analysis, that’s OK. But if you need to edit or print the original Word document, you can run into problems.

The first problem is that if the external Word .doc file is either deleted or is moved to a different volume, the link Path to the file from your database will be broken, so that you can no longer open the external file to edit or print it.

The second problem is that if you use ‘Open With’ to edit a text, HTML, WebArchive, XML or similar text-based document – including the rich text representation of a .doc file – the file will be opened under the chosen application and if ‘Save’ is invoked, the document will appear to be saved. But in fact the ‘Save’ operation will only save to a temporary cache file, and the edit results will not be available in your database, nor will they be available in the Finder where – in the case of a .doc file – you might expect. To avoid that problem, use ‘Save As’ and choose the destination of the edited file in the Finder. For .doc files, if you invoke ‘Launch Path’ the actual external .doc file will be opened under MS Word and after editing the file you can safely use the ‘Save’ command.

Note that the ‘Open With’ command causes a similar result of saving to a temporary cache file for all the text-based file types mentioned above, as these file types are stored in the ‘body’ of the database in a format that’s not readable by the Finder. But those file types stored in the Files folder inside the database package file, e.g. PDF, postscript, image, QuickTime media and ‘unknown’ file types do not suffer from this problem. Note that when version 2.0 is released, all file types will be stored in the Finder, so these complications based on file type will be eliminated.

The third problem is that in the Import mode there’s no synchronization from the edited .doc file to the content of your database. So if you wish the results of editing a Word file to be reflected in your database, you will need to re-import it to the database. Note: this problem will also be eliminated in version 2.0.

So, especially for Word .doc files that you may wish to edit frequently, it’s probably best to capture them using the Index mode. Index-captured files cannot be directly edited by the text-editing tools in DT Pro, but edit changes made to the external .doc file have one-way synchronization from the external file to the database content. Therefore, it’s not necessary to recapture the edited external Word file.

I hope these comments can be helpful in solving that issue.

But there may be another issue in how you are doing backups. If you export to a folder in the Finder, DT Pro will save the selected groups and files to that folder. But DT Pro also places another file in that folder, DEVONtech_storage, that contains metadata about the exported material, including creation and modification dates, comments and so on. If you repeatedly add additional exported items into that same folder, I suspect this could cause overwriting of the DEVONtech_storage file(s) and cause loss of metadata.

One of the virtues of DT Pro is that you can always export your data to the Finder using File > Export > Files & Folders, so you are not ‘trapped’ into your DT database. But I would be wary of repeatedly (additively) exporting into the same folder.

Cautionary note: One can make a Finder copy of a database, or a zipped archive copy of a database in the Finder. But it’s strongly recommended that the database is closed whenever such a copy is made. Otherwise, errors can creep into the copy, as some data may still be in RAM. I extend that recommendation to use of backup software for backing up one’s computer or Home directory to an external drive. It’s safest if any open DT database is closed at the time the backup is run. I feel the same way about open files under many other applications, as well.

For routine external backups I prefer using DT Pro Scripts > Export > Backup Archive. This routine will check for and correct any minor errors in the database (so it’s a good QA routine), optimize the database to reduce memory footprint and file size, and produce current internal and external backups. That external backup is the smallest possible compressed and dated file, that can be saved to a location of your choice, including a mounted external drive.

During a day when I’m making significant changes to the content or organization of my database, I may run Backup Archive whenever I take a break. When I return from break the database is ready to go – QA’d, optimized and with current backups.

3

The file format I was working with was .rtf in an application called Nisus Writer Express, also out of Germany and the best alternative to Word that I have found; but I am glad you discussed .doc.

Concerning the temporary file problem: It seems, then, that Devon should provide a warning dialogue to this effect since, even knowing this about this behavior, it is somewhat counter-intuitive and easily forgotten. Maybe this will be moot when 2.0 is released.

(“For .doc files, if you invoke ‘Launch Path’ the actual external .doc file will be opened under MS Word and after editing the file you can safely use the ‘Save’ command.”} It seems like launch path would always be the best option for editing an external file, unless one desired to change the save location, right?

(“The third problem is that in the Import mode there’s no synchronization from the edited .doc file to the content of your database. So if you wish the results of editing a Word file to be reflected in your database, you will need to re-import it to the database. Note: this problem will also be eliminated in version 2.0.”) Does this mean that there will be a sync command under import in 2.0? If so, do you think that the finder files will be able to be stored on a different volume, thus providing and inherent backup?

It’s good to know about the Backup Archive script. I tried it, and it does solve the problem of frequent backups to another volume. My purpose in using the Personal Backup (described above) was partially to keep the finder files and folders in sync, which it seems will be solved in DTP 2.0 — “Come On, 2.0.” This bring up another question: Given the caveats you have described about text files, how do these apply to image files? Certainly, .psd files have to be edited externally, but are these subject to your warning about using “save as” rather than “save” when using “open with?” Should these and other image file formats that may be subject to editing be indexed as well as .doc files? Also, I didn’t see Illustrator format mentioned in HELP. Should I save .ill files as .eps?

4

Hi, Jerry. The problem about editing text-based files will become moot in v. 2.0.

But there’s another caution. There are a number of ‘flavors’ of RTF text, based on embedded commands. So if you edit a Nisus Writer RTF file using DT Pro’s built-in TextEdit editor, you may lose features in the document that are specific Nisus Writer features.

Although RTF files can be opened by a number of applications, by no means does that mean RTF files will open the same way in different application.

I’ve become so irritated by the Tower of Babel effect of so many different file formats that I switched to Papyrus 12 as my polishing/final word processor. I can use a hybrid PDF file type. That file type can be directly read on any computer platform, as it displays and prints as PDF. It looks in my database exactly as created in Papyrus, and from my database I can open the file under Papyrus, edit and save it. The result is visible in my database without further intervention, next time I open it. (Papyrus is very powerful, but some features are a bit irritating or require a learning curve.)

You asked about backups on different volumes. Indexed files require the volume path to remain constant, unless the Path is relative to the same User Document names, so moving those files to a different volume could break paths.

I use the Import method, which copies the files into the database (except for MS Word files, currently). I do that precisely because my database is not location-dependent. I can move it anywhere I wish, from computer to computer, without breaking paths. The current disadvantages of Imported databases (how to edit text-based file types and so forth) will go away in v. 2.0 because that will use a new database structure.

You asked about image files. No problem, as Imported image files are actually stored in the Finder, inside the database package Files folder.
So you can select an image in your database, invoke Launch Path or Open With, then edit and save it. Next time you open the image in your database the changes will be visible.