I now have all my data back. But boy did I just get a scare. I’d like to figure out what I did wrong, and how to do it right.
A handful of files in my Devonthink database turned up “missing”. After googling advice on this forum and blog, I decided to try rebuilding my DT database and see if that would stop the problem from recurring. The process ran, the database was rebuilt, and that’s when the real problem hit: EVERYTHING VANISHED. What appeared after the rebuild was (1) my existing folder structure, and (2) absolutely nothing else. None of my files; none of my notes; none of my PDFs; etc–all of which had been in place immediately before the rebuild. (See attached screen shot.)
Eventually, I was able to restore from backup, losing my research notes from today–but (I think!!!) recovering everything else from the last 5 years. But I’m now worried there’s something really wrong with my database. (And/or that I’ve maybe lost something that I don’t know about yet.)
Is this a known problem? I have searched for forum and blog items on this problem, but haven’t found anything useful. E.g., there was no power outage etc during the rebuild. It just deleted (or seemed to?) all my files. Am I doing something wrong with the rebuild?
Rebuilding can’t fix missing files, repairing might be able to fix it. Where’s your database stored? E.g. in a synchronized folder or cloud folder or on a network volume? Or did you move/copy the database while it was used, e.g. to a different volume? Does Tools > Verify & Repair still report any issues?
I didn’t rebuild to bring missing files back. I was prompted to rebuild by a suggestion in a DT blog that it’s a good idea to rebuild “misbehaving” databases. And I figured 4-5 important research notes being flagged as “missing” constituted misbehavior sufficient to prompt a rebuild.
Again, it wasn’t until after the rebuild finished that the Great Vanishing happened. Before the rebuild, it was a handful of recent entries that came up “missing.” I don’t even get “missing” notation now–the number of files has gone to flat zero. Before the rebuild there were many thousands of files, mostly .txt and .rtf. After the file, the screen shot in my original post was the result. Does this mean I can never rebuild my database again?
The most likely explanation is that not only 4-5 files were missing but (almost) all of them. Tools > Verify & Repair reports the number & kind of issues.
But assuming that Tools > Verify & Repair doesn’t report any issues, then you could of course rebuild the database (a backup via File > Export > Database Archive… might be a good idea to be on the safe side).
Thanks for replying! It’s definitely not the case that most of the files were already missing. I’ve been working in the database regularly for the last month, accessing files throughout its structure without problem, through to yesterday morning. And after everything did vanish following my rebuild yesterday, I reloaded from a backup made earlier in the day. And all files were instantly back in place (confirmed not just in folder structure per attached screen shot, but per dozens of individual spot checks)–except for the 3-4 “missing” files that prompted the rebuild in the first place.
Maybe we should forget about the 3-4 initially missing files? I mentioned those only because I thought they might be diagnostically helpful. I have written them off as lost–it’s just one days work. I’m now worried they may be distracting from what seems like the real issue. I’m now much more concerned about the fact that there is something seriously unstable about my Devonthink setup, and/or that I can never rebuild the database again, and/or that more things will start vanishing.
Running “Backup & Optimize” produces no error messages.
Running “Verify & Repair” now shows “0 inconsistencies, 0 incorrect checksums, 2 missing, and 0 orphaned files”
What would explain a rebuild resulting in the screen shot from my original post? Are there settings I should check? Additional diagnostics I should run? Isn’t this a pretty troubling result?
Also, to be clear, it was rebuilding the database that caused the Great Vanishing in the first place. So I think (unless I’m missing something) this would probably just cause the problem again??
Rebuilding doesn’t depend on settings and there are no known issues that could cause this. Are you able to reproduce the problem (after backing up your database of course, e.g. by duplicating it in the Finder)? Where are your databases stored? Is there enough space available on the volume?
Database resides entirely on the central work server. There is no space issue there. I’ll back up the database and try rebuilding again. To be clear, what should I expect to see after a rebuild. A brand new “file” in the same folder as the “original” database? Or the same “file”, same interface, same folder structure, same contexts–for all intents and purposes the same “thing” that was there before?
The Log generated a very long list of comments, most of which looked like the screen shot below. All of the files in this screen shot existed with content in them before the rebuild. All of them exist again now that I’ve (for a second time) reloaded the devonthink database from a backup.
EDIT: Well I can’t figure out how to make the entire screen shot appear here on the forum. So, I"ll just describe the hidden portion: at the far right next to each file, it says “Skipped”
I was able to reproduce this by storing the database on a network volume, this is usually not recommended due to the reduced performance & reliability and that’s probably the reason why noone has ever reported this.
Version 2.9.12 will fix this, in the meantime restoring the latest backup via Tools > Restore Backup… is sufficient after a failed rebuild as rebuilding always performs a backup.
OK - thanks for the followup! I was previously told here that while I shouldn’t store the database on a syncing service, storing it on a network drive was ok because equivalent to a hard drive except with a longer wire. I guess that’s not right? It’s odd, because I (like all my colleagues at a very large institution) store/edit everything directly off the server – it’s how our IT department advises us to manage automatic backup process. (Though the applications themselves sit on our desktops.) And I don’t ever have problems with other files.
Two questions to clarify what I should do going forward:
When you say you could reproduce it by storing on a network volume, do you mean that after testing it yesterday, you are now seeing it happen every time you store it on a network volume? Because I’ve been using DevonThink for years from the network volume, and have been able to rebuild etc before without problem. Does this mean there’s something in a recent update that has changed things? Is there some way for me to change settings that will eliminate the problem? I don’t understand the technical issue behind the scenes that would create the outcome.
Perhaps most important, you mentioned that 2.9.12 will fix “this.” Two questions. Does that mean there will no longer be problems with rebuilds like the one I’ve described – this is a known issue and will be fixed? And, when (very roughly) is the version edition scheduled to be released?
It’s okay for files but not recommended for databases. The performance suffers and the reliability too as there are more components that might fail.
Actually rebuilding hasn’t been modified for a while but in case of network volumes this might depend on the used protocol (AFP, SMB etc.), filesystem and OS version.
Just weighing in with my two cents… IT departments always want to advocate server-based setups because it allows them to sit in their offices and administer things remotely, but they’re also slow to respond when problems happen. (I have done IT for many years. I know the people and routines well. ) ALL server-based setups are inherently susceptible to problems from the network. It is always going to be faster, more reliable, and more portable (for those with laptops) to have local resources but Sync to a central location. (This is the decentralized data modal we advocate.)
And, BlueFrog, just to be sure - when you say “local,” do you mean “in your computer’s physical hard drive”? This is probably a staggeringly obvious question, but I just want to be sure. (Because the server I’m running on is 1 floor down from me, not like “in the cloud” or whatever the right phrase is.)
Gotcha. Thanks again to all. I’m going to try to work out some autobackup plan with the IT Dept here that will leave me feeling as secure about backups from my desktop as I do about the central server. And will move forward from there–including perhaps by finally trying the new(ish) native “sync via dropbox” per this link: blog.devontechnologies.com/2016/ … nd-webdav/
Just as a note… Sync is not intended to be a backup, even if it provides a small measure of it. If your IT is Mac-antagonstic (as many IT departments are), you can use Apple’s built-in TimeMachine and an external drive. They can even administer the external drive if they want but it should be Mac formatted (HFS+ currently).
) ALL server-based setups are inherently susceptible to problems from the network. It is always going to be faster, more reliable, and more portable (for those with laptops) to have local resources but Sync to a central location. (This is the decentralized data modal we advocate.)