Hello!
Today I found that database was corrupted — 381 files missing, can’t be repaired. So I restored from backup — same thing. I decided to rebuild it. Still in process. But when I went to indexed folder, I found two folders missing. That was a real surprise! It never ever happened to me before. Now I am trying to figure out what happened. So the question is: Can corruption in Database cause deletion of indexed files?
Or maybe it’s the other way around – two deleted folders causing corruption of your indexed database?
We’re not aware of such an issue. Were only missing files reported or also other problems? Where are the indexed files located?
Or maybe it’s the other way around – two deleted folders causing corruption of your indexed database?
Maybe, that’s why I am asking. Because I try to understand what happened.
Were only missing files reported or also other problems? Where are the indexed files located?
I found that 6 files which were located in 2 folders was deleted. But much more missing from database. They was all located in one folder which is parent folder for two missing folders. But all other files and folders in parent folder was not deleted. This files are in iCloud synced Documents folder. And yes I had an issue of iCloud offloading files because I ran out of disk space. But somehow only two folders was deleted. All other I can see they are just offloaded from disk, exist only in cloud.
Apple iCloud the likely suspect. Do you have “optimise disk” on?
I do not know what happens with iCloud when short of disk space. Could also be culprit.
Do you have “optimise disk” on?
Yes, I turned it off when I understood what happened. When space is low and this option on it offloads part of files from disk. Maybe this is the reason for database corruption. Likely so. But still I can’t understand what happened with 6 files which was deleted. Missing 381, but deleted only 6 from two different folders.
Are you sure the database was corrupted? I’m guessing not. I’m guessing it was only missing the files that iCloud deleted. Their algorithm of what they delete to “help” you with disk shortage is a mystery.
Also remember that DEVONthink, and all other apps, need “spare” disk space to work properly. Perhaps look into clearing out old files or using an external drives for some of you data.
Not sure. Maybe, as you say. But main not understanding and riddle is why 2 folders disappeared? Also, I decided to delete indexed folder with which I had problems to index it again freshly after I did rebuild of database. Then emptied trash, paying very close attention to press button to delete files only in database. And then I got much more files deleted from that folder in Finder! Why did it happen?
I’m confident that any deletions are caused by iCloud. However, at this point, it might be best to re-read the sections in the DEVONthink Manual about Importing vs. Indexing files. Indexing is more complex and requires understanding to avoid misunderstandings.
But I have already switched off “optimise disk” before doing last operation.
Yes, I did that.
Well, I remain confident it is not DEVONthink. And your machine seems short on disk space so no telling what can happen.
Export to website or files and folders is a way to have an outside-Devonthink backup view of a database’s organization along with a copy of all the files. You can also create a roster of your files with tools->metadata overview or file->export->json metadata.
I would not do it that way. Use the Menu → File → Update Items action on a selected index.
As mentioned, please re-read the “DEVONthink Manual”. The above suggestion covered on page 53. To be honest, I don’t know if what you did contributed to the problem you report. I just don’t see a way for DEVONthink to delete files unless told to by you. Maybe I’m missing something, but starting point I think is to get more familiar with the issues associated with Indexing vs. Importing.
Edit. Also make sure you are Indexing when you intend to Index rather than Importing, and visa versa.
I’m in the side it’s iCloud problem because I had it some time ago. As with most clouds, when yo go serious with files (more than about 300.000), use to fail.
I had my entire reference library (about 800.000 files, mostly PDF and ePub) in iCloud, distributed in a rational folder hierarchy. For example, my entire Scientific American collection was stored in SciAm folde, inside that folder, in sub-folders of 10 years: 1845-1849, 1850-1869, and inside each folder, the complete ten-year collection, that, for times before 1912, were about 500 files (one for each week of each year), then 120 (one for each month of each year).
Suddenly I noticed some files weren’t there anymore. Sometimes there was one file for folder, sometimes tens of them. Disappeared. Not in iCloud web, not in recovered files, not in other macs. I even tweeted about this when it happened to me.
And at those moments, DT had nothing to do with them, as I’m not sure if I were using it or not, but if I were using it, it was without indexed files because, well, I didn’t knew DT could be used that way.
Then I moved to Dropbox, then I used indexed files in DT, then I gave up with Dropbox due Monterey issues and move into DT local files one more time, and now I’m using Synology Drive client that works better than OneDrive, Dropbox, iCloud Drive or Google Drive, having tested pCloud and other providers.
I never recommend the use of iCloud Drive except for trivial things, not serious work, and even less if that work is critical.
In my experience with indexed files in DT, the only issue is, if indexed files are in a cloud service, the generation of duplicated real files in disk, or phantom ones, so and then. And even in that case, I suspect the guilt is the cloud service that is not able to handle the sync well.
Just one remark: these are all consumer offerings. I’d süße commercial cloud services like AWS to be (more) reliable.
Thank you very much for confirming that the same thing happened to you. I have decided to use indexing as little as possible. However, I am unable to completely avoid this method with my Calibre library, which is over 10GB in size. My MacBook has only 256GB of SSD storage. Probably should shift Calibre library to external drive…
It is what I have. My Calibre books are out of DT, and last versions can index ebooks in Calibre, then you still can have access to the content of the books, but outside of DT. And search options surely are not powerful as DT ones.
I need my entire library to be available in DEVONthink for work. The best solution would be to import everything into DEVONthink and move the Calibre library itself to an external drive for the sake of convenience regarding Kindle integration. It’s just one more hassle to deal with.
perhaps invest in a new Mac with a disk of sufficient size to meet your needs. 256gb on the edge of failing on everything. maybe not now but eventually.
Apart of @rmschne tells you, sending from DT to Kindle by email now they directly accept ePub files, is as easy as right click in the file and select send by email, and if you want to do it manually, connect your kindle, open the folder and copy/paste/drop the epub into kindle folder.
Other thing is if you use some of the “special” Calibre stuff, like, errr, say, remove the “bug”, or hyphen, or recreate the TOC, etc, in that case you need Calibre yes or yes.