I noticed two of my databases had without warning stopped syncing two weeks ago. This is very disconcerting to me as I trust DEVONthink to keep all my clients synced for remote work when I don’t have access to my primary workstation.
Digging into the logs I found that the issue was that I had a number of empty files in the two databases. I’m not sure why DEVONthink freaks out about empty files. They are nothing special. Everyone can make one typing touch empty_file in the terminal.
Getting rid of the files was not an easy task either. Deleting them (putting them in the trash) did not help as DEVONthink still refuses to sync with empty files in the trash. So I ended up having to delete them and then emptying my trash completely.
If anyone feels like recreating this simply do the following:
Create an empty file (e.g. touch empty_file).
Add it to DEVONthink.
You can no longer sync or repair your database before manually deleting the empty file.
I really hope that something can be implemented to make DT fail more elegantly in the case of empty files. I routinely add directories to DT and it would be quite a hassle to manually filter out empty files beforehand. Also I would LOVE a huge fat fatal error message when my databases fail to sync. This is something that I really really need to have working all the time. So if sync fails, or stops for whatever reason, please let me know!
Edit: Sorry if this came off a bit hard. I’m just always concerned when systems I trust to be working do in fact not. I realize there might be good reasons for the current behavior of DT with regards to empty files. So you can also read my post as a question about how I can avoid this situation going forward. Maybe I’ve missed a setting, or a notification, or something else? I’m considering writing a script to weed out empty files manually. If that sounds like a reasonable solution that would also be great to know.
A brief bit of recent history: with the introduction of DTTG3 a number of users noticed that a small number of files were being damaged in the sense that they continued to exist but had no content. They were 0 kb in size, and became dubbed “ghost files”. The reasons for the damage are - to my knowledge - still under investigation, but are assumed to be due to a bug in a long past version of DTTG2 which propagated via the sync store. One of the mitigations implemented by DEVONtech was to include a mechanism in both DT and DTTG which stops sync when empty files are present; this way, propagation of the defect could be stopped. Whilst some file formats are assumed to be acceptable even when they are 0-size, others trigger the safety mechanism.
Sync errors are logged; depending on your configuration, the log window will open automatically when an entry is written. In addition, the warning triangle symbol at the top right of your DT3 window will “light up”. In DTTG the cloud symbol will be superimposed with a small triangle.
If this safety mechanism is causing trouble for you, you could set up a smart rule which deletes any 0-size files prior to sync:
Do be clear on the ramifications of this, though; a script which lists those documents and asks for permission to delete them (instead of the delete action in the smart rule) might well be a better choice. Also be clear that groups etc. are all 0-size; the rule must include the “kind is any document” provisor.
@cgrunenberg might this safety mechanism be something which could be turned off via a hidden preference? It might well not be desirable for every user to delete all their empty files to be able to sync.
Look in the Log. You might see that the database with the empty file failed verification. My test showed that on a previously-working database. Also in a new test database. Failed verify might be what stopped sync (which I think is probably a good idea). I don’t know much more beyond my test.
Adapted from a script I use to delete expired documents, a script of this type could be used as a smart rule action which - because it lists the documents and asks whether you really want to delete - would be safer than a simple delete action.
-- THIS SCRIPT IS NOT TESTED; USE AT OWN RISK!
on performSmartRule(theRecords)
tell application id "DNtp"
try
set recCounter to 0
set delDocs to {"
"}
repeat with theRecord in theRecords
set recCounter to recCounter + 1
set listName to ("
" & the name of theRecord & " - " & type of theRecord)
copy listName to end of delDocs
end repeat
if recCounter is greater than 0 then
set theDialog to ("Permanently deleting " & recCounter & " 0-size documents: " & delDocs)
display dialog theDialog with title "STOP AND THINK" buttons {"Cancel", "OK"} default button 1
repeat with theRecord in theRecords
-- a smart rule should be used to trigger the script only for "size is 0" AND "kind is Any Document"; the following line adds a degree of safety in case that is not correctly implemented.
if size of theRecord is 0 and type of theRecord is not group then delete record theRecord
end repeat
end if
on error error_message number error_number
if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
return
end try
end tell
end performSmartRule
Thanks as an aside, I can’t get type of theRecord is not smart group to work (the term smart group is not recognised); type of theRecord is not group does as it should. Admittedly, “smart group” is not listed as a type in the dictionary; but apparently a “smart group” is not a “group”, either.
Is there any way to exclude a type smart group in an if/then?
I’m currently facing the same problem. However, I have two databases with indexed folders. Those folders contain empty files required by Marked.app (<name-xyz>.marked). Now the log panel pops up automatically and displays every minute a message that the db check failed and I should repair the db.
So, an option to toggle this check would be highly appreciated.
Thank you for providing context, background and possible solutions. This really helped me understand the reasons for what’s going on.
Do you mean the Sync tab in Preferences? This is not a place that I go to very often, but no I don’t see any warnings here. The only indication I got that something was wrong with the sync was that I noticed that the date of the last sync was two weeks ago. A clearer warning or error message would have been great.
I have this disabled. I will consider enabling it, but I don’t care much about general log messages and floating windows. I would prefer some kind of clearly visible yellow or red warning sign for fatal errors such as failed syncs. That would be fantastic.
I removed the log triangle from my menu, but adding it back the only indication I get is that there is a new log message:
I probably wouldn’t think to check this as I had no idea that something was wrong with the sync before I went on the road and noticed missing notes on my iPhone. So again, I would love to have some kind of log level indicator somewhere in the UI to let me know not just when I have new log messages, but when I have critical/fatal errors in the log specifically.
I realize this is totally in feature request territory now, and I don’t expect anyone to promise anything here. I’m really happy to have been provided some context and I’m also fairly confident I will be able to prevent this issue from reappearing now that I know the cause of the failed syncs and suggestions to mitigate it. So thanks for that!
You’re most welcome I’ve actually taken to a brief daily look at the - some of my daily scripts report any errors to the log, so it’s become part of my routine
I should perhaps also note that I also got no indication in DTTG that anything was wrong. I did have some weird repeated syncs happening, but nothing that was indicated as errors in the UI.
It does return the critical database to syncing across my devices, but a huge benefit was being able to access that indexed r/o directory on mobile devices, so it’s definitely only “partial restoration of service”.
Seems like a clear and critical bug – much appreciated for taking it seriously and hopefully we can have a fix soon!
- some of my daily scripts report any errors to the log, so it’s become part of my routine