This is not a complaint, more about sharing an important experience I just had with DT Pro. I just bought DT Pro and I wanted to test the robustness of the Database, so I dumped a 10 GB dataset into it. It went perfectly, no issues. Next day I was syncing the database and it hang up (on an M1 MacBook Pro), so I force quit the application after a10 minutes.
When I rebooted DT the database went corrupt and the folder structure was gone, only the files were listed. Obviously it was due to the fact that I had to force quit the application - but this can happen anytime, it happens with all software, I think for these situations the tool should be prepared and DT failed.
Therefore unfortunately I will not be able to trust the system going forward, I will use it only sporadically with data already duplicated in Finder. I wanted so badly to embark on DT 100%, but this was an early warning for me not to go ahead.
Again, this is not a complaint, but a heads up for someone who is thinking about going full steam ahead with DT. I love the tool, and will use it, but I will not be able to fully trust it.
It was a regular type of data set, bunch of smaller files, pdf, txt, etc. nothing extraordinary. For sync I used iCloud CloudKit. I have 4TB SSD on MacBook pro and approximately 2 TB is still free.
Please choose Help > Report Bug while pressing the Option modifier key and send the result to cgrunenberg - at - devon-technologies.com - thanks! Maybe there’s a hint in the logs.
Dumping a massive amount of data at once is not the typical usage and if you had asked me, I would have dissuaded you from doing so. Going “full steam ahead” doesn’t mean you should throw all caution to the wind.
The data I dumped was to stress test, it wasn’t the main data. It was a copy, to see for a few months how I can cope with it. I would consider myself an advanced user (built my own server rack, used for many years linux distros, vim, emacs, you name it). I did it deliberately to see how it scales.
So I have lost no data at all, other than the test dataset.
As it happened a few days ago and I have deleted the database, I don’t think it will list the bug report anything as of today - but please let me know if I should still send it.
Understood and as part of my job I also do such things. I’m glad this was done with forethought and intent. Sadly, I deal with people every year that don’t exercise such caution.
Absolutely clear! I was a long time DT user, and I intend to use it in the future as well. Actually I was very glad that I managed to crash it, as I knew deep inside that I shouldn’t use it as a full file (e.g. Finder) replacement tool. That is not the intended purpose of DT, and I proved it to myself
Bad things can happen to the Finder too if you move around 10 Gigabytes of data rapidly and interrupt the transfer.
Moveover you only lost your folder structure - not your files. That’s a great design feature of DT4.
Most notably - DT4 regularly backs up the folder structure, something Finder does not do. It appears this issue occurred because (1) You imported 10 Gigabytes rapidly; and (2) You Force Quit the App; and (3) You did this so early after setiting up the database that no folder backup existed yet.
That is quite a perfect storm situation - omit any one of those 3 factors and you probably would have been OK. I am not sure Finder offers you more resilliency in the long-term.
Thank you so much for the detailed answer! I need to agree, I actually did all three “mistakes” at the same time, as you described - and no surprise, I burned myself nicely
(Offtopic: Actually I am very keen to have multiple redundand data safety - I built a cluster server with two Synology rack servers in RAID10 with two hot spares, I do back up regularly on LTO tape drives, I do have daily background scripts on Carbon Copy Cloner, each computer has it’s own Time Machine backups, all copied on Backblaze service, and I even use Offset - an app which copies files with additional checksum verification. And still, despite all these redundancies I managed to almost burn myself, I was lucky to test it with dummy data.)
But is is very reassuring, that unless I dump all the data at the same time, I should trust the database. I will do another try, using your suggestion to avoid the perfect storm situation. Thank you for the advice!
DT4 is a total finder replacement for whatever I choose to put into DT4. I do not keep anyting in both places.
For the most part, all of my long-term and/or critically important documents go into DT4 - both personal and business. That has been the case for over a decade.
If you look back at DevonTech’s track record, I believe there was only one instance of a software issue on their part causing customers to lose data - and even then their team instantly sprang into action and came up with a plan that recovered a good deal of the data for those who were affected. I berlieve that had to do with a Devonthink to Go update.
That track record is as good or better than most enterprise level software firms. Even Amazon AWS - which is used for extreme high-level security needs - once lost data.
I certainly do not see Finder as safer than DT4. I do not know that anything safer exists at any price except a good backup system that you implement.
Finder is just a view into the file system. It does not “contain” files. Nothing is “in” Finder. It has a rudimentary feature set to manage files, folders, and remote file access. DEVONthink is not a Finder replacement. To think that is very misleading. I use both all the time. For different reasons.
@rmschne - I agree with you. I also realised when I was thinking about my DT data strategy that actually if I am totally honest, I do NOT need sync almost at all. It is a convenience but not a necessity. It feels great to have all data always available, but in reality I do not have any urgent need that can not wait until I get home to access it. I think we are spoiled with all the online tools nowadays, and we assume that we need access to everything at all time.
May I ask you, how do you define what to store in DT and what do you “keep” in Finder?
