Can't download / sync since latest upgrade

Hold the phone, kids… we appear to be talking about two different things now. This thread is about DEVONthink Sync, not DEVONthink To Go syncing. Lets keep these things separate.

Back on topic with DEVONthink Syncing…

Yes, you can (and, truthfully, you must).

Say you have a database Synced on two machines. If you make changes on A, Dropbox is out of date until you Sync. If you makes changes on B, Dropbox is out of date until you Sync. Changes on A do NOT affect B until (1) you Sync to Dropbox, etc and (2) B Syncs with the Sync location.

So you have two machines with independent changes and they are out of Sync. Cleaning the Dropbox store just flushes an out of date store, NOT the changes on machine B (or A).

So you can clean the store and Sync A fresh. Dropbox is now up to date with A’s changes. Sync B and it will incorporate the changes and push it’s own to Dropbox. Sync A again and it will have B’s changes. Now everything is Synced up and life goes merrily on… :smiley:

I had to clean, too.

DTPO said after “clean remote…” : “All records will be deleted”

Great, once again have to wait that it uploads 10 GB on Dropbox… :frowning:

Hope this won’t be oftenthe same thing :wink:

Nope, just a one-time thing for adding encryption support. Believe me, I considered alternatives to having everyone clean their stores, but this was really the best way to do it.

Thank you very much. :slight_smile:

Thanks for all your work.

Sigh. I’m using a local sync store stored on a network drive. Since upgrading I cannot sync with the error message known to this thread. I can go to the preferences and clean the sync store, but this will erase the sync store. Some of my mobile Macs haven’t synced in a while and there are many differences in the databases and I don’t know how my different installations will react, when they have to sync to an empty sync store. Of course, there are some questions:

  1. Why do I have to clean the sync store?

  2. I a local sync store stored on a mounted network drive considered a “remote sync location”?

“DEVONthink 2.7 now encrypts all metadata and all contents stored in remote sync locations such as Dropbox, WebDAV, or AFP. No other document management app on the Mac gives you so much flexibility and control over you data!”

  1. Since “no other document management app gives me so much control over my data”, why can’t DTPO explain what it does, what it expects from me to do and why can’t I control whether I want it to happen?

  2. What does this mean if different existing DTPO databases on different Macs want to sync against the same sync store? -> “All metadata and content files are encrypted using an AES 256 bit key unique to the database. Protect the database with a password to keep even us from possibly decrypting it.”

I’m using DTPO on with my remote stores just fine. Now I have to clean them. I don’t know why. I don’t know whether DTPO considers my sync stores as local or not and I don’t know whether they get encrypted or not. I even don’t know in which way they would get encrypted and who controls the encryption. With no way to control the encryption (and, therefore, the decryption) it is broken.

And, by the way, “cleaning” means I have to bring my mobile Macs back home to sync via a fast connection because it will most likely sync all stuff again and that would be too slow via the normal dsl connection. Great.

Just to make that clear: I was perfectly fine with having no encryption because I keep my data at home and only sync remotely via vpn. I just want to use my DTPO 2.7 the way I used 2.6. Currently I don’t know what happens because the software and blog entry does not explain it.

As mentioned already, this doesn’t matter.

All records are now stored in an encrypted form. This is a fairly major change. Whenever changes this significant are introduced to sync stores, it is necessary to break backwards compatibility because I can’t travel back in time and introduce forward compatibility to DEVONthink 2.6.1.

Yes. If you get the error message, it’s considered a remote store.

Well, it does state that the format of the sync store is outdated and asks you to clean the remote store. I admit that the documentation isn’t sufficient, and I sincerely apologize for that.

As far as a choice is concerned, there are a few choices:
a) don’t use Sync.
b) stay with an outdated version of DEVONthink.
c) rely on direct connections instead of filesystem-based stores.
d) update the sync store.

It means that you don’t need to behave any differently to benefit from encrypted stores, but for the best security set a username and password on the database and the key will be protected more securely.

Uh, why does it mean that? You only need to sync each database from one machine.

Sure. Many others weren’t. Unfortunately, since encryption affects the store and not just the application, an application-level preference is practically unsupportable.

Clean the sync store and sync again.

If I empty the store, sync with one machine, then connect another machine to the same store, will this machine recognize that it already has about 95% of the store entries or will it up/download everything and create duplicates in its own database and the store (and therefore in all other clients as well)?

So, let me recapitulate this:

  1. We now have encryption, which is fine. But we don’t know how the encryption works. There’s one comment for the blog post which mentions that there is a unique key for each database, but we don’t know when and how this key gets generated, what we should do if the decryptions breaks and how other DTPOses handle the whole encryption and decryption, because they use other local databases which talk to the same sync store to, well, sync.

  2. DTPO decides for itself whether it encrypts or not. If you don’t want encryption, you can’t disable it, which means that we have to trust DT, which we can’t, because of (1).

Yes. Each record has a UUID that is, mathematically, globally unique and used to match the records in the remote store/machine to your local machine. DEVONthink won’t transfer anything for which you already have a copy.

This thread has some of the answers you seek.

Ask and ye shall receive. There are no secrets, only questions that haven’t been asked.

I know that most of the discussion here has been about Dropbox sync, but I’m having issues with USB sync since the last upgrade. I thought the two might be, in some way, related.

I’m getting error messages that the store is invalid and doesn’t exist.

  1. I have one database that no longer loads at all. I can’t do a “repair” or “rebuild” because the datbase won’t load.

  2. I now have an “inbox” database that is a duplicate of the “inbox” in globals. If I delete the inbox in open databases, it comes back, even though I can’t actually find it “stored” anywhere on computer.

Things are really messed up.

I don’t even know where to start to sort it out. The problem now seems that the “good” databases (ones that aren’t corrupt) are on two different machines. I guess I need to create new, individual database stores for each of my databases to sync to in order to get things straightened out.

I agree with one of the previous posts. I’m beginning to spend more time on trying to keep things in sync than the the product is saving me. It is a great piece of software on a single computer. But I’ve had nothing but problems since I started trying to sync.

Could you give me an exact error message? That doesn’t really sound like any of the error messages I’ve written (which is all of the Sync error messages), and if the store doesn’t exist it’s simply created during the sync.

Okay, but that’s not a Sync issue.

That’s really not a Sync issue.

Open a support ticket.

If your database is corrupt, it’s not a problem with Sync. Sync does not have the degree of access to the database that is necessary to create this sort of problem – precisely because of this sort of troubleshooting issue.

These things happen (very rarely) with DEVONthink, and I understand they’re frustrating, but simply because you had a problem post-DEVONthink 2.5 does not mean it has anything to do with the Sync functionality.

I updated to 2.7 on my iMac and Air tonight and got the same result, the pop-up instructing me to clean the remote store. When I began that process on my laptop, the activity window language seemed to indicate that it was about to wipe the contents on my laptop.

Yes? If so, dismaying and time-consuming. I can’t afford to be launching lengthy re-syncs when I’m about to go back on the road. If there had been an adequate warning of this effect, I would not have upgraded.

Nathan,

I guess I shouldn’t have made this post. I appreciate all of your questions, and I realize that you are trying to help me, but I simply don’t have time to deal with all of this. From a user perspective, the software “just needs to work”. I shouldn’t have to fight through issues each time there is a software update.

You say that these are not sync issues, yet all the problems came about after the first sync after the upgrade to 2.7. Everything was fine until then and I had been using 2.7 for a couple of days prior.

From my viewpoint, an upgrade shouldn’t cause these types of problems, whether they are sync or otherwise related. Although I would like to work with you to resolve the issues, my apologies, but I simply don’t have time to be a software tester. I need to move on. I’ll continue to use DevonThink for local databases. It is a great product for that.

@wbryanharris:

Just a note from personal experience… I use direct connections and local syncStores exclusively (using Dropbox / WebDAV only for Support purposes) and I have never experienced any of the issues you are seeing. While it feels like there would be a logical connection between Sync and your corruption, my experience and the lack of anecdotal data in Support here (ie. no other Tickets with similar issues) doesn’t support the theory.

Many times, data corruption comes from hardware errors (assuming you’ve heard enough of the mantra about not storing your database in Dropbox). If I was you I would make sure your hardware is operating properly, and testing the syncStores on your local drive to see if the issue is reproducible off the USB drive.

PS: This really isn’t “software testing”. It’s software. Period. Given the complexity of what software developers do coupled with the fact that no two systems are alike, it sometimes amazes me that anything runs at all. (Programmers often have to bank on the concept that machines are “alike enough”.) Even when you’re running OS X, you are “software testing”.

Not on your laptop, no. Unless you keep the sync store on your laptop. In which case resyncing shouldn’t take very long at all, being essentially limited only by disk speed.

This is why the interface prompts you before making any changes to the store. You can still use 2.6.1. (Of course, this is is not recommended; 2.7 has a lot of other improvements under the hood.)

Sync doesn’t have the ability to cause database corruption or create a strange, ghostly clone of the Global Inbox. Sync has a very abstract interface to the database and absolutely no direct control over how databases, records, or the rest of the application actually looks or acts. There are plenty of problems with Sync; these are not among them.

I’m not asking you to be a software tester. I’m asking you to open a support ticket so that someone can help you fix your databases.

I now cleaned my sync stores. It took two hours where I constantly had to acknowledge the “do you want to delete” button, because I have several sync stores. After another 4 hours, the initial sync was done without errors. Then I synced everything again, just to be sure. Now one database gives me this message:

07:25:34: Sync: Mail-Archiv → Mail-Archiv (Local Sync Store) Database Metadata: (The file “metadata” couldn’t be opened because there is no such file. (The file “metadata” couldn’t be opened because there is no such file. (The operation couldn’t be completed. No such file or directory)))

Sigh.

Verify and Repair does not show errors, but since the error message does not explain if that’s an error on the sync store or my local database I’m stuck.

Should I clean it again?

Edit: On the second Mac another database cannot be synced against the store. Whenever I sync this database DTPO crashes. Great. Will open a ticket for that.

Edit 2: On the third Mac, another database cannot be synced:

10:13:28: Sync: Work → Work (Local Sync Store) Design-Dummy: (Could not load a record (Could not scan data. Data wasn’t encoded properly?))

Once again it is not clear whether that’s a problem of the local database (which shows no errors) or the sync store. If it’s the sync store it remains unclear why the other Macs can sync the database to this sync store without problems.

And another thing: One database which can be synced with two Macs crashes DTPO when the sync starts on the third Mac (I enhanced the other ticket for that problem).

Not amused.

As an early poster on this thread I want to express empathy to the support engineers working through the synch challenges.

I’d also like to state for the record that over the past few days since the upgrade(s) in synch, I have spent more time cleaning remote stores, deleting databases on Dropbox, re-uploading, filing DEVONTech help tickets, and generally babysitting the synch process on a couple of Macs than I EVER did when I dangerously stored and accessed my databases on Dropbox.

No malice or snark. That’s just a fact.

I trust that this new synch technology, and the forthcoming Synch 2 (“Project Unicorn” ––ok, that’s snark), will be better, safer, faster in the long run.

But the implementation, at least for me and my comrades in frustration here on this thread, has been something far less than a great experience.

I am a big fan of DT. I use DTPO every day. It was central in the writing of my PhD dissertation, and I envision using the software in expanded ways in my work. So it is a bit painful and frustrating when DT doesn’t work as expected. Worse, there is virtually no feedback in plain language about 1) what went wrong, and more importantly, 2) what I can do about that.

Furthermore, it can be quite frightening when getting one of those cryptic error messages. I think my precious data may be at risk, or it might not be backed up in the cloud as expected. (And I may face yet ANOTHER day of babysitting the upload /download cycle before DT maybe, finally, gets all the data in synch. And I’m still kinda doubtful now…)

Still, I am going to hang in there with you. I appreciate the ambition and complexity of what you are attempting. And I haven’t made it easy. Some of my databases have many thousands of tiny files. Some are rather large with many gigs of data.

I also urge you to take one-half day more before you release another version of your amazing software. Take that time to write Status and Error messages that someone who is not a software engineer can decipher and act upon.

A good rule of thumb: Ask, Would my mother understand what this message means and what she should do about it?

Ok. Make it a full day.

For what it’s worth, I’d like to sincerely apologize for my part in this mess. I’m terribly sorry.

Before and after Edward Snowden’s revelations, a lot of people mentioned that they didn’t feel as though their data was safe on Dropbox or other commercial service providers. One or two members of our staff mentioned that they personally wouldn’t use Dropbox or any other cloud service unless it was encrypted. I thought it was important to add encryption, so I did, and I made the changes necessary.

The last change in the code that I can find concerning encryption occurred August 14, well over two months ago. Since then, I hadn’t received any indications that there were any bugs in the implementation, either in my own testing or that of our beta testers. I was actually very excited about this release.

I was and am, of course, aware of the many design and implementation flaws in the Sync plugin. Many of the complaints brought up in this and other threads on the subject are complaints that I share. I myself have been planning and working on Sync 2, until recently all on my own time, since the Sync plugin was in its beta stages.

Sync was just getting to a point where users felt they could rely on it, and then I made a bad decision. You’re all absolutely correct. I should have left it as it was and merely fixed the cosmetic issues.

But I ask, please, if you have an issue or a bug report, please open a support ticket. Tickets mentioning Sync go straight to my mailbox and I answer them almost sixteen hours a day, in almost all cases as soon as I receive them. The information I need to troubleshoot the issue (specifically, logs) can contain sensitive information (your account name, server address, record names, etc) and shouldn’t be shared on public forums, and without this information it’s almost impossible to diagnose what really happened.