Indexed document suddenly shows in non-existing folder! I'm worried!

Please try a toolbar search for kind:group filename:>-1 item:indexed.

Only by removing the indexed items from the database (without deleting it on disk) and then indexing the folder(s) again.

Very clever! Thanks. (Update: Actually even better in my case: kind:group filename:>-1 name:!>-1 item:indexed)

Ok, I guess that’s what I better do from time to time…

But wait, the odyssey continues:

The same clever search (kind:group filename:>-1 item:indexed) on my 2nd Mac which uses the same DEVONthink database and has the same Dropbox account in the same location (~/Dropbox) doesn’t return any result! No folder 2022_to-keep-1 exists either.

What a mess!

Actually DEVONthink’s sync doesn’t use a suffix like -1 for folders, the synchronization duplicates only documents in case of conflicts but not groups. And in case of cloud folders it’s completely up to the cloud service/app to transfer the files/folders, DEVONthink skips this already to avoid possible conflicts.

Therefore it’s unclear how & when this additional folder was created. Maybe by the Dropbox.app, maybe by a smart rule/script or accidentally.

Yes, who knows… very dissatisfactory. No idea how I can set up things better…

See recent post here: Export and Overwrite Files and Folders - #2 by DTLow

Maybe you did an Export from DEVONthink at some point and you created the new folder?

Gents, I was very pro in relation to clouds but experience has made me completely anti-clouds. The less worse is Dropbox, meaning that all clouds in macOS are crap except Dropbox, that is less crap but still crap.

I don’t know how many files you have, but I have about 1M using about 1 TB. My experience is against clouds, more if you are using the “new” macOS API from Monterey and nexts.

Normally, when I say/wrote this, people should answer “it is working well to me”… until some time later they tell me “do you remember when you told me that xxx cloud is crap? Well, I’ve lost whatever”.

Currently, clouds are child toys to manage as many files as a child can count. At the moment you go really pro (millions of files or big size ones), you start a countdown to failure. And I want to add one more thing: macOS is not anymore what it used to be. It is going the same childish/luser way and if Apple does not rectifies, the future of the entire company is very dark.

Use a good and fast NAS (but not QNAP), don’t use clouds except as a matter of having some files in a third basket but not for real daily work. My two cents.

4 Likes

@rfog Yep. While, as you say, “it is working well to me”, is true for me and has been for quite some time esp when used as Dropbox intends, but I don’t exercise Dropbox with DEVONthink very much. I do not need or want EVERYTHING on my iOS devices. Only my “works in progress” stuff, and even then I do 99% of synch with Bonjour and WebDAV on a Synology NAS while at Global HQ. Synching on the go with Dropbox just not really needed, by me.

From reading stuff here, it occurs to me that a lot of people don’t really need it either, but for whatever reason, they want to do massive synchs to iOS devices via a cloud service. Then they pick one of the known unreliable ones and then start a new thread here on what to do.

And then Indexing files located on “offline” Cloud folders, even if they look to be on the local drive, opens up another big issue caused by the Cloud service providers.

But as file sync services, and sharing with others, the 3rd party services I use work well, frankly.

/rant

1 Like

In my experience, it is safe use clouds with not many files. It is what I do. I use OneDrive as “backup” (please, don’t use a cloud service as a real backup system) and remaining Dropbox time until my last payment ends, to have most of my long term not modified files reachable if I need them, but day (home) work is done in iCloud with only little files.

At work is the same. Despite the fact that my company is agains source-code-stored-in-any-cloud (a thing I agree when the source is related to money management stuff), I have all documentation in my OneDrive-at-company account, and customer updates. Only that and never has failed.

2 Likes

Given that many enterprises rely on cloud services today, and that for example the Berlin Philharmonic uses them to distribute their concerts world-wide: i think your “crap” verdict might be a bit exaggerated.

Also, you don’t explain what “crap” means for you nor what “cloud” is for you or what cloud services you’re referring to. Not to contradict you, but I don’t really know what your experiences are.

4 Likes

I meant Dropbox, OneDrive, iCloud Drive, GDrive and any other cloud software imitating those systems and third party tools that use them as alternative cloud software.

Do you want problems I’ve had?

  • High CPU usage without any reason
  • Never uploading a local change
  • Never updating a remote change in local filesystem
  • Local files not uploaded when a folder contains more than, say one hundred files
  • Duplicates
  • Files disappearing “magically”
  • Files locked and not downloading until restart

Those that I remember by now.

Another thing is a FTP service (as it, or hidden via a HTTP envelope), downloading a file from a server, etc. I bet that Berlin Philharmonic is not using any of the upper services to distribute their files to the public, and if they are doing it, lets give some time until they will have a random problem (not related to normal issues like service outages etc).

1 Like

They’re using AWS. Which qualifies as “cloud”, but rather “enterprise cloud”. I only wanted to point out that the general condemnation of “cloud” as “crap” might be a bit over the top. Many of the services you mentioned are consumer-oriented, and do probably not come with several 9s service guarantees. And I suppose there can always be network issues involved in the problems you mentioned.

I’m only using OneDrive for backups and iCloud for a few files (and I’ll likely move away from that onto my own NAS).

1 Like

And if you use OneDrive and/or iCloud heavily, you will suffer what I say.

@rfog @chrillek @rmschne Thank you for the livley discussion. :smile:

Yes, maybe I should consider moving away from Dropbox for my main file sync between machines: I do have a Synology NAS which I could open up to the internet to provide a sync system (on of my machines is usually not at home, so it would need to connect via the Internet).

However I dread the maintenance of this plus the security implications which come with opening up the Syno to the Internet…

BTW, I disagree with this:

I do want access to all documents stored in or indexed by DEVONthink. They don’t need to be downloaded, but in case I look for a document I want have the option to download it. (I use on the bigger databases DEVONthink’s Download-on-demand feature.)

Well, I think you’d regret to move away from Dropbox in favor of a self-managed Synology NAS exposed to the internet (security, network performance, etc). Up to you. Yes, there are different opinions/experience with Dropbox (and other third party sync services), but Dropbox is pretty solid, frankly–except of course if you use their “offline” features with DEVONthink or any other app that needs all files local all the time. Again, your choice to use or not is completely up to you. And you may “disagree” with me, but the fact is my needs and expectations are different than yours. iOS devices are more limited than macOS devices and I don’t expect them to have same capabilities with respect to CPU, storage space, network performance, etc. Disagreement is not the issue. If your needs are that important and your iOS device(s) works as you want, go for it. Me, if I know on my out-and-about use of my DEVONthink databases, I take the MacBook. Your mileage of course, will differ.

2 Likes

for my main file sync between machines

Please clarify this…

  • What file sync between machines?
  • Are you including Apple mobile devices in the mix?
    • Including DEVONthink To Go?

I’m trying to understand why you’re using Dropbox at all in this way.

1 Like

Thanks for taking my issue seriously. Here my clarification:

I have a Mac at home (connected via Ethernet to a Home network) and a MacBook at Work (connected to a quasi fast WiFi). I want to have teh stash of the information (PDFs, etc) I collect and the documents I write (Markdown and a few docx) availbe in both locations prefarbly directly through teh Finder.

Additionally I use an iPad on the go for searching and reading these documents, not so much for writing. For seraching and reading I rely on DTTG3 for searching all the information and for reading PDFs, and I use MarpleRead for ePUBs and longer PDFs. I don’t want to fill up my iPad with all accumulated PDFs (and webarchives), so having them as avaible for download on demand is perefct.

Extremely rarely I search my information my iPhone, but that’s not very often.

If I could do this via my Synology NAS which I can access VPN-style(?) from work and on the go, that probably would be as good (or better?) as my current Dropbox solution. (Dropbox does not work seamless for syncing: Again and again sync locations stop syncing correctly after a while - recreating sync locations fixes this then for a while…)

DEVONthink Server edition, securely configured and maintained?

Have been using iCloud syncing between DT on macOS and DTTG on iPhone and iPad for years, many GB of data, and this works in general very well. This includes syncing on the road. I guess there could be subtle differences that make or break the experience between different users. With large ticket items (DT, Apple Photos etc) I find that “download content to iOS on demand” does not work out. Whenever it comes push to shove, I end up being on a cell or weak wifi connection somewhere. So 256+ GB iDevices and full upfront sync are the answer for me. In fact, for this very reason, early on, DTTG started to be my “file system on iOS”, as neither Dropbox nor Apple Files allow(ed) to pre-emptively sync to iOS. The internet is ubiquitous, but not that ubiquitous.

2 Likes

Interesting idea! Can the server part run on the Synology NAS? (I don’t want to keep my Mac running all time.)