Index folder accessed by iCloud

[mog]

I have read a similar/same question on this forum. but it doesn’t answer my question.

1.0 Some background. Last weekend, following a mistake I made without realising at the time, the connection between my Mac and my network scanner disconnected. The scanner IT people have been very helpful but unable to resolve the problem because it’s not clear where it is. Yesterday, I had a one hour call with AppleCare, the person identified the issue and resolved it. It appears that the problem was caused several years ago but. only emerged as a consequence of what I did last weekend. I knew that when I changed my computer a few years ago (iMac 27” to Mac Studio) almost all the folders came in via Dropbox even though I didn’t set out for that to happen. Having worked with most of the folders in a mess, I decided last weekend to tidy. Two things I did which I subsequently discovered must never be done was to rename the Desktop and Documents folders, because both.contain hidden system files.

1.1 I changed from Windows PC to Apple Mac in 2008 or thereabouts and bought into DEVONthink. Despite the 17 years experience of several Macs and OSs, yesterday’s conversation with the AppleCare person was the first time I learned how the folder hierarchy works. The person also told me that having iCloud and Dropbox clash and best to have just one, given that I also have and use Time Machine - (he showed me how to access Time Machine without using the ‘concertina’ and it’s much easier and quicker.)

1.2 Probably others know this, but I didn’t so it might be helpul for those that don’t. At root level (Macintosh HD) the system folder Users is the parent of Desktop and the user account folder the parent of Documents. System folders have an icon/logo on the front of the folder, as do application folders. One’s own folders can be added to Desktop and Documents; Desktop folder envisaged for working, Documents for everything else. In my Documents folder, I have a top level sub-folder Folders All for my sub-folders and files. I have added Folders All to Finder Favourites for my ease of reference and use. (I have hundred of folders whose names include the word ‘Documents’ so having them i n a separate sub-folder is easier for searching.)

2.0 When c’onnecting with iCloud, iCloud ‘removes’ Documents from system Users folder and creates a new folder Documents-Mac whose sub-folders are all the sub-folders in Documents. When disconnect iCloud, iCloud reverses that procedure. This brings me to my question.

For indexing files, Devonthink4 (DT4) links to a folder in Documents on the local HD but when I connect to iCloud that Documents folder is ‘taken’ by iCloud. What do I need to do to prevent iCloud from ‘taking’ the Documents folder. Alternatively, if were to put and keep the sub-folder on a external disk would that work for indexing?

Update

I have contacted Apple Care. I am told there is no way of excluding a folder from iCloud, Also, no way of adding a new folder above Users at HD level because iCloud takes them all. Therefore, the only solutions, subject to anything DT4 people and anyone else on this forum can come up with, are either to put the folder onto an external HD (presupposing that would work) or import the files into DT4.

[MsLogica]

@mog

I have contacted Apple Care. I am told there is no way of excluding a folder from iCloud, Also, no way of adding a new folder above Users at HD level because iCloud takes them all. Therefore, the only solutions, subject to anything DT4 people and anyone else on this forum can come up with, are either to put the folder onto an external HD (presupposing that would work) or import the files into DT4.

I’m not sure that I’ve read this correctly. Are you saying Apple told you that there’s no way to isolate folders/files from iCloud if you have iCloud enabled on your Mac? Because that is categorically untrue, most of us using iCloud in here have probably done that, and I certainly have. I use GIS for work, there’s no way my iCloud account would work if it was trying to sync those humongous files all the time I also don’t sync my music library since it’s FLAC and therefore takes up a huge amount of space.

If you’ve got iCloud sync enabled, it syncs desktop and documents (desktop used to be optional but doesn’t seem to be any more). If you want to maintain local files that never go to the cloud, you need to set up folders in the home directory. You might need to add it to your sidebar navigation. In this screenshot, here is my home directory and all the folders that are only stored locally. Your DT databases are here unless you fiddled with settings during installation, because DT databases cannot be stored in a cloud.You can create new folders in here if you want to, you can see I have various folders that don’t come installed on a Mac.

Obviously though, these are all local and don’t sync anywhere…

[rmschne]

@MsLogica

Because that is categorically untrue, most of us using iCloud in here have probably done that, and I certainly have.

Beat me to it.

[MsLogica]

Hehe. In fact my screenshot has accidentally captured what’s syncing to the iCloud, as it’s got little cloud symbols next to the document and desktop folders, the only two that are syncing! Everything else is local only

I am however assuming I’ve missed the point and that there’s a different issue at play here that OP is having problems with. I’m choosing to believe Apple Support is better trained than that

[rmschne]

@MsLogica

I’m choosing to believe Apple Support is better trained than that

My experience with Apple Support (via phone and in Store) is quite good and they are well-trained with Apple stuff. Mainly as I limit my queries to things I know is Apple related, especially hardware. 3rd Party software stuff … we’ll, mixed results. I’m a sceptic by training and orientation and without prejudice say one must take what they say with care.

[mog]

Thank you and others.

While waiting for comment, I disconnected iCloud, then copied the folder onto an external hd, then reconnected the iCloud. I am now indexing to DT4 from the external hd. The indexing hasn’t completed yet but so far. all ok.

As you rightly say DT4 created a new folder in the home director for Databases so presmunably? there’s nothing to prevent copying (or cutting) the folder from the external HD and pasting in Databases folder? As for adding new files, I shall add them direct to the (sub) folder in Databases and copy. and paste new files to the sub-folder in Documents that iCloud ‘takes’.

[chrillek]

@mog

At root level (Macintosh HD) the system folder Users is the parent of Desktop and the user account folder the parent of Documents.

Nope. /Users is the parent of the user folders, eg /Users/mog. For each user, Desktop and Documents are children of a particular user folder, eg /Users/mog/Documents. In other words: Documents and Desktop are children of the “user account folder”

@mog

The person also told me that having iCloud and Dropbox clash

They might have said that, but did they also say under which conditions that might happen? I believe that literally millions of people use iCloud and Dropbox at the same time without having them clash. A statement like that makes no sense without a proper explanation of the details to these “clashes”.

@mog

Desktop folder envisaged for working, Documents for everything else

Very much a question of personal taste, imo. What even is “working” as opposed to “everything else”?

@mog

When c’onnecting with iCloud, iCloud ‘removes’ Documents from system Users folder and creates a new folder Documents-Mac

Only when you tell it do “watch” you Documents folder. Which you should only do if you know what that entails. I would never entrust Apple to “optimize” my local folders. None of them.

@mog

For indexing files, Devonthink4 (DT4) links to a folder in Documents on the local HD

DT does not “link” to this folder; it indexes the folder. Meaning that it goes over the content of that folder and extracts metadata for all the files and folders in it, recursively.

@mog

What do I need to do to prevent iCloud from ‘taking’ the Documents folder.

Finder, Preferences, turn off the checkbox at the bottom.

@mog

I am told there is no way of excluding a folder from iCloud,

BS. Just turn that bloody check box off.

@mog

Also, no way of adding a new folder above Users at HD level because iCloud takes them all.

This sentence makes no sense. whichever way I look at it.

[troejgaard]

@mog

Despite the 17 years experience of several Macs and OSs, yesterday’s conversation with the AppleCare person was the first time I learned how the folder hierarchy works.

I’m not sure you fully understand it yet.

I have contacted Apple Care. I am told there is no way of excluding a folder from iCloud, Also, no way of adding a new folder above Users at HD level because iCloud takes them all.

My guess is the support person was trying to tell you that you can’t exclude children (i.e. sub-folders and files) of a parent folder that is synced by iCloud. If Documents is synced, so is everything in the hierarchy below it. You can’t exclude, for example, Documents/My Project/Notes.

But you don’t have to put all your files in ~/Documents! Or even sync ~/Documents and ~/Desktop with iCloud. I personally never liked that idea. I just use the “iCloud Drive” folder, and only for certain limited things.

For indexing files, Devonthink4 (DT4) links to a folder in Documents on the local HD but when I connect to iCloud that Documents folder is ‘taken’ by iCloud.

iCloud doesn’t ‘take’ anything, it synchronizes what it’s told to. But if you let it sync your Desktop & Documents, their paths in the filesystem change, and DEVONthink doesn’t know where they are anymore.

Did you read the In & Out > Importing & Indexing section in the manual? See in particular Moving Indexed Items. With a little forethought, it’s no problem to update the paths of indexed items in DEVONthink.

[mog]

@chrillek

Finder, Preferences, turn off the checkbox at the bottom.

Thank you very much.

In the same way that in my field my knowledge is deeper than most so I can often find fault with what someone else, in the same field, says but whose knowledge is not as deep, my computer and DT knowledge is superficial.compared to yours!

If I turn off the checkbox as you say then how do I get iCloud to sync my Folder All? Would I remove this particular folder, Law, and put it next to, for example, DT’s Database folder that DT has put in User eg /Users/mog/

I’m begiining to think it would be easier to import all the files into a DT database. (currently approximately 4,000 pdfs) Then I could leave the source folder in the Documents folder for iCloud.. Alternatively, stop using iCloud (turn off the Documents and Desktop as you say) and rely on Time Machine.

[chrillek]

Why do you want iCloud to sync any folders at all?
If you really need that, put your folder in iCloud Drive and index that in DT.

[mog]

My understanding that DT doesn’t like index to a cloud folder, icloud or dropbox, for example.

On page 234 of Take Control of DEVONthink 4 “Note that DEVONthink expressly prevents you from storing your database in any cloudsynced location, such as Dropbox or iCloud Drive (including the Desktop and Documents folders, if you sync those via iCloud), because such locations aren’t safe to use with databases. You can, however, sync your databases via Dropbox or iCloud Drive…

Actually, I tried to index it to the folder in Cloud earlier today. but it struggled, took ages and in. the end I have to. force quit DT

[chrillek]

@mog

My understanding that DT doesn’t like index to a cloud folder, icloud or dropbox, for example.

Not correct. Where does that understanding come from?

@mog

Note that DEVONthink expressly prevents you from storing your database in any cloudsynced location

Where do you read “indexed” there? Indexing is not the same as syncing.

[MsLogica]

Ok, you absolutely can index folders that are in a cloud service (iCloud, Dropbox, etc.). What you can’t do is store your databases on a cloud service.

I think though, stop everything you’re doing before you accidentally break something, and explain what it is you want to achieve, and we can tell you how to do it. Starter questions:

• What is it you want to use DT for?
• Do you need files in DT accessible to other apps?
• Do you need DT and the files in it to be accessible on other devices?

[mog]

I’ve been using DT since 2008. I use this particular database for my law library, approximately 4000 pdfs in a dedicated folder. Before I started using iCloud very recently, I only used Dropbox. The start of the problem is unrelated to DT. As I said in paras 1.0 and 1.1 of my o/p, the problem concerns my scanner and its disconnection from the Mac. When I spoke with the Apple Care person, he explained what had happened and under his guidance i tidied the basic root folders. He advised that the quickest way to ensure everything as it should be would be to erase the HD but I was concerned that.would erase my email setting which are a fiddle -and of which. i have about 850 messages in my inbox currently and dozens of other mail box folders with thousands of emails in them, also I would be dependent upon the Time Machine back-up. The only alternative he said is to go through all the folders and delete the files and folders i don’t want manually. I preferred that option and have made a start. On his advice, I selected iCloud to connect to the Desktop and Document folders.

I deleted my earlier Law database in DT4 which had been indexed because i’d moved the source folder and all the files were missing I opened a new database similar name and put it in the Documents folder expecting to be able to index from there. However, because the Documents folder is now in iCloud, the usual slick first index I normally experience when indexing was a struggle for DT, spinning wheel, etc, and in the end I had to force quit, Effectively what I’ve done is store the database in a cloud folder contrary to DT;s operational requirements. From this thread, I undertand now that i should not store the DT database in a cloud folder so i could either deselect it or put in in a folder iCloud cannot get at. If I deselect it then the other folders and files in the Documents folder would not sync and those I want to sync. I contacted Apple Care to find out if it is possible to isolate a single folder inside Documents and it’s not:

Subject to anything you might add, it seems to me my only choice is either to store the DT4 database.and source pdfs in a single folder or two folders both out of reach of cloud, in which where should the folder(s) be, or import the pdfs to the DT database, making sure the source folder with the pdfs is not connected to cloud during the import process in case that.slows the import pace.

I do not need files in DT accessible to other apps: if I were to import the pdfs then I’d have two sets of pdfs and that would be ok.

I need the pdfs to be accessible on other devices, but i don’t need DT to be accessible.

---

{i should add that using DT4 is now going to be a tempoary stepping stone because i am gradually transferring all the content of files and notes to a private access website - with the help of a website developer designed to my own requirements - on which if need be I can add a pdf to each entry. Temporary likely to be several years because i have three other apps with information to transfer beforehand and it is a one-at-a time process with more 10,000 items, of which i have done about 1,000. The website developer suggested we automate the process, but I don’t want to. I would miss out on the CPD.benefit of reading everything carefully and would still need to go through each item to amend for consistency. The objective is to have everything important for my work readily accessible in one place at substantially lower cost than having to subscribe (payment) to other people’s ideas of how the information I need should be presented. I had high hopes, for example, for AirTable but in many respects have found it wanting and now that whenever I log-in and search for all records it asks whether I want them all as it would take some time, I am left wondering why its servers cannot cope with approximately 9,000 records. Also, there is a tremendous amount of wasted space on the appearance; and now that all and sundry are into Ai, its clutter is in my face whenever despite it being of no practical.interest for my needs - i am more than capable of reading 100 page documents and listing the important points in a minute or two. And for my writing reports generally at least 5,000 words and often towards 10,000 at a time in 2-3 days of concentration the pleasure in expressing what I want to say to suit the circumstnces is I think best summed up in what a client told me recently after reading my report for an arbitration: “well done Mog, you have completely destroyed Mr. X’s credibility and undemined his reasoning”.]

[BLUEFROG]

The Databases folder in your home directory is the default location for your databases. It is isolated from any iCloudery .

On his advice, I selected iCloud to connect to the Desktop and Document folders.

Well, of course that’s what he’d advise. Apple certainly wants everyone to do this, especially as it’s $ in their pockets. It’s not what we recommend, though the choice is up to the individual.

[rmschne]

@mog

He advised that the quickest way to ensure everything as it should be would be to erase the HD

This is also a common “fix” suggested by Apple. I recall I heard it a few times over the decades, but always ignored that advice. Normally, especially to fix a scanner connection, would be a last and regretted choice–if ever.

[MsLogica]

@mog

it seems to me my only choice is either to store the DT4 database.and source pdfs in a single folder or two folders both out of reach of cloud, in which where should the folder(s) be, or import the pdfs to the DT database, making sure the source folder with the pdfs is not connected to cloud during the import process in case that.slows the import pace.

And

@mog

I need the pdfs to be accessible on other devices, but i don’t need DT to be accessible.

This is not a complicated setup and lots of people will run DT like this.

You need to:

1. Create your new law database in the correct location, which is the default place that DT creates databases (basically, just don’t change anything when you create it and let DT put it in the right place)
2. Put your law folder with all your PDFs in it in Documents if you want to and want it on the cloud.
3. Right click on the law folder in Finder and choose “Keep downloaded” (this keeps the files on your local hard drive as well as in iCloud and helps DT). It may take some time if the files aren’t currently on your hard drive.
4. Index the law folder with DT.
5. Leave DT to do all its metadata gathering and whatever, it will take time because it’s a big folder!

You will have: one set of your PDFs that is accessible to DT, accessible to other apps, and available on iCloud for other devices. Any changes you make to the folder will be synced wherever.

[MsLogica]

P.S. It feels like an appropriate time to also suggest after you’ve done all of this you create a full back-up of everything so that you don’t have to do this again.

[troejgaard]

@mog

I deleted my earlier Law database in DT4 which had been indexed because i’d moved the source folder and all the files were missing

That seems a little premature. Was the indexed “Law” folder the only thing in that database? You didn’t lose anything by deleting it – no annotation files, no custom metadata, nothing? Did you ever use item links pointing at these records?

If not – alright, doesn’t matter. But otherwise, as I tried to tell you earlier, it shouldn’t be a problem to update the “missing” file paths. From the help/manual:

Example:

[mog]

@MsLogica

It feels like an appropriate time to also suggest after you’ve done all of this you create a full back-up of everything so that you don’t have to do this again.

Thank you all for your advice, etc. .. Much appreciated.

Resuming work earlier today (UK) I experienced two things that have put me off using iCloud for anything other than what I describe as the very basics (for example, Notes).

(1) The content of some sub-sub-folders in two key sub-folders that in Documents in iCloud does not contain some sub-sub-sub folders and files from the previous two.key folders that iCloud put in iCloud Archive. I cannot afford for folders and files to go ‘missing’ as such as that would mean wading through Time Machine backups to check whether I need to restore missing folders and files before starting work on something.

(2) On printing to pdf and save the pdf to a folder in iCloud, there is a slight lag that is off-putting.

I’m starting again from scratch. I’ve disconnected Documents and Desktop from iCloud, am copying all folders and files from iCloud and ICloud Archive and restoring all folders and files that i’d deleted in iCloud before the 30 days are up. *Restoring is taking ages, I don’t think I deleted that many.

I have Time Machine on automatically permanently so I’ll back-up manually every few minutes.

And once I have everything as I like it, I’ll index my Law library to DT just like it always was before.