Regression in DT? Cannot download from WebDAV again :-(

tja · August 8, 2023, 5:01pm

Hello again,
I moved to a new Mac …

For this, I installed the DT binary, and before starting it, I copied the old “~/Library/Application Support/DEVONthink 3/” folder from the old to the new Mac.

Also, as I use indexed content, I made sure that the content was available at the same location.

So far, so good - most seemed to work.

But after adding my Sync location to DT (a WebDAV server) DT started downloading lots of things - presumable because I enabled “Always Verify” - even as all content should be there already.
Is this already a normal behaviour?

Whatever, I noticed that 2 databases keep downloading and downloading the same number of files: One 315 files, the other 57 files.

I checked those and tried to move them new from a backup to the indexed location and noticed such messages:

mv: ./IMG_3963.jpeg: unable to move extended attributes and ACL from /Users/tja/BAK/IMG_3963.jpeg: Operation not permitted
mv: ./IMG_3963.jpeg: set owner/group (was: 506/20): Function not implemented

This strongly reminded me of an older problem, where @BLUEFROG (IIRC) finally fixed this - and it ever worked since then.
There was a support ticket and a topic in the forum about this: Checksums ... ;-) - #39 by pete31

Whatever, now I am at macOS Ventura … and it seems, some similar problem appeared.

If I recall correctly, the problem was that some files, depending on their origin, can cleanly be copied or moved or downloaded, but not all extended attributes or other related stuff - so even as the command does return with 0, which is OK, it does output something to STDERR and this let’s DT think that the operation did not work.

Any chance that you could look at this again, @BLUEFROG ?

tja · August 8, 2023, 5:17pm

Ah, and in DTTG those files are perfectly fine - and in WebDAV too, as it seems.

tja · August 8, 2023, 5:35pm

And one more detail:

The second part of the warning message is because of my user name has different UIDs on the old and the new Mac.

BLUEFROG · August 8, 2023, 6:33pm

Did you give DEVONthink Full Disk Access in System Settings > Privacy & Security?

The second part of the warning message is because of my user name has different UIDs on the old and the new Mac.

The account name should have no effect on indexed items.

tja · August 8, 2023, 6:39pm

Yes, I did - and not just yet

I tried to fix the situation by copying all files (all databases where affected as seen with a item:pending Smart Group) by copying the missing files from the last backup, which was OK for most.

In one database, I got errors like this, when checking with “Check File Integrity”:

“8.08.23, 20:24:54: DT_XXX > DT_XXX > Revision A1 > IMG_3946.jpeg File doesn’t have a checksum yet.”

This exactly reminds me of the earlier case, as mentioned above.

BLUEFROG · August 8, 2023, 6:59pm

Did you migrate the databases via Migration Assistant or were you importing the databases from the sync location?

I checked those and tried to move them new from a backup to the indexed location and noticed such messages:

mv: ./IMG_3963.jpeg: unable to move extended attributes and ACL from /Users/tja/BAK/IMG_3963.jpeg: Operation not permitted
mv: ./IMG_3963.jpeg: set owner/group (was: 506/20): Function not implemented

Is this an external drive? If so, a new one?

tja · August 8, 2023, 7:36pm

No, I don’t know the Migration Assistant.
I was searching in the forum and read about copying the directory from my Library.

I also copied my ~/Databases folder with the databases and only then started the database.

The external drive with the indexed content is the same, I just connected it to the new Mac.

Currently, I don’t think that the migration is the problem here - the error messages hint at problems with handling the downloaded content, exactly like in the case before.

I could test this, by stopping DT, removing one of the problematic files from the indexed location - then, DT should try to download it from the Sync location again, right?

BLUEFROG · August 8, 2023, 7:58pm

If the sync data is present in the sync location, you would have to option of downloading the content when you have the item selected.

The external drive with the indexed content is the same, I just connected it to the new Mac.

Show the Get Info pane for the drive in the Finder. Is it ignoring ownership on the volume?

tja · August 8, 2023, 10:16pm

Yes, the Sync location contains everything!

I tested it as follows:

Stop DT
Removed a file from the indexed location
Start DT
Check the file: It is gone from DT …
Update indexed items: No change …

This does not seem to be the right way to trigger a download from the Sync location.

About the “Get Info” from one of the problematic volumes:

BLUEFROG · August 8, 2023, 10:47pm

You have an unusual setup here. Hold the Option key and choose Help > Report bug to start a support ticket.

tja · August 8, 2023, 11:30pm

That does not work, as DT only works with Apple Mail, which I don’t use

I could write the mail from an iPad, but that sends the wrong logs, of course.

So, just a mail to support AT devontechnologies.com?
And attach which files or other data?

Can only send tomorrow …

tja · August 8, 2023, 11:43pm

Even as I should sleep already, I had an idea:

Just duplicate one of the items on an iPad with DTTG, which then would get reflected in DT as a new file which cannot be downloaded!

And this is true!

Also also, the not downloaded file can be seen in DT - because the cache for images did work, before DT deleted the file again:

2023-08-09 01:37:36,776 ERROR: Error Domain=NSCocoaErrorDomain Code=513 ““IMG_3930 copy.jpeg” couldn’t be moved because you don’t have permission to access “Revision V1”.” UserInfo={NSDestinationFilePath=/Volumes/DT_XXX/Revision V1/IMG_3930 copy.jpeg, NSUserStringVariant=Move, NSFilePath=/Users/tja/Library/Caches/TemporaryItems/DEVONcloudy 2372/Main Thread/da/IMG_3930 copy.jpeg, NSUnderlyingError=0x60001712a430 {Error Domain=NSCocoaErrorDomain Code=513 ““IMG_3930 copy.jpeg” couldn’t be copied because you don’t have permission to access “Revision V1”.” UserInfo={NSSourceFilePathErrorKey=/Users/tja/Library/Caches/TemporaryItems/DEVONcloudy 2372/Main Thread/da/IMG_3930 copy.jpeg, NSUserStringVariant=(
Copy
), NSDestinationFilePath=/Volumes/DT_XXX/Revision V1/IMG_3930 copy.jpeg, NSFilePath=/Users/tja/Library/Caches/TemporaryItems/DEVONcloudy 2372/Main Thread/da/IMG_3930 copy.jpeg, NSUnderlyingError=0x60001712bab0 {Error Domain=NSPOSIXErrorDomain Code=1 “Operation not permitted”}}}}
2023-08-09 01:37:36,776 INFO: Database DT_XXX: 1 items added, 0 items modified, 1 items changed, 0 items deleted
2023-08-09 01:39:37,186 ERROR: Error Domain=NSCocoaErrorDomain Code=513 ““IMG_3930 copy.jpeg” couldn’t be moved because you don’t have permission to access “Revision V1”.” UserInfo={NSDestinationFilePath=/Volumes/DT_XXX/Revision V1/IMG_3930 copy.jpeg, NSUserStringVariant=Move, NSFilePath=/Users/tja/Library/Caches/TemporaryItems/DEVONcloudy 2372/Main Thread/df/IMG_3930 copy.jpeg, NSUnderlyingError=0x60000f3725e0 {Error Domain=NSCocoaErrorDomain Code=513 ““IMG_3930 copy.jpeg” couldn’t be copied because you don’t have permission to access “Revision V1”.” UserInfo={NSSourceFilePathErrorKey=/Users/tja/Library/Caches/TemporaryItems/DEVONcloudy 2372/Main Thread/df/IMG_3930 copy.jpeg, NSUserStringVariant=(
Copy
), NSDestinationFilePath=/Volumes/DT_XXX/Revision V1/IMG_3930 copy.jpeg, NSFilePath=/Users/tja/Library/Caches/TemporaryItems/DEVONcloudy 2372/Main Thread/df/IMG_3930 copy.jpeg, NSUnderlyingError=0x60000f3739f0 {Error Domain=NSPOSIXErrorDomain Code=1 “Operation not permitted”}}}}
2023-08-09 01:39:37,186 INFO: Database DT_XXX: 0 items added, 1 items modified, 0 items changed, 0 items deleted

Going to send more details in a mai, tomorrow.

Of course the operation is permitted and I (or DT) has the right to access the folder!

This is exactly the same situation as we had in the earlier case, from what I see right now.

BLUEFROG · August 9, 2023, 12:02am

Send the info in a file. Do not copy and paste the log info into the support ticket directly.

tja · August 9, 2023, 8:56pm

Wrote 2 mails

BLUEFROG · August 10, 2023, 2:09am

Thanks. I’ll take a look.

tja · August 21, 2023, 5:51pm

Now, as it seems, DT is “just moving” content that got downloaded from the Sync location to the target location.

And if this content has “extended attributes” (very likely at content from iDevices) while the target location’s file system does not support extended attributes, this move does not succeed - either because 1) the operation of macOS just fails, or because 2) DT interpretes the results wrongly and roll’s back the action by itself.

From what I did see and check, it seems that this is case 2 - so only DEVONthink Technologies could fix it…
As it seems, they don’t want to fix it, from how I understood Jim Neumann.
This is indeed a very very sad thing.

There are only 2 possible workarounds for this:

a) Use WebDAV (Apple Script) instead of macFUSE for the Cryptomator volumes - this allows extended attributes to be written, but has severe problems with reliability.

And then run “xattr -r -c …” on all folders, or from the top.

b) Use macFUSE, but don’t use the option “noappledouble”, so that the file system will be filled with thousands and millions of Apple Double Files - one file for each item that has extended attributes.

And then run "find . -name ‘._*’ -exec rm -f {} ; on all folders, or from the top.

Those Apple Double files are 4k each, and simply double the number of files. Which is both horrific to me.

This is a tough decision to make.

BLUEFROG · August 21, 2023, 6:38pm

As i said in your ticket: You are not running a standard (or common) setup and you are not writing to a native Apple filesystem. The AppleDouble files are a byproduct of what you’re trying to do with macFUSE, not something DEVONthink is doing.

tja · August 21, 2023, 6:41pm

Yes, I understood and wrote the same above.

Just wanted to inform the potential readers of this thread.

So far, I tend to use macFUSE and allow Apple Double files.

As I do not run on the online cloud data, I can then remove the Apple Double files before actually uploading to the cloud.

It still would be much appreciated if DT could just ignore any error messages or output for STDERR for such a “move” operation - as it works flawlessly and the command does not return with am exit code that is different to 0, from what I saw.

Thanks for your time, going to implement my workaround.

BLUEFROG · August 21, 2023, 7:22pm

You’re welcome and development knows of this particular situation.

tja · August 21, 2023, 8:20pm

Great many thanks!