Thanks, going to read this!
Meanwhile, I opened a support ticket (over the link from @BLUEFROG above)
I did as recommended in this article and removed DT from Full Disk Access, then started it again - and I got asked to give DT access to Network Volumes, which I gave.
Also, I added it again to “Full Disk Access” …
Sadly, the problem remains.
And then I tested again:
Created a new text file from DTTG in the above …/ARCHIVE group:
It got uploaded to WebDAV and … bammmm… it also showed up fine in DT:
So, the problem only exists for certain file types, but not for others.
Very strange.
But DT can indeed download new files from WebDAV and write them to the indexed folders - just not for every content.
Did you check the ownership & permissions of the folders? Sometimes a permission issue is just a permission issue. You could also enable the option to ignore the ownership on the network volume.
I think I proved that this is not the problem:
As the logged in user, both a terminal shell and DT can write to the folder!
The permissions are also visibly OK and there are no ACLs that could be a problem.
DT can write itself to the target folder, when creating a new file from within DT!
And DT can write new files that it get’s over WebDAV when such a new file is created on an iPad (and uploaded to WebDAV).
But it does not work for all type of files, as it seems - of course the technical explanation may be different.
All of this was shown in my above text-snippets and screenshots.
DT strongly says, that there is no permission to access the folder “ARCHIVE” while it can write to it without problems for new textfiles - regardless of how it get’s added (by creating from DT or by creating over WebDAV from an iPad)
What excactly do you mean by this?
Anyways, DT can write to it!
Well, that’s at least the error returned by macOS. Therefore it’s either a wrong error message or maybe a temporary issue of the WebDAV server, e.g. maybe due to concurrent access by multiple devices.
I don’t know how to repeat this in a way that is more clear…
DT can add lots and lots of files coming from WebDAV without any problem at all.
There is no problem with WebDAV!
DT can also see all of the WebDAV files (as the names and sizes are shown) and it can download the WebDAV files - but only to some cache folder and then cannot move them to the destination folder:
At least, the log files says that the file got download to “NSSourceFilePathErrorKey=/Users/tja/Library/Caches/TemporaryItems/DEVONcloudy 12795/Main Thread/360/IMG_0001.png”
Also, DT clearly has the rights required to do this - as tested with textfiles:
Created on an iPad, the files get uploaded to WebDAV and DT on the Mac perfectly fine downloads them to the indexed folders!
This problem is strange but seems to be a local DT problem!
Or, of course something from macOS.
Meanwhile, I also rebooted the Mac just to be sure … no change
To prove that the download from WebDAV works, I let this run:
(base) tja@mini:~$ while ! ls /Users/tja/Library/Caches/TemporaryItems/DEVONcloudy\ 12795/Main\ Thread/* 2>/dev/null ; do : ; done
total 400
-rw-r--r--@ 1 tja staff 201443 Nov 22 14:38 IMG_0001.png
(base) tja@mini:~$
As you can see, the loop exited when the PNG file was correctly downloaded from WebDAV to the cache folder!
But DT cannot move it to the destination folder, as it says in the logs.
A network volume mounted in the Finder is actually not the same as directly accessing a WebDAV server (like the synchronization does). Did you also check the permissions of /Users/tja/Library/Caches/TemporaryItems/?
1 Like
Also, to show this again, I added 2 more files to DTTG:
So, they got added on iPad…
And then, looking on DT:
They appeared without problem!
But NOT the PNG files…
Per terminal:
(base) tja@mini:/Volumes/DT_MAIN/ARCHIVES$ ls
total 48
drwxr-xr-x 1 tja staff 192 Jul 30 23:17 archive/
drwxr-xr-x 1 tja staff 416 Jul 30 23:17 DO_NO_DELETE/
drwxr-xr-x 1 tja staff 160 Jul 30 23:17 BACKUPS/
-rw-r--r-- 1 tja staff 5 Nov 21 23:58 createdByDDTG.txt
-rw-r--r-- 1 tja staff 1087 Nov 22 14:38 daily_dates.log
-rw-r--r-- 1 tja staff 1681 Nov 22 14:38 rsync_XTRMQ-T7_Touch2TB.sh
Yes, please see above - the PNG get’s correctly downloaded to the cache folder!
But it cannot be moved - while other downloaded files can get moved!
Very strange
I found the problem!
First, how I did this:
I let a loop run to wait for DT to download a file from WebDAV and copied it to ~/TMP
(base) tja@mini:~$ while ! cp -p /Users/tja/Library/Caches/TemporaryItems/DEVONcloudy\ 12795/Main\ Thread/*/* ~/TMP/ 2>/dev/null ; do : ; done
This produced the following file:
(base) tja@mini:~$ ls ~/TMP/IMG_0001.png
-rw-r--r--@ 1 tja staff 201443 Nov 12 20:35 /Users/tja/TMP/IMG_0001.png
Now, I just tried to simulate what DT would be doing:
(base) tja@mini:~$ mv ~/TMP/IMG_0001.png /Volumes/DT_MAIN/ARCHIVES/
mv: /Volumes/DT_MAIN/ARCHIVES/IMG_0001.png: unable to move extended attributes and ACL from /Users/tja/TMP/IMG_0001.png: Operation not permitted
This gives a harmless error message, because the target file system cannot handle those unimportant ACLs or extended attributes …
The “mv” did work, of course:
(base) tja@mini:~$ ls ~/TMP/
total 0
(base) tja@mini:~$ ls /Volumes/DT_MAIN/ARCHIVES/
total 448
drwxr-xr-x 1 tja staff 192 Jul 30 23:17 archive/
drwxr-xr-x 1 tja staff 416 Jul 30 23:17 DO_NO_DELETE/
drwxr-xr-x 1 tja staff 160 Jul 30 23:17 BACKUPS/
-rw-r--r-- 1 tja staff 201443 Nov 12 20:35 IMG_0001.png
-rw-r--r-- 1 tja staff 5 Nov 21 23:58 createdByDDTG.txt
-rw-r--r-- 1 tja staff 1087 Nov 22 14:38 daily_dates.log
-rw-r--r-- 1 tja staff 1681 Nov 22 14:38 rsync_XTRMQ-T7_Touch2TB.sh
So, the file is where it should be.
The only explanation for this is, that DT “interpretes” the apparent error message from the move, because of the missing ACL / extended attribute support und simply removes this file again!
This is very frustrating, as I see no way to fix this by myself!
Files from anywhere may have added ACLs / extended attributes… and I have no way of removing them.
Only an option to DT that allows to ignore ACLs / extended attributes or ignores the error messages from “mv” could help.
I cannot change the fact that some files come with some ACL / extended setting - but I have no interest in them!
Any chance for such an option to DT?
Any other idea how to fix this?
Many thanks
As it seems, some files from iPhone or iPad just come with ACLs / extended attributes set and WebDAV seems to be able to handle them too.
But not all file systems can do this!
This is normally not a problem at all …
You just get a message that the ACLs / extended attributes could not be copied.
But DT handle this in a way that such files can never ever be downloaded!
This is a problem.
I also checked, what exactly is so important althought it should be unimportant:
(base) tja@mini:~/TMP$ ls -le IMG_0003.png
-rw-r--r--@ 1 tja staff 151794 Nov 12 20:35 IMG_0003.png
So it’s not ACLs!
(base) tja@mini:~/TMP$ ls -l@ IMG_0003.png
-rw-r--r--@ 1 tja staff 151794 Nov 12 20:35 IMG_0003.png
com.apple.assetsd.UUID 16
com.apple.assetsd.addedDate 50
com.apple.assetsd.assetType 2
com.apple.assetsd.avalanche.type 2
com.apple.assetsd.creatorBundleID 21
com.apple.assetsd.customCreationDate 50
com.apple.assetsd.dbRebuildUuid 36
com.apple.assetsd.deferredProcessing 2
com.apple.assetsd.favorite 2
com.apple.assetsd.grouping.state 8
com.apple.assetsd.hidden 2
com.apple.assetsd.importedBy 2
com.apple.assetsd.originalFilename 12
com.apple.assetsd.publicGlobalUUID 36
com.apple.assetsd.timeZoneName 13
com.apple.assetsd.timeZoneOffset 4
com.apple.assetsd.trashed 2
com.apple.assetsd.videoComplementVisibility 2
It’s Apple crap in extended attributes!
And finally, this is really a problem of DT:
Both “cp” and “mv” work flawlessly and do NOT return with an exit code that is not 0:
(base) tja@mini:~/TMP$ cp -p ~/TMP/IMG_0003.png /Volumes/DT_MAIN/ARCHIVES/
cp: /Users/tja/TMP/IMG_0003.png: could not copy extended attributes to /Volumes/DT_MAIN/ARCHIVES/IMG_0003.png: Operation not permitted
(base) tja@mini:~/TMP$ echo $?
0
(base) tja@mini:~/TMP$ mv ~/TMP/IMG_0003.png /Volumes/DT_MAIN/ARCHIVES/
mv: /Volumes/DT_MAIN/ARCHIVES/IMG_0003.png: unable to move extended attributes and ACL from /Users/tja/TMP/IMG_0003.png: Operation not permitted
(base) tja@mini:~/TMP$ echo $?
0
This proves that the commands work totally find - the error is just a message that should be ignored, it is NOT an actuall error that would exit != 0
And finally, this may well explain my constant problems with the number of items being different between iDevices and DT on Mac …
Actually DEVONthink doesn’t use shell scripts for copying/moving but Cocoa’s filemanager instead which returns a useless error presented by DEVONthink unfortunately.
What kind of filesystem does the server actually use? I’ve never experienced this on my own despite using various WebDAV servers. And how did you exactly add/create these files in DEVONthink To Go?
Anyway, thanks for the useful debugging - highly appreciated.
Ah, I replied to this by email.
Did you get this?
I didn’t receive any email actually. Or do you refer to your ticket?