Checksums ... ;-)

I read several topics about error messages in the log about missing or invalid checksums.

Also, I read that updating the item or indexed item or exporting, deleting and re-importing can help.

But …

I don’t really get, why there is no option to just read all the files with such checksum errors again into DT.

When DT has a bad checksum, why does it not automatically update the checksum?

I keep having such errors for files, both in the global inbox and in indexed folders and don’t really want to handle them seperately …

The indexed files from some folder are fine!
When DT has problems with any checksums, why not create new checksums with some option?

Also, there are files that do not seem to have a checksum at all - but running all options like “Verify & Repair Database” and “Check file integrity” do not seem to create the missing checksums. Why?!?

Here some examples:

Screenshot 2022-11-12 at 23.32.41

Screenshot 2022-11-12 at 23.51.09

When DT has a bad checksum, why does it not automatically update the checksum?

Well, it would increase the logging for one… or… it would silently fix the issue and no one would know there was an issue. The latter sounds problematic to me. The former may seem acceptable to some but to others who are easily annoyed by excess chatter, they may have other opinions.

lso, there are files that do not seem to have a checksum at all - but running all options like “Verify & Repair Database” and “Check file integrity” do not seem to create the missing checksums. Why?!?

@cgrunenberg would have to comment on this.

1 Like

I just was expecting those checksum errors to be gone after running any of the maintenance commands :sweat_smile:

For indexed items, DT would just need to “re-losd” the file from storage and also update the checksum.

Is there any command to actually achieve this, or is it really required to do the mentioned steps (export and re-import, or select one by one and run update, or whatever else method) on affected items, one by one?

Of course I understand the nature of checksums, so it could be that DT has the right checksum but that original file is bad - in case of indexed items, there is no cure for this.

The second command just checks the integrity and reports issues, the first one verifies & repairs the database (basically its internal consistency & structure). This does not include checksums as it’s unknown whether the checksum is invalid or the file.

Are the reported files indexed or imported one? And do the files without a checksum still exist and are readable (e.g. valid permissions)? In that case it should be immediately calculated after opening the database.

1 Like

I managed to fix most of the files by one of the above methods, but still have 1 file without a checksum and 1 file with invalid checksum in one database, and 3 files with invalid checksums in a second database.

All of those remaining issues are in indexed databases (only my global inbox is not indexed)

I don’t usually close DT, ever.
My Mac runs for weeks and months and so does DT.

Is there a way to update such files like without closing and reopening the database or DT?

I checked the files in question and most seem totaly fine in the indexed location, but one file is giving an Input / Output Error!

I am going to restore this file from backup.

That’s the only possibility right now (as it should be rarely necessary).

At least in this case the file integrity check was definitely helpful.

1 Like

Many thanks.
I will test in the evening!

And yes, it is good to know that an actual corruption would be found!
:hugs:

I closed DT, opened it again and did the File Integrity Check for 4 databases:

Screenshot 2022-11-17 at 19.34.55

So, 2 databases still have files with bad checksums.

I closed and restarted DT, closed and reopened the individual databases and of course also did “update indexed files”, I then tried Verify & Repair and again Check File Integrity.
Same result

The files with a bad checksum seem totally fine (PDF files).

This is really frustrating!

I tried the following:

Renamed the file (in terminal shell), updated indexed files, checked file integrity: No change

Copied the file to a new name, updated indexed files, removed the original file, updated indexed files, renamed the new file to the old file name (all in the indexed folder, using a terminal shell), updated indexed files, checked file integrity: DONE

This horrible sequence managed to update the checksum.

Really, there NEEDS to be a better way.

I am now doing the same for the other file(s).

EDIT: This was only so “easy” as the affected files did not have tags or something … luckily

Rebuidling the database is one but might require some time.

1 Like

Thanks, I will try this next time - does this work work indexed folders too?
I mean, will it just re-read the indexed files or actually delete and replace them?

actually delete and replace them?

No, this does not happen when rebuilding a database.

1 Like

May I add another, related question?

After my “fixes”, I noticed that DT did again and again try to download some files.

I checked the database and folder (indexed) and tested it by adding new files from an iPad.
The old and the new files got uploaded to WebDAV fine and appeared on the other iPad!

But DT on the Mac only shows the files with a cloud and download symbol.
Also using thr Download Pending Files option did not change that.

It seems that DT can see the new files but not really download them - this cannot be a problem with the WebDAV server, as the iPads work fine!

It can also not be a problem with the local (indexed) filesystem, as I can drag and drop files on the Mac and they appear fine in DT!
Only the files from WebDAV seem to produce this problem - and as it seems, only for one database, not the others.

But what else could be the cause and how to fix this?

I did rebuild the mentioned database and got the message that those files were missing and they got removed from the database.

Adding one file again on an iPad also makes it available on the other iPad, but on DT only the name of the file and the cloud symbol appear.

So, the situation did not change.
Very strange, never saw that before.

For the record, I manually copied an existing file and the copy did appeat well in DT.
I then copied a file within DT and this also worked fine, including the new copy to appear in the folder.

So it seems to be more related to downloading from WebDAV …
Just trying to figure out what’s wrong :slight_smile:

I FOUND IT:

I tested with a another database, that worked fine before on other content and got the same behavior.

This lead me to test other files: They work totally fine!

The problem is a set of 32 PNG files!
They cannot be be downloaded on Mac!

They were created as screenshots on an iPad.

Going to create a new PNG and test with this again

I tested this further, by adding the same files and copying them first to the Mac and then move them into the indexed folder - this works fine!

But uploading the files on DTTG on any iDevice will make them appear on any other iDevices (over WebDAV), but DT on Mac cannot download them from WebDAV.

Is there some log that could explain the problem?

When testing, I sometimes saw the the following error on an iPad:

Cannot load representation of type public.png.

Very strange