Urgent Bug: Custom Metadata

DEVONthink
1

Help.

I can no longer add and persist custom metadata and I lost custom metadata for hundreds of files. This only happened today. Was fine yesterday.

Devonthink 3.9.8
MacOS 15.1.1

2

Perhaps, if not done already, do a reboot of the machine?

3

Yes I did. I also updated to MacOS 15.2, no change.

I can try my backups when I get home.

After my moment of panic I did more checks. The inability to persist custom metadata only affects two databases, and unfortunately the two that are heavily dependent on them.

4

See the Troubleshooting section of the DEVONthink Manual (and in Help). Perhaps do a Verify and Repair, and if necessary and a Rebuild.

However, I’d wait to do anything until others who certainly know more than me comment here.

5

Where are these databases located and what’s their file extension (.dtBase2 or .dtSparse)?

1 Like
6

/Users/me/Databases

File extension is .dtBase2.

As reminded by @rmschne, I did a Verify and Repair. An issue came up when I tried “Check File Integrity” of the affected databases, it returned a huge number of “File doesn’t have checksum yet”. Although it was okay when I last did the check for my last month’s backup export.

7

Did DEVONthink or the computer freeze/crash? Has been always enough disk space available? There are probably only two options:

  1. Rebuild the database (see File > Rebuild Database
)
  2. Restore a backup

In both cases backing up the current database first (e.g. by duplicating or zipping it in the Finder while DEVONthink isn’t running) is recommended.

8

I will give it a try and report back.

I should also add that last month I copied and transferred the databases to a new mac per this guide. Could it have affected the database? And no, neither the app nor computer crashed, there is sufficient disk space.

9

This shouldn’t cause any issues as long as there’s enough disk space, DEVONthink isn’t running during the migration and the permissions aren’t messed up afterwards.

Please select the database in the Finder, choose Show Package Contents in the contextual menu and then check the permissions of the .dtMeta and .dtMetaStore files inside the package. Do you have read/write permissions for all of them?

10

Yes I do have the necessary permissions.

I have sad news to report: my oldest backup created in end October has the missing metadata too. Which may suggest this issue started some time back.

However, rebuilding the my current database managed to return the function to add custom metadata. But I’m worried. I do Verify and Repair Database and Verify File Integrity checks before doing my monthly backup with no issues. I also rebuild my databases quarterly.

Considering these databases were active for about 4 years, I wonder if creating a new database and transferring my files might help. Or if I should try reinstalling DT? Any suggestions?

11

Don’t you run TimeMachine for semi-continuous backups? Or any other automatic service like Arq that regularly backs up everything at least once a day and keeps older backups around for some time?

Maybe you should.

1 Like
12

Did you migrate before or after creating this backup?

13

I have two existing backups and they were made before the migration.

14

This is admittedly a lazy mistake on my part. I have daily time-machine backups for my old mac but have not setup for my new mac. This is a brutal reminder to get it rectified.

15

A good rule: If your data is important to you, a backup should be treated like auto insurance. You shouldn’t drive before your car it’s insured and you shouldn’t start actively using your Mac before getting your backups set up and started.

2 Likes
16

To be clear the problem is not with the documents. But the sudden disappearance of the custom metadata.

My use of the custom metadata are predominantly for reference purposes, which is relatively trivial to recreate per document and per needs basis. It’s only a chore if I want to recreate them for all my documents at once — which is unnecessary.

My files, annotations and tags are still available, which are the most important parts. This is why after the initial panic I am not overly concerned with the lack of immediate backup. Also to be fair it wouldn’t help in this particular case since the disappearing act seems to begin months ago. The next step for me is to go through the time-machine backup drive for my old mac to see if I can pinpoint the moment the issue occurred.

Now that I am calm, the issue I want to narrow down is how and why this happened, and how can I prevent it from happening again.

17

Can I ask if the meta data fields are still there and it is their content that has disappeared, or the metadata fields themselves? If content, I would suggest checking smart rules/scripts (if you have any) that populate these fields. It could be an accidental rerun on a document clears out data already in a metadata field (I had this happen as me as my selection didn’t exclude documents with already populated fields and a second pass caused ‘blank’ to be written into/over what was previously there).

18

Okay, I did some further investigations and I want to apologise for some inaccurate information given earlier.

To sum up the issues:

  1. Database cannot add and retain new custom metadata values.
    Fixed by rebuilding database.
    Only affects databases that contain files with custom metadata values (added in old mac)

  2. Check File Integrity yields multiple errors
    Files that contained custom metadata values (added in old mac) yield the result “File doesn’t have a checksum yet”.
    Fixed by rebuilding database. But values are still missing.

  3. Custom metadata values added in the old mac fail to migrate over

  • The database backups created from the old mac has zero custom metadata values.
  • Since my current databases on the new mac are copies of the databases from the old mac, custom metadata values created in old mac’s installation failed to migrate over to new DT installation.

Other observations

  1. Databases that did not have any files with custom metadata values have no issues with adding new values to custom metadata fields.
  2. There are no issues with adding new custom metadata values on new installation

Given that the custom metadata are imported I think via the DT folder in Application Support, could there be some kind of a linking issue? Like the values still exist in some form in the older database, but this new DT installation doesn’t know how to read it or map it.

(I’m spitballing here, I’m not a programmer. I don’t know how any of these work.)

Additional notes

  1. Will it be helpful to remind users to run the various checks or even rebuild their databases when they migrate to a new computer?
  2. I have a database without any personal information which I don’t mind sending over for examinations if it helps. But I have to delete a number of files to make it more manageable.
19

When you migrated the databases, exactly how did you do this.? Via a wire, disk, cloud service? And what did you copy? A zip archive copy of the database, or the database “package” (e.g. file by file in package)?

And yes, IMHO, after creating a new database from another machine at minimum I would do a a verify and go from there (including repeating the copy).

20

Hi,

Maybe this is a “red herring” but I had problems with my Custom Metadata some time ago. A reinstall of DT fixed that. Custom Metadata missing

Best regards,
Björn

1 Like