How to Clone a Database Safely

I’m aware that each DT database has internal IDs etc which makes copying a database a BAD THING to do. However, I do have the need to create similar databases from a “starter template” database that has various tags/groups/smart groups setup. So I’m looking for a way to safely make a clone of that template which is a new database that contains the same contents as the template but is a genuine new database with correct IDs etc etc. Simply copying all the items from the template isnt very practical as DT does Strange Things when doing that with tags. I’ve looked for some form of “Copy/Clone Database” command or tool but I cannot find one. Any ideas?

Here’s the best I can think of.

Open the database you want to clone and also create a new database.

Select all items except the inbox and the root tags folder. You can select the tags you created, just not the “tags” item itself. Likewise, you can select everything in the inbox - just not the inbox itself.

Right click on the selection and use the duplicate option, selecting the top level of the new database.

Messing with a database’s UUID is documented with a request to contact customer support. There are probably reasons it’s off limits for normal use.

I may be completely wrong here, but what is wrong with in Finder and use the “duplicate” command on the DEVONthink database package to create a duplicate?

That would duplicate the content, too. And if I understood the OP correctly, they’re looking for a scaffold only, i.e. groups and tags, not the whole content.

Duplicate, then delete content in the new database? Once empty, use that as the template for other duplication? Or something like that.

You should not duplicate a .dtBase2 file in the Finder.

I was hoping you’d have a view on that idea. For the record and future searches of this forum, why not? What goes wrong?

I looked in the DEVONthink Manual, and could find nothing with admonished the idea.

Because the UUID of a database is stored inside it. Duplicating the database package creates a copy with the same UUID and no you can’t simply excise that while maintaining anything inside the database itself.

2 Likes

Thanks for the sage advice. As I suspected, I was wrong again. I’m used to it. :wink:

Perhaps copy/paste this blog item into the DEVONthink Manual for the next release?

@BLUEFROG was faster … but the OP already mentioned the UUID issue: If you copy the DB package in finder, you’ll simply duplicate all UUIDs. Which will make them "DUID"s, i.e. Duplicate User IDs in DT.

That’s a sure way to drive DT insane.

1 Like

No worries! I will look for an appropriate place in the manual. I could have sworn I had it in there but couldn’t quickly find it.

:smiley:

It will issue warnings about a duplicate UUID and also suspend syncing the database(s) until things are back in order.

1 Like

Give me a few minutes, please…

Yea, but as i did not find documentation about that “fact” I was was not sure. I don’t believe everything on the interweb thingy.

Meantime, is there a way to clone that works?

I’m working on a script. It’s an interesting challenge.

2 Likes

No script needed but you do you, boo :wink: :stuck_out_tongue:

It’s still an interesting task …

1 Like

et al: Check out our Tuesday Tip this upcoming week. No scripts required so everyone can follow along :smiley:

3 Likes

Here’s a JavaScript script that seems to do what was asked. It does not copy Incoming itself (but should copy its subgroups!), nor trash and anything in it. In addition, it jumps over all smart groups defined in excludeSmartgroups – those are the ones that DT creates automagically with each database.

The code might not be self-explanatory, but I don’t want to over-comment it. Therefore, the comments outline only the basic steps. Please ask here if anything is not clear.

The cool thing (in my mind) is that it re-creates the smart groups. That is, the clone of a smart group searching in the old database will search in the cloned database, and similarly for old and cloned groups.

Beware that this code is not extensively tested – I checked that it creates groups and subgroups, simple tag hierarchies and smart groups.

And, of course: There might be a simpler solution just round the corner

(() => {
  const app = Application("DEVONthink 3")
  const dbName = "Test"; /* Set to the database to clone */
  const db = app.databases[dbName];
  
  const dbIncoming = db.incomingGroup;
  const dbIncomingId = dbIncoming.uuid();
  const dbTags = db.tagsGroup;
  const dbTagsId = dbTags.uuid();
  const dbTagsLoc = dbTags.location() + dbTags.name() + "/";
  const dbTrash = db.trashGroup;
  const dbTrashId = dbTrash.uuid();
  const dbTrashName = dbTrash.name();

  /* Define path and name of new database */
  const newDBFolder= '/tmp/'; /* Set to database folder, make sure of the trailing slash! */
  const newDBPath = `${newDBFolder}${db.name()}-NEW.dtBase2`;
 
  /* Exclude smart groups that are created automagically 
     with every new database. Make sure the name match your locale! */
  const excludeSmartgroups = ["Duplikate", "Alle PDF-Dokumente", "Alle Bilder"];
  
  /* get all real groups from the current database, i.e.
     - neither in Tags
	 - nor in Trash
	 - nor the Incoming group
  */
  const groups = db.parents().filter(p => p.type() === "group"
    && !p.location().match(dbTagsLoc) 
    && !p.location().match(dbTrashName)
    && p.uuid() !== dbTrashId
    && p.uuid() !== dbTagsId
    && p.uuid() !== dbIncomingId
  );

  /* get all smartGroups not in excludedSmartgroups */
  const smartGroups = db.smartGroups().filter(sg => !excludeSmartgroups.includes(sg.name()));

  /* build a hierarchy of all tags using the recursive function addToTagHierarchy */
  const tags = db.tagsGroup.children();
  const tagHierarchy = [];
  tags.forEach(t => addToTagHierarchy(t, dbTagsLoc, tagHierarchy));

  /* create the new database */
  const newDB = app.createDatabase(newDBPath);
  if (! newDB) {
    console.log("Error creating database");
    return;
  }

  /* set up a mapping from old to new UUIDs needed for smart groups */
  const uuidMap = {}; /* keys: original group UUIDs, values: new group UUIDs */
  /* First, add the old and new DB uuids */
  uuidMap[db.uuid()] = newDB.uuid();
  
  /* Loop over all groups and recreate them in the new DB. 
     Add the old => new UUID mapping
  */
  groups.forEach(g => {
    const location = app.createLocation(g.location(), {in: newDB});
    const newGroup = app.createRecordWith({type: "group", "name": g.name()}, {in: location});
    uuidMap[g.uuid()] = newGroup.uuid();
  });
  /* Loop over all smart groups and recreate them in the new DB,
     copying all properties but the searchGroup.
     Then set the searchGroup to the correct group or the database, 
     using the mapping build in uuidMap before
  */
  smartGroups.forEach(sg => {
    const location = app.createLocation(sg.location(), {in: newDB});
    const newGroup = app.createRecordWith({type: "smart group", "name": sg.name()}, {in: location});
    ["searchPredicates", "highlightOccurrences", "excludeSubgroups"].forEach(property => {
      newGroup[property] = sg[property]();
    })
    const oldGroup = sg.searchGroup.uuid();
    newGroup.searchGroup = app.getRecordWithUuid(oldGroup);
  })
  tagHierarchy.forEach(t => app.createLocation(t, {in: newDB}));
})()


function addToTagHierarchy(tag, level, hierarchy) {
  hierarchy.push(level + tag.name());
  tag.children().filter(t => t.type() === "group").forEach(t =>  {
    addToTagHierarchy(t, level + tag.name() + "/", hierarchy); 
  })
}

As it is out now: That’s a cute thing – always something new to discover in DT.

It requires a bit more work (and though) to clone an existing database then using the script, I think. OTOH, doing it manually gives you more flexibility.