See here, for example. Specifically, custom metadata is defined by /Users/yourusername/Library/Application Support/DEVONthink 3/CustomMetaData.plist; if you copy that file from the previous device to the new device (with DT closed, and then open DT), you should be set. That actual metadata itself is stored within the database and will be visible once the file is in place.