Please help, I am really stuck with Metadata and JSON Export

I am really stuck with custom metadata table and exporting the custom metadata to a JSON document based on attributes defined with my imported database of images.

I have over 23,000 custom images that I need to generate JSON data files for. It is already cost me over 2 months trying to find a solution for this.

I absolutely love Devonthink, I have been using it for many years and I would hate to have to move out of it. I use it on several different levels, although I have never had to use it for something like this until recently.

I have attempted probably hundreds of different conversion and import/export options at this point. I am not a programmer and I am completely stumped on how to accomplish this task. I have been learning JS along with an entire catalog of other programming languages over the last 6 months. There has to be a simple solution for this, correct?

I would so much appreciate some guidance regarding this matter. And I think anyone in advance for taking the time to share with me their thoughts, advice or any potential solution to creating 23k+ individual JSON documents for each one of my image files.

Best Regards,
DM

And what data do you want these JSON files to contain? Sorry, but I’m not going to fight may way through a dark-mode image with cute bubbles to understand what you’re trying to do. It is less than helpful given that the screenshot is hardly legible.

What’s the point of learning an “entire catalog of programming languages” at the same time? JavaScript can do what you want, and it’s actually a no-brainer to produce JSON output with it: JSON.stringify() is your friend.

I doubt that this is a reasonable approach. But I may be wrong, of course. Can’t know unless you provide more detail on what exactly you’re trying to achieve.

Well my initial response was, was it really necessary to be so rude. I am a highly creative individual NOT a fully logic minded based programmer. Believe me I wish I was at this point because this has not been an exciting journey working on a solution for this.

And the reason that I had to dig into so many different programming languages was because I have never programmed in my life. So I had to read and learn which one was which just to understand what the hell was going on. This framework is for that framework that uses this plugin for that architecture. It was a nightmare. If programmers are so incredibly smart then why isn’t there one universal language that elegantly and easily takes care of everything?

My “dark-mode” image is actually quite clear. But let me brake it down a little better. I understand that it is quite difficult for backend coders to visualize the front end or simply take the time to look over bubble point that explains everything. I mean a coder that is familiar with DT should be able to look at an image such as mine and immediately understand 3 simple statements or should I have just writing it in code.

Regardless, I am moving over to FileMaker and building an app that will be able to accomplish my goal.

Thank you for input.

3 Likes

This script creates a JSON file for each selected record.

  • property theCMD_Identifiers : List of Custom Meta Data identifiers. Add whatever you want.
  • property theOutputFolder_Path : Path of folder where the JSON files are created.
  • property sortKeys : Whether the Keys should be sorted. If false Keys appear in the order they were added.
  • There’s a part “Add other key:value pairs here” part where you can add your other values.

Please don’t process all records in one go. You can e.g. set a label or a tag to keep track of what’s already processed. It might be a good idea to create subfolders after each script run because 23.000 files in a single folder might cause problems. Don’t forget to change property theOutputFolder_Path.

The JSON filenames are build from the record’s name without extension plus json. This means if you would create all JSON files into a single folder and not all your records got unique filenames then the script would overwrite previously created JSON files. To prevent this the record’s UUID is used when necessary.

-- Export Custom Meta Data to JSON files

use AppleScript version "2.4"
use framework "Foundation"
use scripting additions

property theCMD_Identifiers : {"primarycolor", "secondarycolor", "tertiarycolor"} -- Set the Custom Meta Data identifiers
property theOutputFolder_Path : (POSIX path of (path to desktop)) & "Script Output" -- Replace this with a path like "/Users/username/abc/xyz" (or create folder "Script Output" on your desktop)
property sortKeys : false

set theCMD_Titles to my getCustomMetaDataTitles(theCMD_Identifiers)

set theAttributes to {}
repeat with i from 1 to (count of theCMD_Titles)
	set thisCMD_Title to item i of theCMD_Titles
	set end of theAttributes to {trait_type:thisCMD_Title, value:""}
end repeat

set theDictionary to current application's NSMutableDictionary's new()
set theAttributesArray to current application's NSMutableArray's new()
set theOutputFolder_Path_String to (current application's NSString's stringWithString:theOutputFolder_Path)

tell application id "DNtp"
	try
		set theRecords to selected records
		if theRecords = {} then error "Please select some records."
		
		repeat with thisRecord in theRecords
			
			repeat with i from 1 to (count of theCMD_Identifiers)
				set thisCMD_Title to item i of theCMD_Titles
				set thisCMD_Identifier to item i of theCMD_Identifiers
				set thisCMD_Value to get custom meta data for thisCMD_Identifier from thisRecord
				if thisCMD_Value = missing value then set thisCMD_Value to ""
				set theAttributesArray to my setValueInAttributesArray(item i of theCMD_Titles, thisCMD_Value, theAttributesArray)
			end repeat
			
			-------------------------------------------------------------------------------------------------------------------------------------------------
			
			-- Add other key:value pairs here
			
			set thisKey to "name"
			set thisValue to filename of thisRecord
			my setValue(thisKey, thisValue, theDictionary)
			
			#set thisKey to "description"
			#set thisValue to ""
			#my setValue(thisKey, thisValue, theDictionary)
			
			#set thisKey to "image"
			#set thisValue to ""
			#my setValue(thisKey, thisValue, theDictionary)
			
			#set thisKey to "edition"
			#set thisValue to ""
			#my setValue(thisKey, thisValue, theDictionary)
			
			-------------------------------------------------------------------------------------------------------------------------------------------------	
			
			my setValue("attributes", theAttributesArray, theDictionary)
			
			-------------------------------------------------------------------------------------------------------------------------------------------------	
			
			set thisJSONFile_Filename to (name without extension of thisRecord & ".json")
			set thisRecord_UUID to uuid of thisRecord
			set success to my writeToFile(theDictionary, theOutputFolder_Path_String, thisJSONFile_Filename, thisRecord_UUID)
		end repeat
		
	on error error_message number error_number
		if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
		return
	end try
end tell

on getCustomMetaDataTitles(theCMD_Identifiers)
	try
		set theData to current application's NSFileManager's defaultManager()'s contentsAtPath:(POSIX path of (path to application support from user domain) & "DEVONthink 3/CustomMetaData.plist")
		set {theCustomMetaDataPlistArray, theError} to (current application's NSPropertyListSerialization's propertyListWithData:theData options:0 format:(current application's NSPropertyListXMLFormat_v1_0) |error|:(reference))
		if theError ≠ missing value then error (theError's localizedDescription() as string)
		set theCMD_Identifiers_withMD to "{\"md" & my tid(theCMD_Identifiers, "\",\"md") & "\"}"
		set theCustomMetaDataPlistArray_filtered to (theCustomMetaDataPlistArray's filteredArrayUsingPredicate:(current application's NSPredicate's predicateWithFormat:("self.identifier IN " & theCMD_Identifiers_withMD)))
		set theCMD_Titles to (theCustomMetaDataPlistArray_filtered's valueForKey:"title") as list
	on error error_message number error_number
		activate
		if the error_number is not -128 then display alert "Error: Handler \"getCustomMetaDataTitles\"" message error_message as warning
		error number -128
	end try
end getCustomMetaDataTitles

on setValueInAttributesArray(thisCMD_Title, thisCMD_Value, theAttributesArray)
	try
		set thisDictionary to (current application's NSMutableDictionary's dictionaryWithDictionary:{trait_type:thisCMD_Title, value:thisCMD_Value})
		theAttributesArray's addObject:thisDictionary
		return theAttributesArray
	on error error_message number error_number
		activate
		if the error_number is not -128 then display alert "Error: Handler \"setValueInAttributesArray\"" message error_message as warning
		error number -128
	end try
end setValueInAttributesArray

on setValue(theKey, theValue, theDictionary)
	try
		theDictionary's setValue:theValue forKey:theKey
		return theDictionary
	on error error_message number error_number
		activate
		if the error_number is not -128 then display alert "Error: Handler \"setValue\"" message error_message as warning
		error number -128
	end try
end setValue

on writeToFile(theDictionary, theOutputFolder_Path_String, theJSONFile_Filename, theUUID)
	try
		set theJSON_String to my formatJSON(theDictionary, true, sortKeys)
		set theJSON_Path to (theOutputFolder_Path_String's stringByAppendingPathComponent:theJSONFile_Filename)
		set existsPath to current application's NSFileManager's defaultManager's fileExistsAtPath:theJSON_Path isDirectory:(missing value)
		if existsPath then set theJSON_Path to (theOutputFolder_Path_String's stringByAppendingPathComponent:(theUUID & ".json"))
		set {success, theError} to (theJSON_String's writeToFile:theJSON_Path atomically:no encoding:(current application's NSUTF8StringEncoding) |error|:(reference))
		if success = false then error (theErrorMessage(theError's localizedDescription()) as string)
		return success
	on error error_message number error_number
		activate
		if the error_number is not -128 then display alert "Error: Handler \"writeToFile\"" message error_message as warning
		error number -128
	end try
end writeToFile

on formatJSON(theRecord, usePrettyPrint, useSortKeys)
	try
		if not (current application's NSJSONSerialization's isValidJSONObject:theRecord) then error "No valid JSON Object"
		set theJSONOptions to 0
		if usePrettyPrint then set theJSONOptions to theJSONOptions + ((current application's NSJSONWritingPrettyPrinted) as integer)
		if useSortKeys then set theJSONOptions to theJSONOptions + ((current application's NSJSONWritingSortedKeys) as integer)
		set {theJSONData, theError} to (current application's NSJSONSerialization's dataWithJSONObject:theRecord options:theJSONOptions |error|:(reference))
		if theError ≠ missing value then error (theError's localizedDescription() as string)
		set theJSONString to (current application's NSString's alloc()'s initWithData:theJSONData encoding:(current application's NSUTF8StringEncoding))
		return theJSONString
	on error error_message number error_number
		activate
		if the error_number is not -128 then display alert "Error: Handler \"formatJSON\"" message error_message as warning
		error number -128
	end try
end formatJSON

on tid(theInput, theDelimiter)
	
	set d to AppleScript's text item delimiters
	set AppleScript's text item delimiters to theDelimiter
	if class of theInput = text then
		set theOutput to text items of theInput
	else if class of theInput = list then
		set theOutput to theInput as text
	end if
	set AppleScript's text item delimiters to d
	return theOutput
end tid


PS the QuickLook Plugin you’re using doesn’t fit into the Dark Mode scheme. You might want to take a look at SourceCodeSyntaxHighlight:


PPS FileMaker is nice but can’t replace DEVONthink. Eagerly waiting for the free version for single users.

3 Likes

What about File > Export > Metadata (JSON)?

results in:

@DistinctiveMind needs something like this:

Now you baited me. See below for a JavaScript version. Though I’m not sure it does exactly the same as your script (lack of comments …). I also didn’t bother with pretty printing the JSON since the OP didn’t say anything about that, they only mentioned “JSON” in their requirements.

NOT TESTED! Use at your own risk, and do NOT run it on 23k files at once!

(() => {
const app = Application("DEVONthink 3");
app.includeStandardAdditions = true;
/* get all images from the selected records */
const records = app.selectedRecords().filter(r => r.type() === 'picture');
const metadataNames = ['primarycolor', 'secondarycolor', 'tertiarycolor' /* Add custom metadata names as needed */];
const targetFolder = `${app.pathTo('desktop')}Script Output`;
records.forEach(r => {
  /* Loop over all records */
  const targetObject = {};
  /* loop over all custom metadata names and assign the value to the corresponding
     property of the target object */
  metadataNames.forEach(name => {
    /* normalize the custom metadata name: lower case it and remove all spaces from it */
    const mdName = 'md' + name.toLowerCase().replaceAll(/\s+/g,'');
   /* Not sure if the following works in JXA. Use 
    app.getCustomMetadata({for: mdName, from: r})
    if it doesn't */
    targetObject[name] = r.customMetadata()[mdName];
 }) /* end of loop over custom meta data names */
 /* Add more object properties as needed, e.g.
 targetObject.filename = r.filename();
 targetObject.path = r.path(); 
 */
/* Create the JSON file and write the JSON string to it */
const targetFile = targetFolder + r.uuid() + '.json';
const file = app.openForAccess(Path(targetFile), {writePermission: true});
app.write(JSON.stringify(targetObject),{to: file});
app.closeAccess(file);
}) /* end of loop over records */

})()

Room for improvement: If one needs regular DT metadata in addition to custom metadata, I’d use a second array for their names and handle them similarly to the custom metadata. But given the lack of detailed information in the OP’s initial post, I have no idea what they may or may not really need.

1 Like

OMG !!

First I have to get ahold of myself … THANK YOU SO SO SO MUCH. I have been working on this project for almost two years and this is the issue that I have always had to come back to in order to complete the project.

I can not thank you enough! You have no idea how much this means to me!

But I also would like to explain more of what exactly all this is for. Because I not only want to express my gratitude, I would also like to possibly help others that are out in the big wide world that may be facing this level of frustration because their own personal projects.

You may or may not have heard of block chain development or tokenization, however I have been working on a project that utilizes blockchain development and tokenization. The majority of the hype and buzz that is around NFT for example has been greatly blown out of proportion yet still under valued as a whole. But despite all of that enterprise level organizations are spending an unbelievable amount of financial resources in developing their own path and doorway into web3 development.

When introduced to NFTs and blockchain development I got extremely excited because I saw a great opportunity for me to be creative and get involved in development. So I did and as I began creating unique OBJECTS that are and continue to be like any others that I have seen. I also began to learn that I needed to know how to code. Because all of the larger collections are programmatically developed using basic layers techniques that is essentially a photoshop based approach to cultivating massive amounts of images that essentially hold the same traits. And then defining them as such.

{
“name”: “imagename#0000”,
“image”: “ipfs://QmeFEabvd3ZpvG4o6LYaL5TccSG7LNHP8cAxioqa99SBbJ”,
“attributes”: [
{
“trait_type”: “Primary Color”,
“value”: “Green”
},
{
“trait_type”: “Secondary Color”,
“value”: “Burnt Orange”
},
{
“trait_type”: “Tertiary Color”,
“value”: “Burnt Orange”
},
{
“trait_type”: “Appearance”,
“value”: “Jagged”
},
{
“trait_type”: “Mood”,
“value”: “Calming”
},
{
“trait_type”: “Shape”,
“value”: “Symmetrical”
},
{
“trait_type”: “Image Density”,
“value”: “60%-70%”
}
]
}

But to continue. As I began creating completely unique images that are nothing like one another and at at the same time learning more about the technology, I realized that I was in a bit of a situation. One, the developers of these projects were using code to generate their images and two, I didn’t know how to code. Not 1 line.

So I did what I needed to do and ran into a lot of challenges along the way and now have a rather large task on my hands. Because I know that many of the attributes and be populated and collected from the hex data. Like “Primary Color” “Secondary Color” “Tertiary Color” and even the “Image Density” to some degree with even the potential to isolate when a transparent background is in place.

Thank you for the QuickLook Plugin fix for dark mode

I am not exactly sure what I am doing wrong, but I tested it and nothing happens. I used AppleScript to build it, VSCode, FileMaker, Codye and CodeKit. What am I doing wrong?

YES this would be perfect, unfortunately it does not exist
how did you get it changed over ?

I do not understand the javascript that you included. I have spent the entire day trying to make it work. I am a very sad :cry: and not very good :face_with_diagonal_mouth: coder.

I struggle very much to understand it. :face_with_monocle: :confounded:

I do not know what to do.

I love DT I don’t want to leave it. :cry: :disappointed:

Does the AppleScript I posted not work?

File > Export > Metadata (JSON) is a DEVONthink menu. But it doesn’t produce what I think you’re looking for. See the two JSON QuickLook screenshots I posted above.

I do not have that in my devonthink menu.

The two shots that you have above is exactly what I need. But again, I do not have an export to JSON selection in my menus.

I have no idea why I would not either.

  • Put the JavaScript code as it is (copy/paste three listing from here – that’s why it is in TEXT and not an image!) into ScriptEditor.
  • Set its language indicator in the upper left corner to JavaScript.
  • select a suitable record in DT
  • run the script in ScriptEditor

Same as you’d do with the script posted by @pete31.

Which version/edition of DEVONthink do you use? It should be available in the submenu Export of the File menu.

I checked for updates all the time 3.8.2 , and it says there are no updates available.