Using Azure/Google API in AppleScript to translate documents

Hi there,

I’m hoping to use a script to call the Microsoft Azure Translate API to translate text within documents. My priority at the moment is to translate news feeds but would like to be able to translate any plain text, markdown, html, etc documents if possible.

Ideally the script would translate the filename (i.e the headline for RSS news feeds) and insert an English translation of the original text above the original text with a line dividing the two so that I can easily verify the translation for salient documents (given continuing quirks with online translation services, particularly for more obscure languages).

Thank you in advance for any suggestions / support!

Please provide details on the API, a sample call would be best.

  • Why this service specifically?

  • Define documents more precisely.

  • Do you have concerns about privacy?

  • What format are you using either on an individual feed or set in Preferences > RSS > Feed Format?

  • Do you have expectations of hyperlinks persisting in the translated text?

I hope you are seeing how something that’s quickly described often lacks sufficient information to implement quickly or even solve. :slight_smile:

Here are basic parameters for text translation:

Example below:

set subscriptionKey to “AZURE API KEY”

set endpoint tohttps://api.cognitive.microsofttranslator.com/translate?api-version=3.0&to=en

set textToTranslate to “Este es un pez!”

set requestBody to “[{"Text": "” & textToTranslate & “"}]”

set curlCommand to “curl -X POST "” & endpoint & "" -H "Ocp-Apim-Subscription-Key: " & subscriptionKey & “" -H "Content-Type: application/json" -d '” & requestBody & “'”

set jsonResponse to do shell script curlCommand

set translatedText to do shell script “echo " & quoted form of jsonResponse & " | grep -o ‘"text":"[^"]*"’ | head -n 1 | sed ‘s/"text":"//;s/"//g’”

display dialog "Translated Text: " & translatedText

  • Why this service specifically?

Only Google and Azure translate more obscure languages (e.g. Amharic). Azure more cost effective than Google

  • Define documents more precisely.

Ideally markdown and/or HTML but I’d be happy with plain text.

  • Do you have concerns about privacy?

For my initial needs, no. I want to translate open source RSS feeds articles.

  • What format are you using either on an individual feed or set in Preferences > RSS > Feed Format?

I am currently using “automatic” and “clutter-free” which produces a HTML Text file for my test feed. I’m then converting that to markdown with smart rules and putting it in a group to keep images (due to RSS conversion issues mentioned here). I know “clutter-free” is clear same clutter regardless of format but for my test feed (Caasimada Online) other formats include various social media links.

  • Do you have expectations of hyperlinks persisting in the translated text?

I can certainly live without them!

  • I hope you are seeing how something that’s quickly described often lacks sufficient information to implement quickly or even solve.

I am, and am also as impressed as ever with the responses on this forum!

1 Like

Is this code working? If it is, what is the problem? If not, why – error messages?

In any case, I’d suggest dropping AppleScript for this project. You want to process text and JSON. That’s not something AS is good with. Rather use JavaScript.

And be careful with Microsoft’s documentation. At least in one case their JSON code is not correct.

After some (very basic, BTW) research, I found an example Gist illustrating how to talk to an Azure service using JXA and the Objective-C bridge:

I’d suggest using this as a starting point because

  • it illustrates how to use your API key without putting it into the script (security!)
  • it hardly uses any shell code, which makes it a lot more stable and easier to maintain then all these doShellScript thingies which require very careful quoting.

Thank you again! I will le you know how it goes.

So I used the Python script in the gist you mention as a framework which now does exactly what I want it to do outside of DEVONthink.

import os
import json
from urllib.request import Request, urlopen

# Set Azure Translator API details
AZURE_TRANSLATOR_ENDPOINT = "https://api.cognitive.microsofttranslator.com"
TRANSLATOR_API_KEY = "getyourtranslatorapikey"
TRANSLATOR_API_VERSION = "3.0"
TRANSLATOR_REGION = "your_region"  # Replace with your Azure region

translator_url = f"{AZURE_TRANSLATOR_ENDPOINT}/translate?api-version={TRANSLATOR_API_VERSION}"
translator_headers = {
    "Ocp-Apim-Subscription-Key": TRANSLATOR_API_KEY,
    "Content-Type": "application/json",
    "Ocp-Apim-Subscription-Region": TRANSLATOR_REGION
}

def translate_text(text, from_language="so", to_language="en"):
    data = [{"Text": text}]
    request_url = f"{translator_url}&from={from_language}&to={to_language}"
    req = Request(request_url, data=json.dumps(data).encode(), method="POST", headers=translator_headers)
    with urlopen(req) as response:
        response_content = json.loads(response.read().decode())
        translated_text = response_content[0]['translations'][0]['text']
    return translated_text

def process_markdown_file(input_path, output_dir, target_language="en"):
    # Read the markdown file
    with open(input_path, 'r') as file:
        content = file.read()
    
    # Translate the content
    translated_content = translate_text(content, to_language=target_language)
    
    # Extract and translate the file name
    file_name = os.path.basename(input_path)
    file_name_without_ext = os.path.splitext(file_name)[0]
    file_ext = os.path.splitext(file_name)[1]
    translated_file_name = translate_text(file_name_without_ext, to_language=target_language)
    output_file_name = f"{translated_file_name}{file_ext}"
    output_path = os.path.join(output_dir, output_file_name)
    
    # Write the translated content and the original content to a new markdown file
    with open(output_path, 'w') as file:
        file.write(translated_content)
        file.write("\n\n---\n\n")  # Horizontal line in Markdown
        file.write(content)

# Example usage
input_markdown_path = 'path_to_input_markdown_file.md'
output_directory = 'path_to_output_directory'
process_markdown_file(input_markdown_path, output_directory, target_language="en")

Is there a straightforward way to run this in a Smart Rule applying the script to files matching the Smart Rule criteria? Presumably I would need to adapt the input and output paths in the scripts?

Please note that I am NOT a coder. I’m a researcher trying to strengthen my ability to research.

1 Like

If you’d used the JavaScript code (to which I’d linked), it would be straightforward. With this Python thing, you’d have to use an AppleScript or JavaScript wrapper that runs a shell that runs Python to run the script. Feasible. But awkward.

Also please respond to the actual post, not to yourself. That keeps the thread easy to follow.

Update I modified the original JS code so that it might be usable in a smart rule. Note that I couldn’t really test it since I don’t have access to the Azure API. In principle, it should work, but it might need some adjustments.

ObjC.import('Foundation');
ObjC.import('Cocoa');

const targetDirectory = "/users/YOU/Documents/Whatever/";

// AZURE translation API details

const AZURE_TRANSLATOR_ENDPOINT = "https://api.cognitive.microsofttranslator.com"
const TRANSLATOR_API_VERSION = "3.0"
const TRANSLATOR_REGION = "your_region"  // Replace with your Azure region

const translator_url = `${AZURE_TRANSLATOR_ENDPOINT}/translate?api-version=${TRANSLATOR_API_VERSION}`;


function performsmartrule(records)  {
  /* records is an array of the records targeted by the smart rule */
  const app = Application.currentApplication();
  app.includeStandardAdditions = true;
  records.filter(r => r.type() === "markdown").forEach(r => {
    const result = getTranslatedText(app, r.plainText());
    const targetFile = `${targetDirectory}${r.name()}`; // Use the original filename to store the translation in targetDirectory
    
    // Write the result to the file
    const file = app.openForAccess(Path(targetFile), {writePermission: true});
    app.write(text, { to: file }); 
    app.closeAccess(f);
  })
}

function getHTML(app, input) {

    // This script is a simple example of how to use the Azure OpenAI API to generate text
    // from a macOS system service written as a JavaScript for Automation (JXA) script
    // You can drop this into a Shortcuts "Run JavaScript" action (which will only work on a Mac)
    // or use it as a starting point for a more complex system service

// input is the string to translate

    const DEPLOYMENT_NAME = "my_deployment", // Set this so that you can find the API key with "security"
        // You should store your keys in the macOS Keychain and retrieve them from there:
    OPENAI_API_KEY = app.doShellScript(`security find-generic-password -w -s ${AZURE_TRANSLATOR_ENDPOINT} -a ${DEPLOYMENT_NAME}`);
    const translator_headers = {
    "Ocp-Apim-Subscription-Key": OPENAI_API_KEY,
    "Content-Type": "application/json; charset=UTF-8",
    "Ocp-Apim-Subscription-Region": TRANSLATOR_REGION
}
        postData = {
            "temperature": 0.4,
            "messages": [{
                "role": "system",
                "content": "Act as a writer. Expand the text by adding more details while keeping the same meaning. Output only the text and nothing else, do not chat, no preamble, get to the point."
                ,
            }, {
                "role": "user",
                "content": input
            }]/*,{
            role: "assistant",
            Use this if you need JSON formatting
            content: ""
        */
        },
        request = $.NSMutableURLRequest.requestWithURL($.NSURL.URLWithString(translator_url));

    request.setHTTPMethod("POST");
    request.setHTTPBody($.NSString.alloc.initWithUTF8String(JSON.stringify(postData)).dataUsingEncoding($.NSUTF8StringEncoding));
    
    Object.keys(translator_headers).forEach(k => {
      request.setValueForHTTPHeaderField(translator_headers[k], k);
    })
  
    const error = $(),
        response = $(),
        data = $.NSURLConnection.sendSynchronousRequestReturningResponseError(request, response, error);

    if (error[0]) {
        return "Error: " + error[0].localizedDescription;
    } else {
      const json = JSON.parse($.NSString.alloc.initWithDataEncoding(data, $.NSUTF8StringEncoding).js);
        if (json.error) {
            return json.error.message;
        } else {
            return json.choices[0].message.content;
        }
    }
}

Again, thank you!

I tried your code above but couldn’t get it to function in a SmartRule. Kept getting errors.

I have managed to get the following JavaScript to do exactly what I want it to do using Node.js:

const axios = require('axios');
const fs = require('fs');
const path = require('path');
const { execSync } = require('child_process');

function getKeychainValue(service) {
    try {
        const result = execSync(`security find-generic-password -s ${service} -w`).toString().trim();
        return result;
    } catch (error) {
        console.error(`Error retrieving keychain value for service: ${service}`);
        return null;
    }
}

const AZURE_TRANSLATOR_ENDPOINT = "https://api.cognitive.microsofttranslator.com";
const TRANSLATOR_API_VERSION = "3.0";
const TRANSLATOR_REGION = getKeychainValue("AzureTranslatorRegion");  // Retrieve from Keychain
const TRANSLATOR_SUBSCRIPTION_KEY = getKeychainValue("AzureTranslatorKey");  // Retrieve from Keychain

async function translateText(text, fromLang, toLang) {
    const headers = {
        "Ocp-Apim-Subscription-Key": TRANSLATOR_SUBSCRIPTION_KEY,
        "Ocp-Apim-Subscription-Region": TRANSLATOR_REGION,
        "Content-Type": "application/json"
    };

    const postData = [{
        "Text": text
    }];

    const params = {
        "api-version": TRANSLATOR_API_VERSION,
        "from": fromLang,
        "to": toLang
    };

    try {
        const response = await axios.post(`${AZURE_TRANSLATOR_ENDPOINT}/translate`, postData, { headers: headers, params: params });
        const translations = response.data[0].translations;
        return translations[0].text;
    } catch (error) {
        console.error("Error translating text:", error.response ? error.response.data : error.message);
        return null;
    }
}

function sanitizeFilename(filename) {
    return filename.replace(/[^a-zA-Z0-9 \-]/g, ' ').trim();
}

function translateMarkdownFiles(directory) {
    fs.readdir(directory, (err, files) => {
        if (err) {
            console.error("Error reading directory:", err);
            return;
        }

        files.forEach(async (file) => {
            const filePath = path.join(directory, file);
            if (path.extname(file) === '.md') {
                const content = fs.readFileSync(filePath, 'utf-8');

                // Translate file content
                const translatedContent = await translateText(content, "so", "en");
                if (translatedContent) {
                    const newContent = `${translatedContent}\n\n---\n\n${content}`;
                    fs.writeFileSync(filePath, newContent, 'utf-8');
                    console.log(`Translated content of ${file} successfully.`);
                } else {
                    console.log(`Failed to translate content of ${file}.`);
                }

                // Translate file name
                const originalFilenameWithoutExtension = path.basename(file, '.md');
                const translatedFilename = await translateText(originalFilenameWithoutExtension, "so", "en");
                if (translatedFilename) {
                    const sanitizedFilename = sanitizeFilename(translatedFilename) + '.md';
                    const newFilePath = path.join(directory, sanitizedFilename);
                    fs.renameSync(filePath, newFilePath);
                    console.log(`Translated and renamed file from ${file} to ${sanitizedFilename} successfully.`);
                } else {
                    console.log(`Failed to translate filename of ${file}.`);
                }
            }
        });
    });
}

// Replace with the directory containing your Markdown files
const targetDirectory = "/Users/user/Test/md_files";

translateMarkdownFiles(targetDirectory);

Would you be able to help me adjust this for a DT3 Smart Rule?

Very sketchily:

  • modify your Node script so that it expects a single file name on the command line
  • and returns the translation as a string on stdout
  • create a smart rule that
    • acts on the MD files you want to translate
    • runs a script (AppleScript or JavaScript, though I’d prefer the latter) on these files
    • this script builds a shell command like node <yourscript.js> <filename.md>
    • it runs this command with doShellScript, assigning the result to a string variable
    • it then adds the contents of this variable to the current MD file

You get the <filename.md> from the path property of the current record. To change its content, modify its plainText property.

Though your approach of translating all files in a directory works, it is, IMO to complicated for DT: You’d have to export all MD files into a directory, run your script on that, and then re-import all the modified files into DT. Afterward, you must delete all the old files. Too convoluted for my taste, and too error-prone.

I’ve modified the Node script — translate.js — as such:

const fs = require('fs');
const https = require('https');
const keytar = require('keytar');
const path = require('path');

const TRANSLATOR_REGION_SERVICE = "AzureTranslatorRegion";
const TRANSLATOR_SUBSCRIPTION_KEY_SERVICE = "AzureTranslatorKey";

const baseURL = "https://api.cognitive.microsofttranslator.com";
const apiVersion = "3.0";

async function getCredentials() {
    const region = await keytar.getPassword(TRANSLATOR_REGION_SERVICE, "default");
    const subscriptionKey = await keytar.getPassword(TRANSLATOR_SUBSCRIPTION_KEY_SERVICE, "default");

    if (!region || !subscriptionKey) {
        throw new Error("Could not retrieve necessary keys from keychain");
    }

    return { region, subscriptionKey };
}

function translateText(text, fromLang, toLang, region, subscriptionKey, callback) {
    const url = `${baseURL}/translate?api-version=${apiVersion}&from=${fromLang}&to=${toLang}`;
    const postData = JSON.stringify([{ "Text": text }]);

    const options = {
        hostname: 'api.cognitive.microsofttranslator.com',
        path: `/translate?api-version=${apiVersion}&from=${fromLang}&to=${toLang}`,
        method: 'POST',
        headers: {
            'Ocp-Apim-Subscription-Key': subscriptionKey,
            'Ocp-Apim-Subscription-Region': region,
            'Content-Type': 'application/json',
            'Content-Length': Buffer.byteLength(postData)
        }
    };

    const req = https.request(options, (res) => {
        let data = '';

        res.on('data', (chunk) => {
            data += chunk;
        });

        res.on('end', () => {
            const responseJSON = JSON.parse(data);
            const translations = responseJSON[0].translations;
            callback(translations[0].text);
        });
    });

    req.on('error', (e) => {
        console.error(`Problem with request: ${e.message}`);
        callback(null);
    });

    req.write(postData);
    req.end();
}

function removeExtension(filename) {
    return filename.substring(0, filename.lastIndexOf('.')) || filename;
}

async function main() {
    const args = process.argv;

    if (args.length < 3) {
        console.log("Usage: node translate.js <file_name>");
        return;
    }

    // Join all arguments that are part of the file name
    const fileName = args.slice(2).join(" ");

    if (!fs.existsSync(fileName)) {
        console.log("File does not exist: " + fileName);
        return;
    }

    try {
        const { region, subscriptionKey } = await getCredentials();

        fs.readFile(fileName, 'utf8', (err, data) => {
            if (err) {
                console.log("Failed to read file: " + fileName);
                return;
            }

            translateText(data, "so", "en", region, subscriptionKey, function(translatedContent) {
                if (translatedContent) {
                    const combinedContent = `${translatedContent}\n\n---\n\n${data}`;
                    // Write the combined content back to the file
                    fs.writeFile(fileName, combinedContent, 'utf8', (writeErr) => {
                        if (writeErr) {
                            console.log("Failed to write translated content to file: " + fileName);
                            return;
                        }
                        console.log("File content translated and updated.");

                        // Translate the file name
                        const originalFileName = path.basename(fileName);
                        const newFileNameBase = removeExtension(originalFileName);
                        translateText(newFileNameBase, "so", "en", region, subscriptionKey, function(translatedFileName) {
                            if (translatedFileName) {
                                const newFileName = path.join(path.dirname(fileName), translatedFileName + path.extname(fileName));
                                fs.rename(fileName, newFileName, (renameErr) => {
                                    if (renameErr) {
                                        console.log("Failed to rename file: " + fileName);
                                        return;
                                    }
                                    console.log("File renamed to: " + newFileName);
                                });
                            } else {
                                console.log("Failed to translate file name.");
                            }
                        });
                    });
                } else {
                    console.log("Translation of file content failed.");
                }
            });
        });
    } catch (error) {
        console.error("Error retrieving credentials:", error.message);
    }
}

main();

I’ve checked this works as intended.

I’m still struggling to execute the script in a Smart Rule.

I’ve tried:

(() => {
    const app = Application('DEVONthink 3');
    const selectedRecords = app.selectedRecords();

    selectedRecords.forEach(record => {
        const path = record.path(); // Get the path of the selected file
        if (path) {
            const command = `path/to/node /path/to/translate.js "${path}"`;
            const doShellScript = Application('System Events').doShellScript;
            try {
                const result = doShellScript(command);
                console.log('Result:', result);
            } catch (error) {
                console.error('Error:', error);
            }
        }
    });
})();

But no joy. Can you help me get over this final hurdle?

Having a look at the default code that is shown when you add an execute script action might have helped. Or reading the documentation :wink:

Anyway, the basic structure is

function performsmartrule(records) {
  const app = Application('DEVONthink 3');
  records.forEach(r => {
   ...
})

I think checking for the existence of path is not needed if your smart rule selects only MD records – those must have a path.

And console.log will not print anything if you run the code from a smart rule. Use app.logMessage() instead and look at DT’s protocol window.

More: Remove all this “newFileNameBase”, “translatedFileName” etc. stuff. Your code should only take the input file and write the translation to stdout. No fiddling around with the file names or anything – those are internal to DT, and you must not modify those internal structures!

So, as I already said: Write translatedText to stdout. I don’t know enough about node, but perhaps console.log does that. In your smart rule script, use result to add to the markdown record’s plainText like so:

r.plainText = r.plainText() + `\n\n${result}`;

or however you need to combine the text and the result.
If you do not want to modify the original record, you should use createRecordWith… to create a new one, setting its plainText in that method call.