Inline script in markdown files in DTTG

DEVONthink To Go
1

I currently have a lot of markdown files with < script > elements defined in them. Inside the < script > is a JSON object that sets up some data. And then below that I have an externally referenced .js file linked in that does some work / rendering of the data and presents a UI.

This worked great on both desktop and mobile DT until recently. Now it looks like DTTG no longer parses inline js and only externally referenced ones (<script src=”…).

I can probably figure out a workaround but it’s going to take me some time. I’d rather things like this stayed the same and didn’t change from release to release. Certainly not reductive changes. (It’s one of the big draws of DT for me… stability, consistency, long-arc feature development and support… and not being one of the many, many apps out there that change things every time I open the app.)

Thanks, Dom.

1 Like
2

Also… related… have you considered a local js API so we can hook into the filesystem?

Seems to me it’s only two custom functions away (some kind of js.readFile() and js.writeFile(…)) from unlocking some serious nerd-level self-contained / embedded apps.

3

Afaict, there is no filesystem on iOS you can hook into. And Apple will certainly not allow an app to provide a JS implementation fussing around with data that may belong to other apps.

As to your original question: this night be related to the new MD parser in DTTG.

4

Yikes!

I may be wrong, but last time I heard of something like this it sounded like a recipe for evil. Like, embedded code in pdfs, SQL, etc…

hm.

1 Like
5

What is “evil” about SQL? Yes, embedding code in documents can cause trouble, see VB macros in Office files. OTOH, the web wouldn’t be what it is now without JavaScript. Which is code embedded in HTML. But that can’t access local files (and for good reasons). As can’t, for example, the JS embedded in Drafts.

Apple does not allow general purpose scripting languages on its mobile platforms, AFAICT (and they didn’t even bother to port AppleScript to them). They refer to security reasons for that decision. Though they might have a point there, that would also speak agains Node.js, Python and the like on macOS. Not to mention Swift and other programming languages. So, perhaps are more important reason for them to ban programming on i*OS is Apple’s interest to keep others out of their platform.

1 Like
6

Not SQL, SQL injections :slight_smile:
But yes to your point about accessing the FS.

7

Yeah I don’t mean hooking into the iOS filesystem. Realise there are plenty of restrictions in that direction… it’s one reason I try to keep as much as possible inside DT… to avoid all the spaghetti hoops jumping from app to app!

I meant a javascript API for reading / writing files inside DT / DTTG (from within a markdown or HTML document). If there was a local (sandboxed is fine) custom API for doing that… one could write all kinds of mini-apps that functioned within DT / DTTG. It’d be like a very simple local web server. I think that’d pass Apple’s scrutiny… there are plenty of apps that already do that kind of thing (with more of a Developer focus… Python / PHP servers etc.).

Is there some info about the new MD parser you mentioned somewhere? (I’ll go look at release notes and see if I can find something…)

I spent hours yesterday trying to workaround this issue and failed. I started looking at using html format again. Which is what I originally used before hitting this issue… Html is treated as formatted note in DTTG 4 iOS - #10 by dominicwa It looks like that issue might have been fixed now though? In which case I’d rather use html to do this so I’m not trying to force markdown too far beyond it’s use cases.

8

Actually… if there were a local / sandboxed js API that let us read / write to DT from within a document… I’d be jumping straight in to see if I could leverage sqlite too!

Don’t think we need to worry about injection attacks here… this is all sandboxed within DT… it’s for personal data management only (well in my case anyway). It’d just be my document → my js code → my sqlite.db (and back again) = mini SPAs in DT! :partying_face:

9

I wouldn’t hold my breath waiting for that. First, what you reported is most probably a glitch in the MD parser that will be fixed, as the developers fix everything they know of. In my experience.

Second, JavaScript is not (yet?) overly popular with the developers.

Third, what’s the point having an API that works only inside an MD document?

Fourth, they’d have to do the same for DT to avoid the two platforms drifting further apart.

Fifth, there already is a JavaScript API in DT. Not in DTTG, because Apple does not provide its OSA on iOS.

1 Like
10

1 Not expecting it… I’m sure they’re busy with an enormous backlog and this likely far outside the scope of many users / use cases.

2 I hate Javascript too so we’re aligned there :wink: Unfortunately, it’s what we have for client side web, and it’s more portable… so I work with it where / when I have to. In this case… we’re talking about markdown (or html) files… the whole point is they’re portable / future proof. That rules out Applescript / Shortcuts etc for me in some - many instances.

3 It wouldn’t be just for md files. In fact, it’s actually more relevant to html files. Md files just happen to translate a lot of html too. (And I only used md files in this particular instance because DTTG kept converting html files to formatted notes.) The point though is… SPAs. You could pseudo-host a web application within DT. That’s huge! It opens up all kinds of possibilities. But the single block right now (and I know it because I tried all sorts of hacks to workaround) is… from within javascript within a html document you cannot retrieve or save the contents of another document. If you could… you could use a html document UI (built yourself) as an interface to your documents… saving page / data states down to another document (either a json file, csv, or even sqlite db). Next time you open the html page… it reads from the other document and so on. Basically… a web app. Ta daa!

4 Yeah I’m sure it’s already challenging keeping things working coherently across both platforms. I’ve developed natively for iOS and Mac… I know there are significant differences. This might add another fork in the ongoing maintenance… but I’d expect it to be a thin layer providing the hook between the document file system and the browser engine… vs a potential fat reward for a library of SPAs they could then include alongside all the scripts and plugins and so on.

5 The js API in DT… are you talking about the Applescript alternative? If so, is there any way that code can be executed from within a html (or md) document via embedded javascript? I suspect not… in which case it’s not the same thing. As for whether DTTG could support this… unless Apple has changed / restricted the iOS SDK further, I remember seeing / tinkering with the capability of doing exactly what I’m talking about (event / messaging between safari containers and the native environment). There are a whole heap of other scenarios where that is needed. This one is not really that special?

There’s an app to save tiddlywiki states on iOS that does what I’m talking about… Quine. And of course… there is all those js plugins in that “other” app that shallt not be named Minecraft netherworld block ahem… they all interact just fine with the local app’s documents. Seems to me DT is only a thin layer away from the same prize of rich web-based plugin ecosystem?

11

A single page application (if that’s what SPA stands for) in a DT document – while I can see that you might find that useful, I doubt that there’s much enthusiasm in the DT user community for it. Not least because JavaScript is not very popular here.

In principle, there’s no problem having a script element in an MD document whose content is executed when the MD is shown in preview mode – what you see is HTML and WebKit will run the script code. This currently does not work in DTTG because there’s a bug in the new MD parser. I have good reason to believe that the developers will fix that bug. So, in an MD document you can put whatever code you want to be run when the MD is rendered as HTML.

And of course, this is run in a sandbox. In a user agent, aka browser. So no, no escaping it and doing something to other documents. Same as in the browser – you can’t manipulate another document from your current one (unless something is broken in the browser).

DT (not DTTG) though offers the possibility to work with any of its documents (and those of other, scriptable apps) through OSA, using AppleScript or JavaScript. But that is a completely different thing from what happens when you run a script in a user agent. A lot less secure, too.

Code running in an HTML document is interpreted by the user agent. And limited to whatever functionality this agent provides. Writing anything to the local filesystem is not part of that functionality, AFAIK. If you want that, someone would have to write a new user agent with its own JS engine that does what you want. Opening a huge security hole – someone sends you an MD file with a script element uploading your private documents to example.com.

I for one would be very much against that kind of “feature”.

They use Apple’s framework for the HTML rendering. So they are limited to what these frameworks offer. I detailed the limitations above. They’d have to rewrite everything.

Yes, there are apps providing JS APIs, notably Drafts (I haven’t looked at other ones). But Drafts is not about HTML, it just runs JS code using its own JS APIs. And it doesn’t use “documents” in the traditional sense, everything is stored in an SQLite database.

1 Like
12

Yeah… I think I managed to hack some elaborate experiment together by going from a DT html document referencing a data.json (just another DT document) → Share Ext → Shortcut (now outside the sandbox) → write / update the data.json → reload DT html document and voila… but that roundabout mechanism is painful, temperamental and absurdly fragile :slightly_smiling_face: But it kinda proved the theory / usefulness for my scenarios. I just can’t use that with the fragility and dependency on Shortcuts.

Yep, realise it’s the same restrictions put on it as with web browsers… W3 introduced local data storage in the html5 spec to try provide some alternative and I thought about using that… but it wouldn’t sync between user agents - DT & DTTG - and there are size limits, and the UA may empty the storage cache at any time if it needs to.

Indeed, no typical browser / user agent is going to allow read / write to the filesystem. No web browser does it… it’s all locked down. Otherwise, we’d all be visiting websites and they could happily read / write files from our computers (which, of course they actually do to some extent with cookies… but that’s a much more restricted mechanism).

I’d argue that this DT / DTTG isn’t a typical browser / user agent though. It’s a sandboxed filesystem. You would limit any use of the js read / write hooks to internal doc ←→ internal doc only. The js hooks would / should do nothing if loaded in from any remote URL.

I think you can turn off the local file access restrictions when viewing locally-stored web pages in some (most?) web browsers if you go hunting in the configs. (Probably relates to CORS which always confuses me a little each time I have to consider it in projects.)

I’m not downplaying or ignoring there are security considerations. But they have been managed elsewhere to enormous gain. It’s like any tech architect consideration (my line of work)… weighing up the pros and cons. I’m sure the risks could be managed / mitigated if the other drivers were strong enough. (However, as interesting as this conversation is… I recognise they probably aren’t ha!)

Don’t think so… I’ve done this myself (developed native iOS apps with webkit and hooked into the DOM and listened / triggered js events)…

Obviously I’m not savvy to the DT and DTTG codebases so can’t speak with authority on the complexity involved. But again, I’m confident webkit SDK on both platforms supports this out of the box (and likely handles the cross-origin security issues we’re talking about too… underneath the developer APIs I doubt the SDK’s private APIs would allow any DOM js event capturing / triggering if the source of the DOM was remote).

13

You’re still in the WebKit sandbox. One can modify the DOM, yes. And CORS can get in the way. But you can’t read/write files on the local disk. Not with JS in the browser, not with WebKitJS afaict.

14

WebKit is a container / component within the DT and DTTG apps. The devs can “talk to” anything inside webkit from external swift / obj-c. The devs can also manipulate documents stored within the app’s sandboxed filesystem from external swift / obj-c code.

So from the app they observe the web component for custom triggered js events e.g. “writeToDocument('DEVONthink-document-id’, ‘here is my text’);”. They catch those events in the external swift / obj-c code (can be done, have done it myself!). Then they pass the text to the appropriate document.

For reading documents and passing to the html file, the same, observe for a “fetchDocument('DEVONthink-document-id’);” event using the webkitjs sdk… feed it through.

If this was happening on a remote URL I’d expect the webkit sdk to refuse to allow due to security. But given it’s local-local and we’re just bridging from container to the broader application architecture… I’m confident-to-certain it’s possible (as I said, I actually did this a few years ago).

I never submitted the app I did this in (it was a research project for lab-use only) but… this is what Obsidian and other apps already do. They have tons of javascript plugins which have read/write document capability. And those documents are all stored within the secure / sandboxed filesystem for the app.

15

[ [sandboxed filesystem] ← send / fetch docs → (DEVONthink code) ← listen / respond custom js events → [webkit sdk] ]

It’s a really thin layer of code required actually. And just TWO custom js event handlers away from unlocking tons of capability! :slight_smile: