Suggestions (from a nutty novelist)

ndouglas · October 15, 2006, 7:47pm

Back History: I’m writing a series of ten big books. Lots of characters, storylines, text, et cetera. Been working for ten years. Sanity–;

MediaWiki: What I’ve been using for a couple years now. Or, really, MediaWiki + EasyTimeline + GraphViz + dozens of other extensions. Works well, capable of almost anything, completely bewildering to track since it’s not very hierarchical, and in general a real pain in the ass.

DEVONthink: I love it. Been using it for a week and enthusiastically switched over my MediaWiki the other day. Handled the import of 1047 documents like a pro, and I don’t mean prokaryote.

What I Want: I don’t think I’m the first to request any of these features, or at least features like these. I just want to add my voice to the list. Here goes:

GraphViz support. GraphViz is a fantastic open source tool for plotlines, storylines, genealogies, all sorts of things. Current integration (with the command line tool as well as the OS X version is basically non-existent. Import a vector graphic, with the WikiLink functionality in a text layer on top of it? If this was available only as a plugin, I’d still pay for it, and I think a lot of other writers would too.
Timeline support. I’m not crazy about EasyTimeline, and I don’t think many people are. But some kind of timeline mechanism would be fantastic.
Template support for sheet/record display. I’m sure you have something like this in the works, but I’d love it. Even if the sheets were just HTML with a little markup sub-language to show DT where to display the values. Anything. AN-Y-THING.
The Smart Groups are clever and heroically extensible, for those with a little Applescript chops. My problem is that I’d like a way to specify variables for the scripts. Right now, I use the contents of comment fields in Groups as variables used in order to find documents containing those same words in their comments. Kinda kludgy. Or, alternately, a “real” smart folder-esque arrangement?
I’d like to be able to add fields (metadata?) and values to a specific group or file more-or-less on the fly. I don’t know how possible this is, or how it would affect database size, or anything like that. I’d like to be able to say “Hmm, this was written by Mark Twain” and add an author field and type in the value. Or add a copyright date field.
Import a URL (whether from local drives or the intertubes) as a PDF, with an automator action to match. I have a tremendous workflow to accomplish this very act, and I’m not sure why.
More labels? I’m not sure if I’m missing something, or if I’m the only person who would ever use these, but I’d love to have about twenty labels. If they’re dependent on the numbers and types of labels in OS X, I guess I’m just flat-out screwed.

That’s pretty much all I have to say right now. I have other ideas, but they’re a bit more involved…

Any thoughts?

Bill_DeVille · October 15, 2006, 8:13pm

Welcome to DT Pro.

The developers have incorporated many features suggested by users. But there’s a danger of bloatware if the application were to try to be everything to everyone.

Some of the features you have noted – or other features that would allow you to do something similar – will be appearing in future versions.

DT Pro does use standard OS X features, including Labels. You noted that you would like to have more label colors available. Suggestion: Think about the logical extension of label “keys” if you were to define properties defined by a label color PLUS a State (Any, On, Off). With 9 Label search criteria (none + 8 colors) and 3 State search criteria you can define a pretty large list of logical conditions for your data.

ndouglas · October 15, 2006, 8:36pm

Thanks. Happy to be here

Naturally – this is a danger with the timeline and mind-mapping/flow chart functionalities. I think it can be beaten, though. I’ll go a little more into detail (forgive me if you’re accquainted with this already):

Both EasyTimeline and GraphViz are CLI apps. How MediaWiki deals with them is that they have third-party extensions that send everything between and tags to the apps responsible for rendering that output, which then return an image. I believe both md5sum the instructions passed, so that if they have already been followed, the cached image matching that sum will be passed along instead. The image comes back as a PNG, with a .map if the image is rendered as an imagemap.

So I’m about the commit the cardinal sin of non-programmers and say “sounds easy to me.” I’m not naive enough to think it’d be effortless (in fact, if I were a bit smarter, I might be able to do it myself with no labor at all on DEVON’s part), but it seems like a good, extensible featureset. The basic feature could be used with any number of CLI apps…

As far as most of the others are concerned (import data_x as PDF), I can deal. If they’re websites, I just save them normally and then run them through an Automator workflow. And none of the ideas I have were dealbreakers – I love DT just the way it is

I was afraid of that.

A good thought, but I’m already using my state indicators. I’ll review what I’m doing and see if I can shuffle it around.

Thanks!

edf · October 16, 2006, 8:50pm

Some sort of plugin architecture for handling embedded data or unknown file types would be quite nice, I agree. There are many useful command-line tools out there that return text, html, pdf, or image data based their input.

I’ve done quite a bit of DT scripting myself in this area, the most advanced one being a source code syntax highlighter that uses the ‘smart record’ feature to replace its contents with html output of a syntax highlight program run on the on-disk source code files.

Something similar would work for what you are doing as well. A script could parse a record, extract the graphviz/etc commands, send them to an external program, retrieve the resulting image, and insert it in-place.

This would require some finesse, though, since you are overwriting the contents of the smart record. What you need to do instead is have a single, empty smart record that is used as a display of other records; the other records then contain the raw data, and the display is read-only.

One way to handle this would be to have a smart record called ‘display’ that reads and renders every other record in the same group. It would actually be quite nice (thinking aloud here) if a smart group could be used for this: that is, since selecting a group causes nothing to be displayed in the 3-pane view, a smart group would be very useful if it could display an interpretation (via the attached script) of its contents. Making it act like a sheet, actually.

As far as adding a new feature goes, the graphviz/easytimeline tags do provide an interesting avenue for plugins, i.e. plugin could define tags that it handles and list the mime-type that it returns.

ndouglas · October 19, 2006, 3:01am

What if the applescript read the configuration data from the document comments? I’m not sure if DT has a limit in place on the length of comments, but I just checked by pasting in a decent-sized text file and there are no ill effects so far.

Applescript could read the comments to a string, md5 the string, and compare it to the names of the images in the /files directory. If it’s not matched, then the command is passed to the external program, with the final image name specified as the md5, and then import the image into the database and perhaps change an image name embedded in the HTML to a symbolic link that points to that file and deletes the old one? Processing power, I’d think, is less of a commodity than database size, hence the deletion…

That’s… complicated.

pokes around in DEVONthink’s dictionary
notes to his dismay that Graphviz has no Applescript dictionary, apparently

works

ndouglas · October 19, 2006, 3:53am

Some Graphviz progress.

I now have an Automator action that does the following:

“Get selected records”
“Combine selected records”
Echoes that input to ~/.temporarygraphvizfile.
md5’s ~/.temporarygraphvizfile.
Exports the hash and a /tmp/ and a .png to $graphviz
Checks to see if that file exists.
If not, it creates it using dot and then imports it into DEVONthink.
If so, then it exits fairly ungracefully.

So you end up with a ugly hash-named .PNG in your database. Not too shabby, and it seems pretty damned fast too. I haven’t done it on anything really extensive, though.

I suppose the next step is to actually use that to do something interesting… like build an HTML file around it. It’d be fantastic to get the imagemap functionality taken care of, but I haven’t dug down deep enough into DEVONthink to find out how this could work.

All in all, not bad for my second working Automator workflow :-/

Seems like I should be able to convert this to Applescript fairly simply, now that I have it working right in Automator.

ndouglas · October 19, 2006, 5:36pm

I’ll post what I have done so far.

It’s not terribly efficient or good code, since it’s the first one I’ve ever actually written myself, but it works.

What it does is evaluate the comment of a group, md5’ing and all, and checks if a file by that name already exists inside that group. If it doesn’t, then it passes it to graphviz and imports the result to that group.

I haven’t figured out yet how to make this genuinely useful (and I’d much rather read from a “graphviz” plain text file inside the group or something else better than this), so comments are appreciated.


on triggered(theRecord)
	tell application "DEVONthink Pro"
		activate
		set graphvizCommand to (comment of theRecord) as string
		do shell script ("echo \"" & graphvizCommand & "\"> ~/.graphvizCommand")
		do shell script ("export graphviz=`md5 -q ~/.temporarygraphvizfile` 
			export graphvizfilepath=\"/tmp/\"$graphviz\".png\" 
			if [ -e $graphvizfilepath ]
				then
					exit
				else
					/usr/local/bin/dot -Tpng -o ~/tempgraphviz.png ~/.temporarygraphvizfile 
					cp ~/tempgraphviz.png $graphvizfilepath 
					echo $graphvizfilepath > ~/.graphvizfilepathvariable
					echo $graphviz > ~/.graphvizvariable
			fi
			")
		set graphvizfilepath to (do shell script ("cat ~/.graphvizfilepathvariable")) as string
		set graphvizsum to (do shell script ("cat ~/.graphvizvariable")) as string
		try
			tell application "DEVONthink Pro"
				if exists record with file graphvizsum & ".png" then return
				import graphvizfilepath to theRecord
			end tell
		end try
	end tell
end triggered

edf · October 20, 2006, 12:26am

kalisphoenix, you’ve been busy

I’ve been storing record-specific configurations in the comment for the record as well, and for a very similar purpose (storing gnuplot options for DT data sheets).

The approach I took was to encode these options as “gnuplot{key:value}”, so that I could also store non-gnuplot comments with the record if need be. This script then builds the gnuplot script from the options.

I’m a bit unclear as to how you want to use this graphviz functionality. Based on your original post, I’m guessing you have an HTML (or RTF) file into which you add blocks of text like . When this file is viewed, everything between the is replaced with the output of the graphviz command.

The tricky part here is that you lose everything between the graphviz tags when it is replaced by the image. This is because in DT you cannot control the display of a record, only what its contents are.

So let’s start with a simple HTML record that allows graphviz commands. This can be a standard DT HTML record which you then run an applescript on, or it can be a ‘smart record’ (created, for example, with an applescript called NewGraphvizRecord) which has an attached applescript.

In either case, the applescript processes the source (or rich text) or the record. Every time it finds a set of graphviz tags, it does the following:

Create a graphviz command script from the contents between the graphviz tags. This command script is then saved in the graphviz_cmd variable.
Invoke graphviz (via do shell script) with the script as a parameter, and send the image to STDOUT, where it is captured in the variable graphviz_image
Create a new image record (name can be random, or based on a name parameter in the graphviz tags) with graphviz_img as the image property, graphviz_cmd as the comment (or as the plaintext, it doesn’t matter), and UpdateGraphvizImage as the attached script.
Replace the graphviz tags with a link to the record crated in step 3.

UpdateGraphvizImage is a script which reads the graphviz command from the record comment (or plain text property), calls graphviz to generate the image, and sets the image property of the record to the output of grapgviz.

A more sophisticated version would have the html-processing script create an empty image record, e.g. with


create record with { name:image_name, type:picture, attached script:path_to_UpdateGraphvizImage, comment:graphviz_cmd }

– that is, an image record (with an empty image property) with the graphviz commands embedded in the comment and the UpdateGraphvizImage script attached. The UpdateGraphvizImage script would then create the image when the record is accessed (the first access of course would be force by the script creating the record), and could do whatever caching you like (btw, the MD5s can be stored in a record in the database that is exempt from classification).

The benefit here is that you write the initial graphviz commands in the manner to which you are accustomed, then modify the comments of the image if you need to change the graphviz commands.

Well, to wrap this rather overcomplicated post up, this is similar to some fancy trigger stuff I have written and some xml-processing code someone else has wriitten, so writing it won’t be as intimidating as my poor explanation makes it sound. We should probably discuss this in a thread in the Scripting forum, actually.