Simpler pdf workflows on Mac / iPad for writing & research

Dear Forum

Currently my workflow looks something like this:

Collection of external data (pdf) sources ->
|
Imported into a ref manager (Zotero) pdf database and indexed by DTP ->
|
IPAD/MACBOOK PDF ANNOTATION?
(reading/writing between DTP / reference manager - MESS!!!) ->
|
On to writing with Scrivener/Word

I recently purchased an iPad which I thought I’d use to annotate pdfs more comfortably, away from my desktop setup. Now I’m looking for a simple but efficient workflow with a MINIMAL number of apps. I realize this kind of “holy grail” question has been rehearsed countless times but I hope people might still share their advice.

I currently use a Macbook with DTP that indexes my Zotero standalone pdf database (about 4000 pdfs). I’ve considered switching to Bookends for the advantages it offers (pdf viewing / annotation, spotlight searching) but I haven’t yet made the switch. Zotero has always been solid enough. Ideally I’d like to set up a shared pdf folder (in the cloud?) that DTP, Zotero AND other apps (Bookends?) could read/write to/from, but I fear data corruption (Zotero advises against it). I do have my Zotero database synced with Box (a free Dropbox competitor) and have tried things like Papership for iPad pdf annotation but it’s all beginning to seem like overkill. I’ve tried the notes export scripts for Zotero-DTP and also Skim-DTP but it seems cumbersome and I find it difficult to keep that part of my “workflow” flowing. I’d consider doing away with a reference manager entirely but DTP doesn’t offer citation features as far as I know. I do my writing in Scrivener and Word and that part works relatively well.

What simple workflows do people suggest for pdf annotation (on or off the iPad) that can integrate with a DTP / reference manager database? Do people find the DTP built-in pdf annotation features sufficient or is it necessary to annotate in something like Skim/Goodreader/Goodnotes and/or Zotero/Bookends iPad apps (Zotpad/Bookends On Tap[)? What about integrating something like Circus Ponnies Notebook? I haven’t tried DevonThink’s own iPad apps (To Go, Notes) - do they help?

I’m ready to refashion a new and simpler workflow from scratch - ideally centered in the Devonthink ecology - and one that allows me to focus on the writing and research rather than workflow tinkering. Thanks.

The trick for me (A PhD candidate, Humanties) was to recognise the stages of the workflow and then find the apps that fit.

You need four apps:

A pdf reader - You need something to annotate all those pdfs on whatever device you choose. But here is the absolutely key thing: whatever you choose it must allow you to export the annotations and your comments to a central resource. Which app you choose doesn’t matter as long as you can get those annotations out of there.

A central resource for all your stuff- the pdfs, the notes and annotations. DevonThink is where every annotation and comment goes. There it is indexed and cross-referenced. The fact that DT also indexes the pdfs as well is an absolute bonus.

A writing App - Scrivener for drafting, Word for output.

A citation manager - something that supports Cite-As-You-Write. I use Bookends for this and it’s really good. I write and then company Y to switch to Bookends, select the cite and command y and I’m back in Scrivener with the cite inserted and continue writing.

I went round and round the houses for about six months trying to figure this one out, and that is what I came to. You don’t need a citation manager that also annotates. You don’t need a central Repository that also supports cite as you write. You just need apps that work for each category.

For instance, both Bookends and Sente support annotating the pdfs, and Sente in particular syncs very well between your Mac and your iPad. But neither is especially graceful at sharing out the annotations, neither supports full database search. So, you have a note attached to a page in a pdf, but unless you’re in that pdf it cannot be found easily.

So, when I find a something I want to annotate I highlight in the pdf, create a text file in DT, paste in the material, write whatever comments I want to add, give it a some tags and save. Now as well as the pdfs, all my notes and comments are saved and indexed for searching in DT.

I automate this process with a Keyboard Maestro workflow.

Result, each app does a different job, and has a specific role. It just makes things clearer.

So, find a pdf reader that allows you to get the annotations out of it, settle on a citation manager and just let them do those jobs.

That and a dollar will get you a cup of coffee someplace… but only cheap places :smiley:

The answer is probably different for everyone. As a historian (PhD in East Asian Studies), I found it possible to write much of my dissertation on the iPad, though I am not sure I would do it that way again, because I’ve grown rather accustomed to all the benefits that DEVONthink provides :slight_smile: Here are a few thoughts about my dissertation process.

Dissertation Abstract
christopher-mayo.com/?p=654

Reading
christopher-mayo.com/?p=1066

Writing
christopher-mayo.com/?p=1275

Paper to Paperless
christopher-mayo.com/?p=282

If I were to do it all again, it wouldn’t be terribly different, but it might look more like this:

  1. Digitize as much as possible. I did this for the dissertation, and the stuff I collected has paid off every day now. I have a library of many thousand PDFs (books and articles) with me 24/7 no matter where I am in the world. OCR it all and it is searchable. For me, this was the key to getting everything done. It was quite time consuming and I had to overcome some mental resistance to the process, but once I saw the benefits, there was no turning back.

  2. Don’t annotate PDFs unless you are highlighting to export the annotations later. Even this isn’t worth the effort, in my opinion, but I will do it if I am only able to read and can’t take notes at the moment (traveling, for instance). The iPad is a great e-reader, and it is on my desk all day every day showing me the research materials. Far more effective are reading notes (see the link above) created separately, but linked back to the original PDF if necessary. Of course, if you are following a standard titling scheme for PDFs (see my site for my naming conventions), you don’t even have to bother with that.

  3. Don’t bother with bibliographic software. Unless you are manipulating your data a lot for different citation formats and different projects, there isn’t much point. A single text file (using markdown) with the Chicago Manual of Style Bibliography and Footnote style (easy to master) serves as my master bibliography. For projects like my dissertation, I just copy/paste from there. Honestly, a few hundred citations is no big deal. As a historian working with Chinese, Japanese, Portuguese, Spanish, and English sources, I knew I’d have to fix everything anyhow, because citation software does a pretty poor job handling translated titles and whatnot.

  4. Write in plain text if you can. I like Scrivener a lot, it syncs beautifully with Notesy on the iPad, and you can get a lot of work done without worrying about formatting. BUT, there is something broken in my head. For some reason, I just write a lot more effectively in Word with all of the formatting stuff. I somehow get immense satisfaction from seeing it as close to the final format as possible, even though I am also frustrated that I am not using something far better suited for the writing process such as Scrivener. Eventually, I’ve had to accept that this method (Word) works better for me, and I am stuck with it, even though the app is blurry on my Mac, and a poor imitation of its Windows brother.

  5. DEVONagent (automatically scouring the net for stuff every day, clipping it as a one-page PDF, and putting it into DEVONthink) + DEVONthink (especially smart folders) has been great for organizing everything. There is a little by researchers on the web about how they use DT.

Each to their own etc. etc…

Some ramblings from a fellow PHD student.

Tying together my thoughts about what has been raised above:

1.) I too prefer to use tools that specialise in their particular field.

I also use Bookends. The support is second to none - which helps, since the Manual etc. can be rather confusing… I’m still discovering features, where a face-palm or three has me thinking “if only I realised that earlier”…

I haven’t really found a way into its annotation and notes feature. I don’t find it to be particularly intuitive, and haven’t had the time to dig in… Others appear to find it really good - so maybe its worth exploring.

Regardless, I use BE exclusively for the citation side of things. The ability to generate an instantaneous bibliography, is a real plus as well. And whereas many might not see the need for a reference/citation manager - if you’re one that intends on publishing frequently, then it’s a no-brainer IF your discipline makes use of varied citation styles. My field, sees virtually every Journal use a unique referencing style - being able to edit one Format, and having all citations transform instantly to match that style at the click of the button - priceless, imo.

Similarly, DTPO takes care of managing/accessing my research. All the PDFs are OCR’ed on my mac - as suggested, this is basically non-negotiable. To truly get the most out of your machine, it needs to be able to “read” everything. Take the time to do this - it will pay you back a-thousand-fold in the long run.

[I used PDFPen Pro and Hazel, to automagically scan and OCR several thousand PDFs on my mac, over the space of about a week. It was the reason I bought Hazel. Of course, the latter does much more - but even if only used for OCR’ing - worth every penny.]

Scrivener to write. Word to convert right at the end.

Keyboard Maestro in conjunction with ActionsApp for iPad, ties everything together - I have workflows set-up where I can type an author’s name in Scrivener, tap my iPad which invokes a KM macro, jumps into BE, and using ShortcatApp, runs a search for the author as selected - one more tap on my iPad to insert the relevant pages, and back to Scrivener to insert. That sounds far more complex than what it is - a few taps, and done.

Scrivener and DTPO play nicely together. BE and DTPO - there are basic functions that would be far more useful were my BE library to be complete - since its still a work in progress, I prefer to search inside BE, than accessing it from with DTPO. That being said, when the day comes that everything inside my BE library has been autocompleted, with will become really useful.

Annotation.

FWIW, a few suggestions that I found really help.

I use several primary highlight colours to highlight PDFs inside DTPO, using the built in annotation feature. I bought a little Bamboo Intuos Pen & Touch Small, and find it does a fine job of allowing quick highlighting.

I use 3 colours to indicate “summary variants”. Then, in addition: Red for VITAL bits; Purple for “To quote”/Also NB parts"; Electric blue for “look up later”; Green for “This is why I’ve added an Annotation RTF file” [DTPO’s built-in feature].

Those 3 summary variants, actually work really well. Initially, I always used to highlight in one colour, typically yellow. I’d maybe differentiate NB parts with Red. Then the purple etc.

Thing is - when you go back to review that article 6 months later - you are visually met with pages and pages of yellow paragraphed text… It took a while to find what you are looking for, since there was no easy visual way to distinguish the text… Then I started using the 3 colour approach.

First point worth highlighting - Colour 1. NEXT (unrelated) point worth highlighting, Colour 2. NEXT point, Colour 3. NEXT point, back to Colour 1, and repeat.

This way, when I come back to review that PDF, I start reading the 1st highlighted part. If its not valid, I can skip along to different points far quicker, by being able to distinguish between the different colours. It sounds kind of lame, but it really works a treat for me - it’s something I suggest you consider if you’re expect to do plenty of highlighting, digitally.

I ONLY use the built-in DTPO annotating service to highlight - anything else is done on the iPad, since making sticky notes, will mean they remain unsearchable - something to be avoided, if possible.

I set the same colours up in Goodreader on my iPad. That way, when annotating on my iPad, I’m using the same criteria - which provides useful consistency. I popped something up over here [[url]Need some help re my Annotation workflow-DTTG<>Goodreader], about my use of Goodreader on the iPad, with DTTG as the conduit between DTPO and Goodreader. I must probably look at things again, given the new developments in iCloud, but haven’t had the chance yet.

The above procedure is a bit clunky, but it works. Whether it’s for you, only you will know. Skim would obviously make this process much easier, as would the Highlights App [see here: [url]"Highlights" app - export annotations to DEVONthink, etc.]] - but then it would ignore using the iPad as an annotation tool - something it excels at, imo. My clunkflow at least allows me to generate annotation-sticky-notes inside my PDF on my iPad, that end up being searchable inside DTPO…

To conclude - and hopefully, this isn’t too presumptuous of me - but since this also interests me, I’ve been following many of these questions as well - here are some useful forum posts, on the different workflow-type options:

** Make annotation files for each citation plus notes+tags: [[url]Make an Annotation with Links, Notes, Tags v2]]

** PDF Annotation (made easier) - KM and ActionsApp: [[url]PDF Annotation (made easier) - Keyboard Maestro & ActionsApp]]

** Mystified by Annotation - is there a plain-language tutorial: [[url]Mystified by Annotation - is there a plain-language tutorial]]

** Linking Bookends entries and pdfs in Devonthink: [[url]Linking Bookends entries and pdfs in Devonthink]]

** Integrating Scrivener with DTPO and BE: [[url]Integrating Scrivener with Devonthink and Bookends]]

** Make an Index Document (RTF) for a selection of Documents: [[url]Make an Index Document (RTF) for a Selection of Documents]]

** Manually cross-linking & adding a RT note to items?: [[url]manually cross-linking & adding rich text notes to items?]]

TL:DR >> There are no simple workflows. :laughing:

1 Like

Thank you for these replies which offer some useful perspectives on different workflow approaches. I want to hop in here to add my reflections AND I hope others will still join the conversation.

@terrydev
I enjoy your “four app approach” which helps clarify in simple terms some primary workflow phases/tasks. Although different apps increasingly adopt new and overlapping features, you highlight the importance of picking one app per primary task/phase and sticking to it. Since you didn’t mention one, I wonder which app you might recommend for the pdf reading/ annotation phase (on the iPad and elsewhere) since this is my core problem area. Also, in the course of trying to simplify, I wonder if we might consider some exceptions to the rule of “one app per task.” For instance, I’m still curious, in specific terms, how people make pdf reading/ annotating work with an iPad / DT-centric environment. You seem to imply this at the end of your reply.

@FROBGOBLIN
I really enjoy reading about what I interpret as your more “barebones” approach (e.g. minus bibliographic software). Like terrydev’s reply, you also focus on the basics. I will however need to follow the recommendations of a dedicated cite-as-you-write app like Bookends or Zotero. (Currently I also use Zotero’s database as a pdf repository indexed in DT.) I enjoyed your other post on notetaking and organization. I wonder if you might elaborate further on how you integrate the iPad in your workflow (e.g. Notesy). On another point, I also sympathize with your need to write with formatted text in Word. I often create rougher drafts in Scrivener (that sometimes verge on a veritable database of ALL my thoughts for a given topic) and then export to Word for finer writing and argument development. I would like to use the iPad more for this type of writing but so far I’ve found it clumsy so I stick to writing on the Mac.

@Cassady
Your reply is full of some nice tweaks and thanks for linking to other discussions. My head is still spinning from all the suggestions! You underline some of the issues I find when trying to sort out the best path through what has become a digital jungle of possibility. I don’t really have much to comment on but I am struck with potency of your opening words: “Each to their own etc. etc…” This is the advice one often reads when it comes to workflows: essentially “figure it out on your own.” On the other hand, my sense is that once an app or suite gets it right people tend to gravitate to that way of working. Consider the popularity of Word or Google and their influence on millions (billions?) of workflows that trace a series of steps in more or less the same manner. My sense here is that once an app/suite “gets it right” it then becomes the dominate workflow for that given task. Diversity and experimentation seem to occur where no clear advantage yet exists. I trust that one day there will be an innovation that collapses the writing/research workflow into an intuitive and simple process but it’s still emerging. Until that happens I suppose tweaks like the ones you suggest are something of a necessary evil. My aim for posting was an attempt to take stock of the current situation and see how people might inform my own approach. Thanks for introducing several apps to consider. If you reply I would love to read any further clarifications about how you use annotation tool(s) - primarily iPad app(s )- in conjunction with DT to avoid the problem of unsearchable pdf notes/ annotations.

Thanks again and I look forward to more replies!

In my field, it’s mainly one kind of citation style (CMS), and the bibliographic software, even when it gets it right, isn’t worth the effort for me. If you are changing styles a lot, of course, then it is too much to mess around with on your own. Perhaps, if I worked with fewer primary sources, only worked in English, and didn’t have to cite page numbers, I’d find it more appealing. I’ve tried a bunch of apps. They all seem OK.

If you use Scrivener, it can sync to Dropbox, and you just point notesy there so that you can work on your notes on the go. It’s easily done and works really well. I did the first draft of the dissertation there. Then, I switched to Pages and finished my dissertation with that. Pages was OK, and it did enable me to get a lot done on the iPad. Here is a post about my iPad equipment.

christopher-mayo.com/?p=1477

I also found it useful to have all of my PDF content with me on the iPad, so I textified my PDFs. This was pretty helpful, especially putting it into Evernote. Once DEVONthink To Go is updated, it might be a good place to search for things.

christopher-mayo.com/?p=551

But, Pages isn’t really very robust (on the iPad or the Mac), and it is buggy at times. They dumbed it down immensely after I finished my dissertation, so it is even less appealing to me now. Because I work better with formatting, and Pages is so awful (in my estimation), I do less content creation on my iPad these days. It’s a shame, but perhaps inevitable. Besides, as I mentioned above, DT is really tough to live without.

Thanks, everyone. This is very useful. I keep hoping someone will post a sample DTP database that newbies could play with, that has all these features activated… but I guess that is harder than it sounds.

For reference, I’m transitioning from “2nd generation” to “3rd generation” (roughly) tools for academic writing. I have written a book draft using mainly Eaglefiler (database for PDFs), OmniOutliner (for note taking), and Word. Long ago I OCR’d as much as possible. (Still not easy for books.) Now I have shifted to Nisus and Scrivener for the writing, but taking notes on my readings using an outliner program is not effective once the number of documents gets in the hundreds, and I’m way past that. So, I have been looking at Bookends, DTP, various iPad tools, and the like.

thanks again.