DEVONthink for academic research - ADVICE

Echoes a lot of mine. I would also endorse use of replicants and importing over indexing. I used to index but always worried that the link would break.

DT3 works best when you have separate databases for separate projects but those projects are in themselves comprehensive. The more detailed and thorough and structured the more useful it is as an adjunct brain. I’ve found that the structure evolved over time, and my initial forays were eventually remapped as research progressed. Paralleling understanding in a way, but don’t worry about that, get in there and stick with it. It can take a while, even years, but it’s really worth it.

There’s other tools, like the supercharged mind mapping tool Tinderbox, which can link to items in DT3 and set up other views and ways of interacting. The rather marvellous Beck Tench demonstrates a detailed use of this over a series of videos https://m.youtube.com/watch?v=IOWLOMGFAEw

4 Likes

Just a curious question - isn’t the above the reason we need indexed folder(of literature) if projects are sharing the same/similar pool of literature. Importing articles into different databases may/will duplicate the same material? Perhaps the +side for import is highlighted points in the same pdfs can be different for different projects?

The second curious question is for all experienced academics regarding the recall of knowledge.
I assume that we all take notes after reviewing each journal(or other) article and by different methods. Some may use text highlights or annotations or comment field directly in pdf, some may take one main note for each article, some may take many snippets for each article (me), or as mentioned-above some may take hand written notes on paper or use Apple Pencil in different notes app.
Curious to know what methods are used by different people to recall their knowledge through these notes. For example, search by key words, tags or grouping?

Thanks for sharing.

1 Like

There are as many ways to use DT3 as there are users, so just consider this food for thought as you experiment.

Should you have a separate database for each project? Maybe yes if you have a half-dozen or so “projects” to work on long-term. But if a “project” is a publication you are writing or a client you are working with and you have hundreds of such “projects” then you might well want to put all projects of a given type in one database, such a “active publications” vs “complete publications” vs “active consulting cases” vs “completed consulting cases.”

As for a database of academic literature, for me at least I have one database with all academic literature I want to read or keep on file. It is fairly large because it includes RSS feeds which. automatically import new articles regularly. I keep one database for academic articles for all disciplines - because as you say you never can predict when a project will cross disciplines. When I start a new project that will require referencing academic literature, I create a Group in my academic literature database and then insert an Item Link to that group in my new project’s database. That way the project database is “complete” but might achieve that completeness through a link to a group in a different database.

4 Likes

This is a wonderful question. I’ve just been rereading an essay on historical research method that I love: https://www.lrb.co.uk/the-paper/v32/n11/keith-thomas/diary.

What really speaks to me here is somethign I’ve never managed to get going: a system in two related set of notes: one mapping onto the source and respecting the chronology of argument and flow of ideas; and another that organises these smaller sets of ideas into themed/headed snippets (in line with the early modern practice of commonplacing under themed heads that Thomas describes earlier in the article). As Thomas says he wishes he kept duplicate notes, but many have been cut up and fill an entire basement. Enter DT as our basement!

At this point in my academic life I regret not having invested much in the second, snippet style of note-taking (I’m still thinking through how useful it would have been, but looking at it now is also a reflection of new projects and new research questions). I’ve tried getting a Zettelkasten going but didn’t follow this through. Yet Thomas seems absoluly right that in order to generate ideas, it’s critical you are able organise your snippets by theme/subject, and reorganise, until patterns start to emerge and a narrative presents itself. To him the hard intellecual labour lies in the process of arrangement, and the writing follows more naturally.

I’m currently overhauling some of my own note-taking practice in DT. I will keep producing the reading notes per source/publication/work (book, article, chapter, primary source) – but I’m looking at the best way to extract themed snippets. adding many subject tags to notes per source doesn’t appeal to me – since there would be way to much irrelevant noise in long documents. If I get back to 1,000 words two years later, I just want what is relevant, not having re-engage with the rest. Unless of course I’m reminding myself of the arc of an argument of evolution of ideas, in which case I do want to first kind of note.

Happy to report back once I’ve landed on a good solution!

8 Likes

I have the same needs for both chronological perspective and thematic perspective.

DT is great for the thematic perspective, I do make heavy use of replication and group tags.

All my reading highlights end up exported as Markdown or PDF excerpts which i can categorize more precisely than the document itself.

I also categorize My thoughts, open questions, observations, notes … in the same way which make every topic group a very rich bundle of sources, or in other words, the dum of my knowledge and know sources on the topic.

What I miss is a better chronological view, a way to scrolldown all my notes/thoughts and scan their content visually.

3 Likes

@SebMacV & @benoit.pointet, I have similar goals w.r.t an academic reading / note taking workflow. Having atomic notes that are self-contained is key so that these knowledge elements can be filtered & gathered independently, reused, and arranged in new ways. With https://keypoints.app I‘m currently trying to develop a Mac app that supports this workflow. I plan to deeply integrate it with DEVONthink so that users can use DT‘s unique features to further work with their notes.

Here‘s a short demo screencast that shows a new note being created from a PDF highlight which is then tagged, rated, labeled, commented on and cross-linked:

More info is available in these threads:

https://www.outlinersoftware.com/topics/viewt/8814/35

4 Likes

That preview is very, very interesting - I will definitely want to try this as DT3 integration is added - this could be a terrific piece of software.

2 Likes

Thanks @msteffens for pointing that out!
Also interested to see that integration come to life.
Though I would favor staying in one app as much as possible, that one being DT atm.

Great question, @ngan! It’s often taken as given that we all develop an idiosyncratic process/workflow that is built around the requirements of our own fields/projects/methodologies. While this is true, by not talking about them (and asking these kinds of questions!), we’re missing out on important opportunities that could make our own processes work more effectively. In fact, it was while searching for answers to a very similar question that I first stumbled across DT.

@SebMacV, I had never come across Zettelkasten before you mentioned it earlier in this thread – and, of course, a simple Google search took me down a very deep rabbit hole. It seems like a pretty solid technique, and one that DT seems suited to. What was your experience of it?

1 Like

does the imac have an ssd?

Thanks to @rkaplan, @SebMacV, @benoit.pointet, and @richard.d for sharing the experience.

It seems that the search for a suitable method to recall knowledge is an ongoing process/challenge. I have been experimenting to use a custom script for taking note snippets on a journal article/book. By using the script, those snippets are consolidated under its original source on the one hand (it is almost like each article/book is having its own shoebox of index cards) and all snippets from different sources can be filtered and recombined into a consolidated summary by way of tags-selection or keyword-searching. I found this method quite effective for me.

However, while I believe that I have largely solved the puzzle from a technical aspect, I also found that the real challenge is methodological. Even when I have a highly organised superset of shoeboxes and the index cards in each shoebox can be recalled effortlessly by any combination of tags and search, the challenge is how the tags and keywords should be designed. In other words, the system of knowledge-retrieval is only as good as the way a researcher is categorising knowledge in their mind. Here, I think we are potentially in a catch 22 situation: we couldn’t have established a well-thought categorisation of knowledge in our mind at early academic career, and it would a rather impossible task to rearrange/redo our knowledge/notes once we have established such categorisation. Perhaps AI will take over the task of knowledge retrieval for us in the near future, while a human can keep focusing on seeking the right question and the methodology for answering the question.

Just sharing my 5 cents.

3 Likes

Indeed a catch 22.

But don’t underestimate the value of the process of reviewing those boxes and maintaining your system.

I use heavily replication and smart groups, have 1000 different topics covered in my main reference DB and everytime I take the time to curate/review a corner of it, I find it an immensely useful experience. Like clearing my mind.

First an aside in response to bosie’s question, one advantage of the Mac is that you can easily run MacOS from an external drive. I have a 2011 iMac, with HDD, but to speed it up I have plugged an external SSD into the Thunderbolt port, and it runs a lot faster.

But the main point is to echo the point that DTP can be used in different ways, and that cgruneneberg’s suggestions for searches should yield some useful pointers. I have been using DTP for ages to manage the photos I take of documents in archives. For me the main function was processing photos of single pages into multi-page searchable PDFs and then searching for documents on certain topics (unfortunately, with 50-year old carbon copies, OCR often fails, so manual search is necessary as well). It is excellent for this, and my basic group structure corresponds to the archives (as in National Archive, Harvard University Library, Duke University, …), Collections, Boxes and Folders from which the documents were taken.

I also dump a lot of other stuff into DTP, because it is searchable and I won’t (I hope) lose it. I started with a single database, though when it got too big, I split it, because it was just a pain moving and backing up a file that contained over 100GB of data. This is one advantage of indexing files. I have something like 120GB of documents, but the database indexing the lot, enabling me to search, add keywords, create and group replicants, is under 7GB.

The slow but thorough method for me was to dump stuff I am going to use from DTP into a bibliographic database, first Sente, until support for that ended, and then Bookends. I say slow, because if the documents are “home-made” from photos, bibliographic data has to be typed in for each one, and that is a slow process even with software such as Textexpander to provide shortcuts. So my workflow was DTP then Bookends and then into Scrivener (very highly recommended for the final writing). Now I am trying to identify more relevant stuff in DTP being more selective in what I drop into Bookends.

Another problem was the note-taking. I have done some in Bookends, and I created dozens of Groups in Bookends, though was not finding that perfect. So I began to explore other options, such as linking all my Bookends references back to DTP, so I could use DTP to sort stuff into groups, and create notes that would be the basis for writing. Avigail Oren and a colleague have developed a workflow that looks very interesting if you make much use of archival material (her target is historians, and you should find it via the searches cgrunenberg mentioned), and had I been starting from scratch I would probably have adopted it, but I decided that jumping to it mid-stream, when I already have around 28k refs in Bookends, and 70k docs in DTP, does not make sense.

Instead, I decided to add Tinderbox to the mix, and to try using that to organize materials and notes. The key is dragging references from Bookends into Tinderbox, where I can create a visual map of how documents link together, along with other notes I create. It seems to be working with the one topic I have tried it on, though I have yet to find out whether it will continue to work as I venture into other topics and try to create a map with far more complex interlocking networks of connections linking the various topics within my project.

So there are reasons for holding data in three apps, because they all do different things well, and for me the key is being able to go easily between a document stored in DTP, Bookends and Tinderbox. I do a search in DTP, find stuff that I either locate in Bookends or add to Bookends, from where it gets put into the network in Tinderbox. A bit chaotic, but as I have said, it proved harder than I thought to switch to a more rational system, making better use of DTP’s capabilities, mid-project. Hence my interest in picking up ideas from this thread.

1 Like

@msteffens That does look very interesting. I’ve been looking for the right tool to extract PDF annotations for a very long time and this would be a powerful possible solution. However it doesn’t look like much progress is being made? Nice professional looking website made with last blog post in 2017 and forums with no traffic and last posting in August of '19. I signed up for the notifications email list but am not very optimistic??

@greasemonkey Thanks for the feedback. The app is under continuous development, and I’m very much dedicated to bring this app into release state. However, progress is unfortunately rather slow due to this being a part-time effort, and due to real-life issues (like multiple losses in my family). Still, I’m working on this as much as I can.

1 Like

Besides keeping my notes atomic, self-contained and properly tagged, it also helps me to directly link notes with each other. In addition, I find it helpful to add overview/structure notes where I can organize individual notes for a certain topic/project into a hierarchy, and add sub-headings, comments, etc.

The goal is to create a network of semantically rich and linked notes. This network can then be visualized which further helps me not to get lost. Here’s an example which visualizes some reading notes.

1 Like

What software do you use to create that visual map - Keypoints?

Is there any way for you to share a set of notes in such a diagram format, including the links - or is the end result only for your own personal use?

I am pretty much in tune with @SebMacV who gave a nice outline. I have two databases several Groups in each, I go two levels deep, almost on principle. That is Groups within a database and a group within a group. I don’t even use tags now. I find search does the work for me and I just put everything in DEVONthink 3 for academics and research. I don’t index and blah di blah. I have way too much stored too. I use Houdah Spot via Alfred and I can find anything easily that way I have found, so much so that there are one two things recently I had forgotten to put into DEVONthink 3 and I hadn’t even realized, I was looking and opening them from HS and Alfred.
I do have a prefix system for anything I write myself, or heavily edit. That is in the file name, is quite complicated but used by me for years now and again I can just ‘call it up’ easily. I could, in truth do it with DEVONthink 3 and Finder alone these days probs. Big set back for me was when Apple shielded Mail from HS etc. I wish that was back. Though if anything is of purely research interest I just put a mail into DEVONthink 3 and it gets found that way. There is no clear principle on that one though and I will lose something one day. Good luck with the research.

1 Like

Yes. It gets dynamically created from your currently displayed notes as you browse, filter & select them. This little screencast shows this in action:

The end result is a regular PDF (generated using Graphviz). The elements in the PDF have keypoints:// URLs (so that clicking any of them will select the corresponding note in Keypoints). But I could also envision an exported PDF version where the URLs have been set to the corresponding x-devonthink:// URLs, or similar.

Also, since this is done using Graphviz, I guess that a script could also generate a similar graph output directly from any inter-linked DEVONthink records.

Or did I misunderstand your question?

Thanks

Have you thought of a way to publish such a graph on the web, either publicly or privately?

The one stumbling block I realize I have encountered when I have looked at similar programs previously is that there is no good way to share my map or linked notes with a colleague or client. I would think that most academics or proessionals using something like this would want to share their work product with a student or another professor or as a post on a discussion forum or with a client etc. It looks gorgeous but if only I can see it (or if other people have to set up special software to see it) then that limits its use considerably.