Managing context/token usage when working with long books

DEVONthink Artificial Intelligence
1

I’m just getting myself started with DT4. I’ve been a longtime DT3 user, but only recently have I started to find LLMs useful for the kind of academic research and writing in the humanities/social sciences that I do. I don’t fully have a handle on all the new LLM-oriented AI features in DT4.

The problem I ran into when using other non-DT harness (ie. Claude Code) for working with many hundreds of pages of academic monographs and anthologies is that I very quickly hit my token limit. For instance, I might want to use the LLM to help me find specific passages from some books I read a year or two ago, with the aim of refreshing my memory and providing me with the specific relevant passages of the book so that I can reference them or quote them in my writing. The raw .txt of a 300-500 page book might be somewhere around 500kb - 1MB. That requires a lot of tokens to scan!

So I started brainstorming and running small experiments to see if I could figure out how to make this process more efficient. I started designing a prompt for ingesting the full text of academic sources, dividing it into smaller chunks that the AI would compress into an abstract form to store in a lookup table, which would refer back to the original pages of the source text and indicate ranked relatedness to similar passages throughout the book. In effect, I was almost creating something like a neural network within the LLM. I realized this was silly, I shouldn’t need to reinvent the wheel here. Perhaps this is the kind of problem that other DT users might have spent time trying to solve.

So I’m wondering what strategies or tools other users might have found to be helpful in working with lots of text. An important thing is that this workflow must not be over-optimized to distill the “lessons” and “insights” or “interpretations” of the source text. Rather, I need the LLM to be able to point me to specific page numbers/specific quotations in the source text for me to reread on my own.

My desire to find a workflow like this is heavily inspired by my use of DT3’s advanced search features and boolean search operators. I love being able to enter a sequence of words or phrases with the NEAR or AND boolean operates, and have a ranked list of books and chapters from my research library returned to me, with specific pages highlighted for immediate access. Basically, I’m curious if theres a way to use LLMs to give a similar workflow with the ability to enter more sophisticated searches. So for example, instead of searching (econom*|financ*|money|monetary|cost|purchas*) NEAR (media|text|litera*|news*|communicat*), i’d like to be able to describe in prose that i’m searching for passages that discuss the relationship between economy and communication media.

Have others here found ways of dealing with this type of task?

2

One option is to use DEVONthink’s search assistant to generate the search syntax, e.g. the above quote was converted to this query: econom* NEAR ("communication media" OR "communications media") kind:document by Gemini 3.7 Flash.

Both the chat assistant and MCP tools can also search in PDF documents for keywords and their synonyms and also take advantage of the table of contents of PDF documents (if available). That way you could tell the AI to look in a certain chapter or to search for certain keywords or the AI could try on its own.

Finally, for simple tasks (e.g. summarizing) it’s also a good idea to use less expensive models (e.g. Claude Sonnet instead of Claude Opus or Claude Fable) which reduce the API costs and do not use the subscription limits that quickly.

3

Hm. Thanks for the response, it clarifies that the functionality I’m looking for doesn’t seem to be present in DT natively (so I’d have to build it myself). The Gemini-generates query is actually a less effective search query than the one I created by hand.

Does this leverage DT’s existing internal database format to make the search more efficient than simply pasting the PDF’s content into another harness ie. Claude Code?

4

More simply put, the external AI can make use of DEVONthink’s search function and also look for synonyms. Not only is that more efficient it’s also more cost-effective than pasting into some other app.

5

Is time a limiting factor here? (E.g. you want the answer in seconds?) Since you just want to query an existing text (which I assume is in DT4) you could just install a local model. Then you wouldn’t be limited by costs. It’s slower (I assume) than the big cloud models but there are no costs (and nothing needs to leave your device) and you’re only using it to search an existing document so you’re less concerned with its abilities.

Personally I find DT’s native search function suffices for tracking down paragraphs in books (all my digital books are in a database for this purpose), but I have a local model installed for DT and have used it once to look up some data in a government report that didn’t have keywords for me to use a Boolean search. It took a minute or two but the LLM tracked it down.