I’ve been so impressed with DEVONthink’s LLM integration that I tried using DT4 as my go to AI app. This AI expert inspired me to give it a try.
Here’s What I Learned From the Process
You technically can’t get the same experience from DT4 as a Chatbot. I expected an Apples to Apples result but it’s Apples (API Integrations) to Oranges (LLM Chat Clients)
Below I’ll provide more technical explanations but first a couple of questions for the DEVONthink team (@BLUEFROG and @cgrunenberg
Questions for DevonTechnologies
- Anything I’m misunderstanding here? – see below for detail:
- Did I miss anything in the Help documentation or forums that speaks to this?
- What’s on the roadmap for future DT4 integration? While the API is the primary limiter there are a couple of things the development team could add (streaming) to improve the DT Chat experience.
- Would protocols like MCP or A2A address any of these issues?
- NOTE:I am in no way finding fault with DT4 integration with LLMs. Rather, in learning these details it helped me understand and appreciate the level of development DT has already done that additional LOE required to create parity --* it is not trival
Technical findings from Perplexity:
What’s different about the APIs
When DEVONthink talks to an LLM, it is just making stateless API calls. Each call has to include all relevant context (your prompt, some document text, maybe a bit of history), and the model returns a single response, then the interaction is over unless DEVONthink explicitly makes another call.[1][2]
By contrast, ChatGPT, Claude, Gemini, etc. wrap those same models in their own orchestration layer: they maintain conversation state, manage long histories, schedule tool calls and retrieval steps, and surface streaming output and intermediate “thinking” in a chat UI. None of that orchestration logic is automatically exposed through the raw API; a third‑party app like DEVONthink would have to re‑implement it itself.[3][4][1]
Why chat UI feels more “continuous”
Provider chat UIs are optimized to:
- Keep a running conversation context server‑side without you bundling it on every request.[1][3]
- Stream tokens as they’re generated so you see incremental progress rather than an all‑or‑nothing response.[3]
- Chain multiple internal steps (retrieval, tools, re‑asking the model) before you ever see a final answer.
In DEVONthink, each AI command is usually a single request/response around a defined selection or document. If it hits token or time limits, or if DEVONthink doesn’t program a retry/continuation strategy, the process just ends without the conversational back‑and‑forth you’d naturally do in ChatGPT or Claude.[5][6]
Actual limitations from providers vs app design
There are some real provider‑side constraints on the API:
- Token and rate limits mean DEVONthink has to chunk large documents or long research tasks; if it doesn’t manage chunking well, runs can stop part‑way with no obvious feedback.[2][1]
- Some behaviors allowed in the hosted chat products are more restricted or differently configured via API (e.g., safety policies or tools only exposed in specific SDKs).[7][2]
But most of what you’re seeing is not “the API can’t do what the chat UI can,” it’s that:
- The chat UI has a rich, proprietary orchestration layer you don’t see.[1][3]
- DEVONthink’s integration is intentionally simpler: it issues API calls tied to documents or selections, not a long‑lived, self‑correcting research agent that keeps asking for more context and confirming with you.[6][5]
How this maps to your DEVONthink experience
Putting it together for your specific symptom:
- In ChatGPT/Claude/Gemini, a long research task looks “continuous” because their UI keeps the conversation alive, retries when needed, and lets you steer with micro‑feedback.[4][3]
- In DEVONthink, a long research task is just a series of API calls DEVONthink chooses to make; if the operation exceeds context, token, or time constraints and DEVONthink hasn’t implemented streaming, chunking, or resumable states, it just stops and you see no incremental feedback.[2][5][6]
So the explanation you “heard” is directionally right: the APIs expose raw model access with stricter constraints and no built‑in conversational orchestration, and DEVONthink’s current design doesn’t bridge that gap the way provider chat UIs do.
Sources
[1] ChatGPT vs OpenAI API - Key Differences and Use Cases | Learn ChatGPT vs OpenAI API - Key Differences and Use Cases | Learn - Predictable Blog
[2] Differences in API and ChatGPT end user app Differences in API and ChatGPT end user app - API - OpenAI Developer Community
[3] Why Chat is the Best Interface for LLMs (for now?) | Ben Perlmutter Why Chat is the Best Interface for LLMs (for now?) | Ben Perlmutter
[4] Rethinking LLM interfaces, from chatbots to contextual applications Rethinking LLM interfaces, from chatbots to contextual applications - Eric J. Ma's Personal Site
[5] Using Local AI in DEVONthink? - DEVONtechnologies DEVONtechnologies | Using Local AI in DEVONthink?
[6] Really impressed by AI implementation in DT4 beta - Page 2 Really impressed by AI implementation in DT4 beta
[7] ChatGPT’s API returns worse results than it’s web UI and it can’t … https://www.reddit.com/r/ChatGPTCoding/comments/1k2y2lr/chatgpts_api_returns_worse_results_than_its_web/
[8] Understanding how AI API’s work in DT4 - Artificial Intelligence Understanding how AI API's work in DT4
[9] Stuck in the (ChatGPT/Claude/Gemini) Chatbox? Here’s When You … Stuck in the (ChatGPT/Claude/Gemini) Chatbox? Here's When You Actually Need the (DevonThink API Integrated AI)
[10] DT Integration with AI - DEVONthink - DEVONtechnologies Community DT Integration with AI
[11] Devonthink 4.0 is out - Software - MPU Talk Devonthink 4.0 is out - Software - MPU Talk
[12] A Proposal for the Integration of DEVONthink and ChatGPT API A Proposal for the Integration of DEVONthink and ChatGPT API
[13] What’s Your Go-To LLM Interface for API Chat & Why? - Reddit https://www.reddit.com/r/ClaudeAI/comments/1hl6yx7/whats_your_goto_llm_interface_for_api_chat_why/
[14] DEVONthink 4 - Use AI to SUMMARIZE documents in your databases https://www.youtube.com/watch?v=FADhoGX06K8
[15] Why We Need Standardized UI for AI - OpenAI Developer Community Why We Need Standardized UI for AI - Use cases and examples - OpenAI Developer Community