No worries. Thanks for the clarification.
I’ve tried using a local AI with LM Studio. In my case calling the same AI modell from Devonthink has been significantly slower than within LM Studio using the same prompt, e.g. something trivial like “hello”. Is this common?
DEVONthink isn’t LM Studio and vice versa so logically the behavior won’t necessarily be the same. Also, queries from within DEVONthink use things like internal “guardrails” to preserve privacy and safety for your data(bases).
DEVONthink provides a system role, context information and/or tool calls (all depending on the model and its features and context window). This is of course much more information that has to be processed.
Yeah, I get that. It seems to depend heavily on the model too. With one model the difference was seconds versus minutes, making them unusable from Devonthink. With other models the differences are much more subtile.
I know goals can vary much from one user to another and therefore there is no “right” AI model for everyone. But is there a list of local AI models worth trying with Devonthink? To give people something to start with?
The answer to that will depend on the specs of your Mac, although the answer is generally “the biggest you can support on your machine”.
With regards to times though, are the answers taking longer because the questions are more complex? E.g. I’m assuming you’re asking questions of a file, which you might not have been doing in the LM chat window, and you’re now directing the LLM to find specific answers? I hadn’t noticed a big time difference during my tests, but equally I was assuming my questions were more complex when I was testing it against a file in my database.
1 Like
Problem is that there are so many models available. I can’t try them all. Maybe a possible approach could be recommended models for each category. The category could be defined by needed size in RAM (6GB, 16GB, 24GB, 32GB).
Trying that approach my experience so far has been:
16GB:
- Good:
- gemma-4-26b-a4b-qat (responsive, good for summaries, discussions on papers)
- Not usable:
- qwen/qwen3.6-27b (answers take minutes if at all)
6GB:
- Good
- gemma-4-e4b (responsive, still have to try)
- qwen3.5-9b (responsive, still have to try)
Is something like that helpful to others?
It’s purely anecdotal and the results could also vary on your machine.
I recently stumbled over https://canitrun.dev and found it quite useful for choosing an appropriate model depending on the given hardware resources.
1 Like
Thanks for sharing this page. It definitely looks like it could be helpful for many AI-interested folks.