If you want to use AI in DEVONthink, you can choose between server-based models (the big ones) and local models. Whether it’s for privacy reasons or just the novelty, running AI locally on your Mac may be something you’re considering. Here are some thoughts on choosing a local model.
First, you’ll need an AI application for hosting and running the model. The two directly supported in DEVONthink are LM Studio and Ollama. If you go to either site, look for a Models link where you can view a curated list of models. Regarding downloading models, see each app’s documentation.
Running local AI effectively is largely tied to the amount of memory (RAM) your machine has. The more memory is available, the larger the model you can use. As a non-technical rule of thumb: calculate half to 70% of your total RAM and use that number as the number of parameters of the model you use. For example, on a Mac with 16GB RAM, 50% is 8 and 70% is 11.2. So when you’re looking for models, look for ones with 8B (billion) up to 12B in the name. These will not be as speedy (or as accurate) as commercial models – compared to the large models, that’s actually not many parameters – but using them should also not lock up your machine from a lack of resources.
Also, you’ll want to look at some of the information about the model you’re considering. Models are built with certain functions in mind. You may see vision meaning the model can examine images, tooling or tool use meaning it may be able to perform certain actions, not just answer questions, or thinking/reasoning meaning the model can reflect on its responses and change them before providing you with a reply. Some models support multiple functions.
If Ollama is running, DEVONthink can chat with your local model through it. In LM Studio, click the Power User button at the bottom of the window, then the green prompt icon. Flip the switch to start the server and it should be available to chat with too. As a bonus tip, choose one app or the other but don’t try to run them simultaneously.
We’ve also written an introductory post on local AI, which you may want to take a look at.
3 Likes
Thanks for this. I’m currently using the standard edition of DV 4 but already have LM Studio installed (not sure I’ve had great results to be honest). What would I gain from integrating the two? Happy to pay the extra, but would it just make it simpler to get things into LM Studio, or is there a really useful way of getting more out of both by bringing them together?
Usefulness depends on the situation but that also applies to using AI, in general. If you feel you have a use for AI in DEVONthink, you could use LM Studio instead of a commercial LLM. But as noted in the blog post and the documentation, local AI comes with its tradeoffs.
1 Like
Fair prompt. Mostly I’m hoping to get summaries of long or multiple documents, usually living in a folder in DV. They’re private so they’re not being fed to a corporate machine, they have to stay local. Currently I’m doing things like ‘select all, copy, paste into LM Studio’. Would DV Pro mean I didn’t have to do those mundane steps, for instance?
While some things depend on the AI model you’re using, yes you can make inquiries about selected documents. Also, your hardware will affect the speed and possibilities.
1 Like
Hi - I have installed Ollama and Ouen but am having difficulty populating the dropdown in the AI settings so that I can select Ouen as a model. Can anyone suggest why this might be so?
You probably used the wrong URI in your settings. Not possible to say without more information.
Thank you — in fact I found that the problem was that I was inserting http:// before the address. For some reason DT doesn’t like that…
I doubt that that’s the reason. At least here, it likes http://localhost/whatever.
Post a screen capture of the AI > Chat settings with your URL filled in.
Here you go – it’s working now. DEVONthink added the “http://” automatically after I inserted the rest
Is there any plan to support oMLX?
There are no plans currently to support additional providers/tools (as a new one pops up every day) but the OpenAI-compatible endpoint should actually work in this case.
1 Like
Agreed the OpenAI-compatible endpoint is very helfpul in these situations.
Am I correct that it is only possible to configure one OpenAI compatible endpoint?
Any chance you could consider the abiltiy to choose among multiple OpenAI endpoints when selecting an LLM model?
That’s right.
There’s always a chance but there are no such plans yet.
I was in trouble setting up oMLX until I found out that I should use “localhost” instead of IP address in API URL.
Yes, the OpenAI-compatible endpoint works. I had it all wrong.
