I’m currently trying to optimize the performance of my local AI models. I’ve already experimented a lot with LM Studio and Ollama, and I’ve now come across oMLX, which offers more options for speculative decoding. I’m hoping this will speed things up. The problem is this: I’ve entered the local server’s URL in DEVONthink as the OpenAI API URL: http://localhost:8000/v1/chat/completion.
However, this means that the selection of available models isn’t immediately accessible in DEVONthink; instead, oMLX loads each model into memory and then unloads it (once the memory limit is reached). I have over 30 models I’d like to try out. Consequently, it takes forever—with high CPU usage—before I can select a model in DEVONthink.
Of course, I’ve already asked ChatGPT what’s going on. The theory is that DEVONthink does more than just call GET /v1/models, which is why oMLX is forced to load each model. Is that correct? Does anyone know their way around oMLX and have any ideas? The AI just suggests to use fewer models. But that makes comparing different models quite cumbersome.
If the service doesn’t provide the models’ capabilities (e.g. LM Studio, Ollama or OpenRouter do this), then DEVONthink probes each model once to check whether it supports chat completions, tool calls and/or vision. A future release might skip this in case of local, unknown services & models but then the possibilities would be limited.
At least on my machine (M1 Ultra) using the default settings and a context window of 32k Ollama is approx. 1.6 times faster than oMLX when using Gemma 4:26b and 2.8-3.5 times faster when using Qwen3.8:27b. Latest app versions in both cases and 4-bit MLX models, tested using DEVONthink (Data > Tags > Add chat suggestions to documents).
And oMLX does indeed not support loading of model infos:
CHAT: ERROR: Loading of model info failed: Invalid HTTP status 404
Thanks for testing. Very interesting. My hope was to get beyond Ollama by using MTP with a draft model. Maybe Ollama is doing this out of the box and I should give it a chance again. I’ve ditched Ollama because the models are not interchangeable with different inference engines. Due to slow internet it’s not easy for me to try all those fat downloads.
Have oMLX running and model loaded. Then Open DT.
I’m currently doing this with MTPLX but also have run oMLX (but not yet with DT).
Nonetheless, this is a DT finding the model, not an issue at the model server side.
You don’t need to select the model in the AI configuration? And do you have multiple Models (>30) available in oMLX? With only 3 models on your disk, my issue wouldn’t be a problem.
I’ve not tried so many models at once, and as you say, with only handful at once (currently 5) it’s not a problem. Is the reason you have so many that you want to test them with DT? I use Pi and oMLX for research and I’ll find I quickly end up narrowing down to a shorter list of models if I think about testing new ones. Say one thought, I don’t know how you are downloading the models, but I use LM Studio to determine if I want something new, then download there. I point oMLX to the LM Studio location as well as its own. You could probably easily segregate your models, without losing any, while you test a subset of the 30.
