Training local models

DEVONthink
1

Hi,

I just testing the new functionality with local AI Models via Ollama and LM Studio. So far the models can just summarize a PDF but weren’t able to answer me a single question about the particular document. Do I miss some settings beside the AI pane? Do I need to train the models?

My usecase is: I have buckets (folders) of distinct cases which are a collection of documents belonging to one person. I want to be able to ask questions about that person like: When is xyz born, which health insurance was he, etc.

2

I suspect you are expecting more from a local LLM model than is practical at present.

3

Which model did you use?

4

I used llama3.2 and mistral via Ollama and deepseek via LMStudio.

5

Apparrently :slight_smile: Do paid models will meet my expectations?

6

I would recommend Mistral Small 3.1 or Gemma 3, ideally with 24b parameters if that’s in the range of your hardware.

1 Like
7

The mistral model gave me these instructions after I asked for a summary:

To summarize the document in DEVONthink, follow these steps:

  1. Retrieve Text Content: Use the “get_contents” function to extract the text from your PDF document.
  2. Manual or Automated Summarization: Once you have the text content:
  • Manually read and summarize the content.
  • If possible, use another tool within DEVONthink (if available) to automate the summarization process.
  1. Considerations:
  • Ensure that the “get_contents” function is compatible with PDFs in your version of DEVONthink.
  • Be aware of any limitations regarding file types and built-in functions provided by DEVONthink.

By following these steps, you can effectively summarize your PDF document using the available tools within DEVONthink.

What does that mean?

8

Which Mistral model? There are tons of Mistral models available.

Basically that the model you’re using seems to be completely confused. Or doesn’t support tool calls.

9

Which mistral model

ollama list
NAME ID SIZE MODIFIED
mistral:latest f974a74358d6 4.1 GB About an hour ago

10

Sounds like this one which is a rather old and limited model, its support for tool calls (see above) is obviously very unreliable:

This one is a lot better but requires more RAM and a faster computer:

11

Hi, I just upgraded to Devonthink 4 (v4.0.2), and I’m having the same problem. I’m using the latest LM Studio beta (0.3.21 Build 1), and I’ve tried both mlx-community/Qwen3-30B-A3B-Instruct-2507-8bit and google/gemma-3-27b. Both are supposed to be enabled for tool use.

With Qwen3, when I selected a web archive and asked, “Can you please summarize the selected item for me?”, it replied:

I’ll summarize the selected item for you. Let me retrieve its contents first.

<tool_call>
{“name”: “get_contents”, “arguments”: {}}
</tool_call>

Then nothing happened.

With Gemma3, the response is:

I can certainly help with that! To provide a summary, I need to access the contents of the selected web archive.

[TOOL_REQUEST]
{“name”: “get_contents”, “arguments”: {}}
[END_TOOL_REQUEST]

Then that’s it.

12

By the way, I just tried my Google Gemini API key, and that worked great.

13

The latest LM Studio versions seem to have major issues related to tool calls and are unable to parse the model’s response and therefore forward the response unfiltered. This issue might be related:

1 Like