DEVONthink and DeepSeek?

Curious if anyone has figured out a way to use a DT database with DeepSeek (or any other AI tool) offline? I feel this may be the moment I’ve been waiting for.

  • Why?
  • What kind of Mac do you have?
  • How much RAM?
  • What do you imagine would happen?
1 Like

The key word in that question is “offline.” There are lots of ways to use an LLM locally, but they mostly perform much more poorly than their online cousins.

Hi @BLUEFROG , I have decades of content, my content, in DEVONthink. I’d enjoy building my own LLM with my data, in a secure, offline way, and explore what I am unable to see on my own. The fact that DeepSeek is among a few solutions that would allow me to build my own LLM, offline, is quite appealing. I’m surprised how many articles have come out in just the last 24 hours about how to do such a thing, even with a Mac laptop. Take a look.

See Use DeepSeek R1 via API in Swift - NatashaTheRobot

Although DeepSeek R1 is open source and available on HuggingFace, at 685 billion parameters, it requires more than 400GB of storage!! So the answer is no, you cannot run it locally on your MacBook. Note that there are other smaller (distilled) DeepSeek models that you will find on Ollama, for example, which are only 4.5GB, and could be run locally, but these are NOT the same ones as the main 685B parameter model which is comparable to OpenAI’s o1 model.

1 Like

I have a 2 TB drive. My MacBook Pro is primarily for DT.

Just try it then :wink: The hard part is getting a local setup that works to your expectation. Integration with other apps is relatively more trivial. Be aware, though, that if you run a LLM locally on your MacBook, it would not qualify as “primarily for DT” anymore. Especially if you don’t have enough RAM to host the monster and still have spares.

I would not use that article as a template to work from. While exciting in some ways, you should manage your expectations. Look much deeper into it and you’ll come to find it’s less feasible than you think it is. An 8 billion parameter model is small. And if you’ve got a 16GB Mac, you’re not going to be running much beyond that performantly. Having a 2TB hard drive is not the critical factor.

Yes, DeepSeek is very interesting but it’s not the be-all, end-all for every purpose. In fact, it’s can be wrong just as easily as any other LLM.

Also, it’s certainly not going to ingest “decades of content in DEVONthink” and let you explore it. Even commercial LLMs wouldn’t let you do that, at least (1) not in any timely fashion and (2) also prohibitively expensive.

Are you familiar with the Spring-Heeled Jack ?

1 Like

AI integration with DT (either locally or by secure cloud) → Great idea

DeepSeek integration with DT → Not for me. Risk of data breach seems much too high. There are numerous options which perform better than DeepSeek and have well-established privacy policies.

You’d have to clarify what you mean by “perform”. :wink: DeepSeek isn’t shaking things up for no reason. It is 100% something to contend with for the rest of the industry. However, it’s also not a panacea any more than Claude or Gemini are.

1 Like

DeepSeek is “shaking things up” because it is priced cheaply - likely subsidized by their government. I haven’t seen any reviews yet confirming any specific performance claims - and I certainly would not take their word for it. This is a country that creates knock-offs of entire Apple stores where even the employees don’t realize they are fake.

1 Like

That’s definitely not the cause of the fuss and interest is not just in the business sector. Universities, research groups, AI think tanks, etc. are all abuzz with what DeepSeek has done. NVIDIA lost ~$600 billion dollars in valuation overnight, the largest in US history because DeepSeek is reported to be working/training/tuning on older, far less expensive hardware. It also has unusual results in format and approaches responses differently with a chain of thought model similar to having to “show your work” in school. It has an unusually “thoughtful” <think> element where it goes through what appears to be a logical process.

And no, I am not jumping on anyone’s bandwagon and yes, the country of origin has a history of flagrantly misleading or false information. So grain of salt is included. But if you haven’t tried it, it’s interesting, even if still ultimately as flawed as other models.

Watch Mike Pound discuss it on YT:

(Bad thumbnail; great channel and educator)

3 Likes

Also note that the hype appeared out of nowhere just before the Chinese New Year, which is highly unusual and strongly suggests a political cause (贺年礼, a new year’s gift for the political leadership) of the Chinese company behind DeepSeek. It made flagrant claims and then quickly moved to restrict new user registration and add usage limits — marketing shit which is nothing abnormal in China but might surprise westerners.

I have no respect at all to the influencers cashing in on the hype with a bare minimum understanding of DeepSeek or the company behind it. I’d rather wait a couple of weeks for the industry to clear things up.

1 Like

Yes I tried it - aside from the censorship it seemed more simplistic than ChatGPT.

OpenAI Discourse has a good bit of discussion about it. Apparently it does as claimed on the specific metrics the company quotes but speed is an issue in realistic use and there are concerns raised by some about its ability to perform other complex tasks. In other words - it is possible it was “fed the answers” to the benchmark tests. Imagine that - who would think it?

Anyway - my point is simply that there is no reason to consider DeepSeek a “success” unti its abilities are confirmed by independent third parties. And even then - the censorship and data privacy issues are obviously an immense concern.

2 Likes

I’m no fan of nor apologist for any LLM. Just reporting information as I’ve seen it.

Then I would expect you also have grave concerns about how current LLMs got their training data, especially OpenAI and Gemini. In fact, it’s ironic OpenAI and Microsoft are trying to claim intellectual property theft when they’ve scraped unauthorized data for a long time then shrug and say “It’s all fair-use, man! I mean, it’s on the Internet, so it’s public.” :roll_eyes:

6 Likes

Recently it was revealed by a popular Chinese tech influencer that all Chinese smartphone manufacturers do this for SoC benchmarking — they whitelist benchmark apps and use more aggressive performance settings when a benchmark is running. The video has since been deleted from the channel; you can see user discussions here (in Chinese).

So, yes, it’s the kind of …

1 Like

Back on-topic:
@straylor things aren’t anywhere near where you hope they are regarding handling your decades of content. This is especially true when trying to run offline, local LLMs. However, DEVONthink’s built-in AI still functions amazing well for many purposes. :slight_smile:

5 Likes

Agreed - but they didn’t hide it or lie about it. Meta has said they do not believe LLMs as we know it would even be possible without using copyrighted data for training.

More pertinent to my point though - OpenAI, Google, Anthropic etc. have clear policies about when user data may be used for model training and how to opt out. I am not confident there is any way to truly opt out of DeepSeek data collection.

I tried the 70b version locally on my Mac Studio using Ollama. The performance was better than expected, especially for a reasoning model and a model of this size.

And contrary to the DeepSeek.app which I didn’t try I could easily get responses to questions which are supposed to be censored (like the above one). Not sure whether that’s due to the smaller model or due to additional censorship by the DeepSeek.app. Especially typos are useful :slight_smile:

But nonetheless the more powerful local models are just way too slow, their context window is very limited, tool and/or vision support is usually not existent and they require lots of disk space & especially RAM.

Smaller local models are of course somewhat faster but even more restricted, e.g. support for multiple languages, and more likely to hallucinate. Like Apple Intelligence for example - limited, relatively fast, English only, mixed results.

Apart from running it locally using e.g. LM Studio or Ollama on a powerful Mac you could also use services like replicate.com which host open-source models.

2 Likes

But is it only an LLM? I was under the impression that they are “reasoning” (for some meaning of that term). Are they also only stringing words together without “understanding” what those mean?