DT3 large database performance versus new hardware choices

Hello all

I wonder what Mac configurations users recommend who work daily with large databases.

To explain – my main research database (8 thousand items; 4.3 million unique words; 121 million total) is starting to lag in places though it is nowhere near the ‘comfortable limit’ for a large DB I’ve seen cited in several places on this forum. Particularly when I fire up DT3 and right click a file (e.g. for moving) the beachball always spins for 5/10 seconds (this doesn’t persist, it’s a one-time thing upon opening); full text search is slowing; and I’ve had some hang-up with more complex searches.

I work on a 2013 MBP (2.4 Ghz i5; 16GB ram; 256 SSD) that is otherwise still in good service but needs replacing because of battery swelling and impending failure. (My employer favours replace over repair, but is no longer happy to fork out for a new MBP).

My options are a beefed up Mini (quad core i7; 16GB RAM, 512 SSD); or a Macbook Air (16GB RAM, 512 SSD). I favour portability, but also like power. I may yet bite the bullet and get my own MBP, but worry this is just overkill for most of the daily needs of an academic in the humanities.

Is DTs performance footprint large enough for this to matter on new Macs or will any new modern machine with 16GB of RAM see me happily into the future for the next 3-5 years, with several databases nearing 200 million words?

Thanks folks.

You have several databases approaching 200,000,000 words simultaneously open?

And as far as hardware goes, the more resources the better, including RAM.

No - the largest one runs to 121 million and growing; other ones running alongside, say, 5 million; 8 millions; or smaller.

But yes the more resources the better, that does make sense.

In case of very large databases RAM is more important than e.g. the number of CPU cores.

Thank you – that’s where I’ll invest most then, it’s good to know.

How large, in gigabytes, is this main database of yours?

About 40 GB - a portion is taken up with scans of old books that aren’t OCRd (so image only) but the majority of this is PDFs or other types of text.

thanks-- just trying to compare this with my own experiences… Am I right in assumming this database is stored on your laptop’s SSD?

My largest database is 22.GB, 518 thousand unique words, 15.3 million total. It’s largely populated with scans of antique magazines,

Devonthink uses under a gigabyte, so I’m nowhere near stressing it. The built in image viewer is slow compared to preview.

Yes that is right local storage on SSD. It sounds like yours doesn’t tax your system too much and so large DBs perhaps not an issue.

I wonder about having to try a clean install of DT3 as I am having some regular problems where the expanded search window (options below the bar) stalls and the app crashes.

Out of interest what machine do you run on? I was wondering if a Macbook Air would do me for the next few years. Bluefrog’s “the more the better” obviously right but I am curious about minimal requirements and some future proofing since upgrading Macs so difficult these days!

2014 5k imac. External SSD. 24 GB Ram, 3.5 GHz i5 (haswell).

My sister has a 4 core mac mini (2018)., It is, somehat surprisngly, quite a bit faster,

Your current MBP seems like it should have enough horsepower. What you described sounds very similar to what happens when you don’t have enough RAM and the OS needs to cache to the drive. With a SSD, this is hardly perceptible - unless your SSD is full or almost full. If your drive is full or near full (less than 10% free space), this could be the culprit of a slow responding system.

Do you happen to be running any new RAM-hungry software, or running a large amount of apps at once that would use up your RAM? Is your SSD running out of space? How many apps are running when you open DEVONthink?

If your drive isn’t full, I would think you could test this by rebooting the computer and not allowing any apps to open to their previous state - IOW, a fresh reboot. Then, open DT check its performance.

I know this is off-topic, but can I ask which processor the new Mac Mini has? I have a 2014 Mac Mini, which only has a dual-core, and I hate it. I’m trying to decide between a new Mini or a used iMac. I’d appreciate and input on how well that Mini runs.

The mac mini can be configured with
Core i3 (I3-8100B) 3.6 Ghz, 4 cores, doesn’t turbo
Core i5 (I5-8500B) 3.0 Ghz, 6 cores, turbos to 4.1 Ghz
Core i7 (I7-8700B) 3.,2 Ghz, 6 cores, 12 threads, turbos to 4.8 Ghz

Sorry, I meant which cpu your sister’s Mini has. I’m looking for real,world observations on performance. I can’t stand the low performance of the 2014 Mac Minis.

She has the 8100B-- 4 cores. It may be that the “benchmark” we used-- the tesseract OCR program, takes fuller advantage of Intel’s latest designs than a typical program.

1 Like

Thanks! I have about 30GB free in 256SSD, so that’s approaching the 10% rule. I’ll try and clear some space and see what difference that makes.

I typically run a few comms apps in one space (Airmail, Fantastical, ToDoist); then in my ‘writing space’ across two monitors run DT3 (2 databases open), perhaps a few PDFs in Preview (can be image heavy), Scrivener for any long-form writing, perhaps a Word doc open, Zotero for my ref manager, and often a browser with 3/4 tabs open. I’m not sure what there counts as RAM hungry but I’ll start to monitor more closely.

IIRC, with a Mac Laptop, when it goes to sleep it caches everything in RAM to the drive. With 16GB of RAM, your sleepimage is 16GB. I don’t remember if the sleepimage is counted as free space or not, but the more the drive gets full, the less space the system has to operate with. Also, I think at some point the SSD needs a certain amount of free space in order to do its internal maintenance. I’m sure someone here can check me on that.

You can use Activity Monitor or the iStat app (https://bjango.com/mac/istatmenus) to check how much RAM is being used by each app and how much is being written out to the swap files on the drive.

Battery swelling is DANGEROUS – please recycle your laptop and get new one before the battery could explode and cause damage to you and your surroundings.

Thanks – I read some alarming things on the interwebs about this though the Genius bar tech I talked to was less concerned and even said there’s some built-in tolerance for this sort of thing. Not a health risk unless I start prying at it with a screwdriver, which I won’t, obvs. Anyway replacement machine now imminent so this can be sorted out – recyling or repair, indeed.

I’m running into the same problem and am really not sure what to do. I used to have one giant database of approximately 200gb. Recently I split it into three: one of 130gb and two of 40gb. These databases are pdfs, almost all of which are OCR’d (except older fraktur materials that I can’t ocr). I would really prefer to have all four open, plus my ‘Zettelkasten’ at all times.

Will it make performance run more smoothly if I were to further split up my 130gb research database, even if I am going to keep all of my databases open at the same time?

I have a 2017 MB Pro 3.1 GHz Quad-Core Intel Core i7 with 16gb of memory.