Hi everyone. My current computer is a Mac mini (Late 2014) with a 3GHz Dual-Core i7, 16GB RAM and 250 SSD. I have a large database and it has become so slow it’s unusable. Search is slow, adding and dragging items is slow, etc. Because of these performance issues I have stopped using the database unless absolutely necessary. The database is about 7GB on disk. Below is a screenshot of its properties:
I was planning to buy the M1 Max MacBook Pro 14" with 64GB RAM which costs over €5000. I can’t afford this anymore so now I’m considering the new M2 MacBook Air with 24GB RAM and 1TB SSD which will cost about €2500.
Fast interaction/performance is crucial for me to be productive. Do you think this new M2 MacBook Air with 24GB RAM will be enough to achieve the best performance with my database? I always have multiple apps open, including multiple browsers with many tabs open in each one. For example, right now Activity Monitor shows 12.49 GB used memory and 4.76 GB of used Swap.
my experience running DT on an M1 Macbook Pro with 16GB RAM has been absolutely flawless.
DT is extremely snappy, no lags whatsoever. Also, my database is significantly larger than yours… I don’t think you need to worry about performance issues esp. with 24GB RAM.
My sentiments exactly; I routinely have 6 databases with a combined size of over 25 GB and something like 20k items loaded - fluid at all times. However, I second @chrillek‘s question…: why on Earth have you got 340k groups?
And most likely the number of groups is one of the reasons for the performance, the aging computer of course another one. Especially the RAM seems to be major bottleneck currently, therefore the more you can afford for the new machine the better (and that’s more important than M1 vs. M2)
You probably noticed that I use DT to organize webpages. I used to just save the bookmark (link to the webpage) but then I started to save the webpages as rich text documents.
There are more groups because I use groups to categorize everything inside a hierarchical tree structure (every category or topic is a group). I also put each webpage item (bookmark or rich text) in its own group with the same name because sometimes I add notes about the webpage inside its group, and I put these notes in the name fields of more groups .
I did not think that groups could impact performance so much since I thought they are probably the smallest/lightest type of item in DT. I can try to lower the number of groups in the future by only putting a webpage inside its own group when necessary. Still, the number of groups will keep increasing because this is how everything gets categorized and organized.
RAM seems to be the most important for performance, but I’m not sure if 24GB would be enough. The new M2 MacBook Air does not support more than 24GB RAM. The next option would be the M1 Pro MacBook Pro 14" with 32GB RAM which costs €2960. If we just look at the RAM, that would be 8GB more RAM for about €500 more. Do you think these extra 8GB will make a difference with my database performance? I’m leaning towards the M2, but of course smooth database performance is the number one priority.
Why not capture each webpage as a PDF, then annotate the page instead of using an overly complex system which does not take advantage of DTP capabilities. Use groups for general, high level filing, sub sort into secondary groups and then tag your PDFs. Anything deeper than 3 group levels generally means you should rethink your strategy. Migrating your structure to something less burdensome will yield better perfromance and easier slicing and dicing of your data.
For comparison,I presently run 3 databases averaging 46 -50gb each comprising of around 4000 PDFs, Plaintext and the occaisional .docx files on a 2009 Intel based Mac Mini 8mb ram, 512gb SSD on Sierra. The only real slowdown occurring when I need to OCR large documents.
I appreciate your thoughtful feedback, but I’m not sure it’s suitable for my use case.
My main task is organizing groups/folders into multi-level tree structures, the same way you would organize files with folders in a file system such as the Finder (in list view). With so much information you end up with very deep group levels. Tags are not easy to navigate, structure and edit. I have captured and organized over 330,000 webpages as bookmarks and text files. If they were PDFs the database would be too large I think. I also like to outline/structure my notes which would not be easy by annotating the page.
I’m not doing any DTP. There are other tools like OmniOutliner or WorkFlowy which are more suited for publishing I think. But I don’t think they can handle a huge tree structure like mine smoothly. All I’m interested in is being able to organize information on a large scale and navigate the tree structure quickly with search and navigation tools (expand, collapse, reveal, etc.). DT in list view is the best tool I found for handling such a large deep folder structure. Unfortunately, DT3 introduced some changes that made my workflow more complicated and slow. I plan to contact the developers about these issues soon and hope they will bring back the functionalities I relied on in the previous version.
@cubicray The project sounds interesting. Are you able to share a bit more about the nature of the project and/or a sample of the kind of data you are collecting? Or is that proprietary/private info?
I run all my web captured PDF’s through PDF Squeezer and they are not much larger after compression. Its an automated workflow using Hazel so I never have to think about it.
The new M2 Air describes the ram as “shared” memory. Back in the day, that used to mean shared between regular ram usage and monitor. If this is the case the 24 gb might mean 16 gb ram. At any rate I’d go with more ram.
I have about 500 GB mostly of PDF across 18 databases (and billions of words), all synchronized between 3 Macs, 2 iPad and one iPhone.
My least Mac today is an iMac 2017, i7 with 24GB RAM and 2 TB non-original SSD disk, and performance is good having in consideration the size of my stuff (it takes ten seconds or so a search). My highest Mac is a MacBook Pro M1 Pro 16", 16 GB RAM + 1 TB disk and performance is exquisite.
I sold a MacBook Air M1 (same configuration but M1 is normal one) and performance was very good.
If I had to select a new M2/M1, depending on my budget, and for my DT configuration, I will select as much RAM as possible over M1 Pro/Max/Ultra vs M2. At DT level, I think performance differences won’t be much between M2 and M1.
It started when I began bookmarking websites inside the browser using the built-in bookmark manager. I remember it was Firefox and at some point they changed their bookmarking system and it messed up my whole workflow. That’s when I started using DT.
The data is not specific. There is all kind of different information that I come across the internet. A lot of it is related to web design and development because that was my main interest for many years.
I’m also curious, is there anything specific that caught your interest? Do you think there could be something valuable besides my own private use?
That is a good question and I also wondered about that before. I plan to reevaluate my workflow once I get the new MacBook and see where it can be improved. I don’t think it’s easy to switch to a whole new system because there is some automation that I would have to convert. Also I think it’s better to keep the database separate from the file system. Nevertheless, using the Finder sounds intriguing and I might look more into it in the future. My priority is speed, not just searching and navigating the database, but also quickly adding items to the database from Safari.
DTP, oh ok, my bad, that makes more sense now haha.
PDF Squeezer is great. And I’ve been planning to try Hazel for a long time. That’s a smart workflow you set up. Makes me think about how I can use Hazel to improve my own workflow.
.