Best hardware solution to improve performance?

Hi

I’m running DT2 with a 3Gb, and growing, database on a 2008 Macbook Pro with 4Gb of ram and a 2/3 full 320 Gb hard drive. Performance is generally good, but I get occasional spinning wheels and searching isn’t as snappy as I’d like.

Any thoughts on what the best bang for the buck in terms of a hardware upgrade might be? I can take the RAM up to 6Gb, replace the HD with an SSD (I can trim a lot of the files I currently have cluttering up the laptop HD), or replace the HD with a faster (7200) HD.

Any suggestions greatly appreciated.

Kevin

My advice would be: go for the SSD.

Within the last weeks I did three speed-updates:

  1. new MacBook Pro
  2. Snow Leopard
  3. SSD
    The SSD was the one with the most noticeable speed improvement. The bottleneck of most operations (at least in my workflow) has been the hard disk. An additional benefit of the SSD is, that it reduces the noise of the machine (a 7200 HD will be much noisier).

My databases are far from 3 GB. So I don’t know, wether more RAM would be more important in your case. Bill will be able to advice on that point. Anyway: more RAM is always good :wink:

Johannes

Johannes

Thanks for the real world input :slight_smile:

I’m also on Snow Leopard, and although I’m very happy with it - well worth the 29 dollars ! - the speed improvements were marginal.

Kevin

I recently purchased a new 13 inch mac book pro. I debated back and forth over more RAM, a larger HD, or an SSD.

Ultimately I took the standard 4 GB of RAM and sprang for the 256 GB SSD.

I am so incredibly pleased with my choice. The laptop is unbelievably quiet, cool, and the battery life is fantastic. I don’t know how much the SSD contributes to the low temperature, but I suspect it must help with battery life.

I thought I would not like the glossy screen, but glare has not been an issue.

mpm

I went for the SSD, and although expensive, it was worth it. My Macbook is now much more responsive - searches and moving from one document to another in a results list are all very fast. Before the SSD, each search, and moving from one document to the next in the preview window could each take several seconds; now they’re pretty much instantaneous.

Thanks for the suggestions.

Kevin

I’ve been considering this avenue. How much was the SSD you purchased?

128Gb are 350-400 dollars, 256Gb for twice that.

I’ve got a 256 GB SSD, for which I paid about $600. It certainly speeded up everything on my ModBook, with Core 2 Duo 2.2 GHz CPU and 4 GB RAM.

Ultimately, however, as a database grows and/or you are hammering it a lot, the limiting factor on speed will be RAM. I watch to keep some free physical RAM available, and quit/relaunch DTPO2 (to release some RAM) or restart the computer (the latter to clear Virtual Memory swap files) when free RAM gets low. Apple’s Virtual Memory is wonderful, as it will allow memory-intensive procedures to go to completion even when RAM resources have been exhausted. However, VM does this by swapping data back and forth between RAM and disk swap files. A hard drive is orders of magnitude slower than RAM in read/write operations, so the dreaded spinning ball is likely to appear with heavy VM use. My SSD handles this more gracefully than does a slow 5400 RPM HDD, but still isn’t as fast as physical RAM.

So my next computer will come with 8 GB RAM, expandable to 16 GB.

One of the tricks to keep databases responsive is to split out essentially unrelated items into separate databases. Although my main database – which focuses on environmental science, technology and policy – covers a multitude of scientific, legal and policy matters and is certainly wide-ranging, there’s no need to burden it with other content such as my banking, investment and tax data. I find it easy to break out such materials that reflect different interests and needs into separate databases. Altogether, I’m managing more than 150,000 documents among various databases. But by using topical designs as noted, most of my searches take 50 milliseconds or less, and See Also suggestions pop up almost instantaneously.

So why go for a computer with more RAM? My databases are constantly growing, and I want to maintain headroom for that continued growth so that I don’t start seeing spinning balls. :slight_smile:

I purchased my laptop online at www.expercom.com. They charged $649 for the 256 GB SSD. It is apparently made by NVidia (so says System Profiler). The SSD from Apple would have cost $783. Additional savings from no sales tax. I have purchased a variety of mac hardware products from expercom and their prices and service have been great.

I still got the original 256 GB hard drive that the laptop came with from Apple, which I put into a cheap enclosure and use for backups.

I have a Mac Pro with 12GB Ram, my database is 7GB. But i don’t think DTP keeps the whole DB in the RAM - or am i mistaken?
Because if it doesn’t, isn’t your argument kind of moot?

You are correct. The actual document files that are stored within the database package file are not loaded into memory unless and until they are called by an Open command (or some other command that involves pulling the file from disk into RAM).

But that doesn’t make my comments about RAM moot.

The artificial intelligence features that make DEVONthink distinctive are built into the core of the database. When you drop a new file into the database, its text content is analyzed and merged into the database Concordance. That’s rather like indexing text content in ‘ordinary’ databases. Now DEVONthink ‘knows’ that a term used in a search will be included in that newly added file, for example.

But DEVONthink can do some things that ‘ordinary’ databases can’t do, and they involve a lot of ‘artificial thinking’ and use memory.

Now that you’ve added a new document into your database you can select it, click on the Magic Hat icon and let DEVONthink suggest where it might appropriately fit into your database group structure. That’s the ‘Classify’ AI routine.

Classify looks at the text content characteristics within each group in your database and compares the text content of your new document with those, in order to suggest one or more groups into which you might place that new document.

Classify doesn’t merely look at the occurrences and frequencies of use of words in the new document and in the content of each group, but at their contextual relationships, i.e., patterns of usage and occurrence.

Assuming that you started the organization of content into groups whose content has real (non-random) kinds of similar relationships that are relatively distinctive for each group, Classify will become more and more effective as your database grows in size.

That is, in a very literal sense, mind-boggling. The human brain isn’t ‘built’ to do that sort of thing very well, especially when your database contains hundreds or even thousands of groups. DEVONthink and your Mac can do that almost instantly — but there’s a lot of data being processed in RAM.

Notice that when you pressed the Magic Hat button the side panel also contains a See Also list, suggesting other documents in the database (regardless of their group organization) that may be similar to the new document you are viewing.

This is even more mind-boggling. That list of suggestions involved an analysis of the contextual usage of word in your new document and a comparison to the contextual usage of words in each and every one of the tens of thousands of documents in the database.

I can’t do that. I’m probably aware of some other documents in my database that I know are related in content (terms, concepts) to the one I just added. But I can’t hold in my head the content of all of the many thousands of documents (comparing each to the one I’m viewing), and it’s likely that there are some relationships that I haven’t thought about.

That last point is what often excites me about See Also suggestions when I’m exploring my database for ideas; I really appreciate a tip that there is a relationship of words/ideas (ideas can be represented as patterns of words) that I hadn’t thought of previously.

Perspectives for using Classify and See Also:

You are the human being, and DEVONthink is a tool you are using to help you mine and analyze the information content in a collection of documents. You actually have enormously more computing power than does your Mac, but the Mac and DEVONthink can handle some tasks that would be difficult or time-consuming if you tried to do them yourself.

DEVONthink doesn’t know anything about your kinds of knowledge and expertise, such as chemistry, genetics or writing novels.

You are responsible to evaluate and judge the utility to you of DEVONthink’s Classify and See Also suggestions. Some of them should be dismissed immediately as useless, on the basis of your knowledge and expertise. Others are useful, and once in a while See Also can help one get a startling new insight about something.

Often enough, those suggestions save me time and effort (thinking is really hard, sometimes). On that basis, I consider DEVONthink the best research assistant I’ve ever had. :slight_smile:

Give DEVONthink enough free RAM to do all the kinds of data processing you ask of it, and it remains quick and responsive. Run out of free RAM, and you move into Virtual Memory use and start seeing spinning balls, which I find irritating.

If your Mac can hold more RAM, adding RAM is pretty cheap as a way to avoid that spinning ball. Another way, especially if your database has grown and your RAM is maxed out, is to spit databases topically or historically so as to keep your processing needs within the limits of free RAM. There have been a number of discussion threads on spitting databases. As DEVONthink Pro 2 and DEVONthink Pro Office 2 allow multiple open databases, you can always assemble a set of open databases like informational Lego blocks, when necessary. (Although currently, in public beta 7, See Also and Classify cannot work across databases, that will come in the future, per Christian’s recent post.)

Bill, thanks for the short reply :wink:

About the moot-point, you are right. I wish DT would provide better Search features. Ie. mark a sentence in a document and go for ‘search’ and i get a direct list of all the documents, but instead of the documents i see only what i need from the document, maybe even summarise. To keep such a feature quick, you would probably be forced to keep most documents in the RAM.

I don’t want to get too off-topic, since this is about hardware and not the AI.
But what i found to be somewhat bad about the classification is that it can’t filter the context properly. I have a lot of papers and archives containing words like Springer Verlag, Twitter, IEEE etc. Those hardly ever are relevant (at all). But since they are mentioned on every page of every paper at least 2-3 times (if they come from ieee or springer), too many papers are connected, according to DT.
And the other problem is the category. I keep all my recipes in a folder, problem is, in my opinion a single recipe should be in at least 5-10 folders (ingredients, type, origin, country,…). At the moment i find it difficult to retrieve data. What if i need a recipe with quinoa. If i search for it, i will get documents which are not recipes but contain quinoa (ie. nutritional documents, wiki entries etc). Same goes for the “See Also” feature.

Back to RAM. I have 12GB, but only 10 are in use. I got 2GB completely unused/free. Wondering why DT isn’t jumping at that and using it? But i never have a spinning ball either.

Please encourage him to post these gems in KB articles, blog posts, or anywhere they’re more easily discoverably/accessible to everyone than buried (often deeply) in forum posts. :exclamation:

Uhm, it isn’t really up to me whether or not he is allowed to post on the blog or publishes in the KB?

Nope, it’s totally Bill’s choice what and where he posts. :slight_smile:

Maybe you misunderstood “encourage”? In my reply can mean “help inspire”.

I think it would be a lot easier to check somewhere else for some of his informatively detailed writing about different topics instead of watching for them to “randomly” appear in forum threads. Also seems unfortunate not to be sharing more of that outside the forum community where other people might learn and benefit from it.

As a 32-bit application, DT can only address 4 GB of RAM. Once it has that much, any additional RAM in your system is moot.

Katherine

Katherine is correct. A 32-bit application can’t address more than 4 GB RAM.

4 GB memory space seems adequate for your database now.

Currently, you still have 2 GB free RAM.

But if you were to lose that 2 GB of free RAM by running out of physical RAM (because other applications have gobbled it up), your computer would move into the Virtual Memory mode and you would begin to see DEVONthink affected by slowdowns, as well. Virtual Memory, to make room for data to be processed, would ‘grab’ data in RAM that’s not currently being used by DEVONthink and send it to a VM swap file on disk, in order to make room in RAM for something else. Next time DEVONthink calls for that data in a procedure, it will have to be read from disk and written back to RAM. That will slow things down a bit.

All things considered, the more RAM, the better. :slight_smile:

At some point the DEVONthink applications will be switched to 64-bit mode. But given the current users and their existing Macs and versions of OS X, it will be necessary for some time to come to allow the option of 32-bit mode.

Apple has turned on 64-bit mode for many of the applications that are installed with Snow Leopard, such as the Finder, Mail, etc. But for some Intel processors such as the Core Duo CPUs, applications such as Mail still run in 32-bit mode under Snow Leopard.

64-bit mode doesn’t necessarily mean faster operation than in 32-bit mode, but will have an increasing impact in the future because of the ability of applications to address much more RAM space. As the size of RAM chips continues to grow and they become cheaper – this will certainly happen – computers in the not distant future will be able to handle much larger amounts of RAM memory than do today’s consumer computers.

At the same time there will be a shift from the current hard drives to solid state drives, which also have read/write speed advantages.

Those technology advances will be welcome to those of us who hammer away at increasingly large databases, while expecting instant responsiveness. :slight_smile:

Bill, forgot about 32-bit :confused:

Do you have a ratio handy for database size (when mostly readable text) vs. ram consumption?

No – too many variables. But Christian has posted some upper limits that are pretty huge, more than most people’s needs.

Which makes a good case for making the forums into a DTPO database that we could access and “classify” and “see also” and…and, well, just make searching so much easier. We could tag Bill’s “white papers” and so forth.