Index 150gb disc

consiglieri · September 16, 2021, 3:06pm

Hi
I have an external hard drive that contains approx 150gb of documents. I’d like to have a DT database for that drive which indexes (not importing) the files. The drive will be “offline” for the most part but I would like to be able to check if there is a file there I might need hence the index. My questions are thus

Is 150gb too much to index? (assuming the index will be significantly smaller in size)
Will search work well even if the drive is not connected to my machine?

BLUEFROG · September 16, 2021, 3:35pm

Is all 150GB necessary to index, remembering DEVONthink isn’t a Finder or Spotlight replacement.
- If so, is it 150GB of text files or video, or a mix of the two?
Yes, search would still work as the index is internal to the database. However, you will only see small thumbnails and the path of the found files.

consiglieri · September 16, 2021, 5:46pm

Ok, I see. It’s mostly word processing and pdf content, although I could prune away 50-60gb that I know are un interesting. I guess I should have realized that indexing does take some space given that all the text is indexed. There are a bit of images also

Just seeing the path is fine as long as I can search on content.

I’ll give it a try anyway. Thanks for the input.

BLUEFROG · September 16, 2021, 6:57pm

You’re very welcome and sorry if I made it seem like DEVONthink couldn’t handle that amount of data. It surely can!

Seeing as how easy it is to fill a database with extraneous things - and oh yeah, I am as guilty of that as anyone! - I wrote this blog post that may be of interest to you…

I hope it’s useful to you.
Cheers!

rfog · September 16, 2021, 7:55pm

It can. I have about 470 GB indexed, but in about 14 different databases. Biggest has about 5K files, 350 millions of words, 3.8M of unique. Performance in a MacBook Air M1 is really good (a global search takes not more than 2-3 seconds). Good in my iMac and MPB Intel.

BLUEFROG · September 16, 2021, 8:01pm

How many of the 14 do you keep open?

rfog · September 17, 2021, 6:50am

All of them.

Almacén (Storage), Correo (email), Feeds (RSS feeds) are non-indexed. Memory footprint is not too big even when searching:

Summary:

[No sensible data in the captures. RFOG verified ]

Ah, forgot to say, my Macs are modern: iMac i7 4.2GHz from 2017, MacBook Pro i7 from 2020 and Macbook Air M1, all SSD-only 1TB disks and 16 GB RAM at least (iMac has 24).