Pro's/Con's of Indexing over Database Use

[Ronin]

I currently use Devonthink Secure Databases and don’t have any real issues/complaints with it. The sync works great and I can easily backup my databases. What I am trying to determine is if I would be better off/happier with a different option.

I am considering useing Tresorit to hold all my files and then indexing my local storage of the Tresorit drive with Devonthink. This would allow me to still keep everything in one accessible place and encrypted, while allowing more flexibility. I.E. easy sharing, photo upload from iOS devices, versioning of files, secure document scanning on the go, etc.

As I understand it the indexing feature of Devonthink is quite robust and I could still use Devonthink to actually manage these files and maintain my workflows in virtually the same manner.

Can anyone point out what friction points I might find or issues I would run in to? This seems to be the “best of both worlds” for cloud storage and security.

Thank you!

[rkaplan]

There are way more potential gotchas with indexing.

Indexing is a great way to get documents into DT3 in an automated fashion. It may be helpful in selected, very deliberate situations for long-term use indexing specific local files. Indexing your entire database, particularly coordinated with some sort of backup to cloud, can result in unanticipated issues syncing or maintaining the database.

In short, I would suggest using indexing sparingly and very deliberately in specific situations when there is no alternative. I would discourage an entire DT3 database based on indexing when you have an alternative which you in fact report works well.

[Ronin]

yea. I have made an extremely small database here to test it out. Devonthink has crashed a few times and the overall experience isn’t great. Thanks for your input. Confirms it wasn’t all user error.

[Altostratus]

From my experience there were issues with indexed files stored on external media, where starting a search in DT resulted in a spinning wheel while sleeping drives were being spun. This was very annoying, especially in my case with a 4-drive RAID. I don’t know if this has been fixed since then, as I’ve removed all indexed files from my DT database and imported them properly. If you’re indexing because you’re concerned about database size getting too large, I’m happily running several databases of around 20 gigabytes and 10k items each. Others have probably pushed this much further. Selective sync to iOS makes this perfectly useable.

Just my 2c.

[FROBGOBLIN]

I keep a lot of my files in Dropbox, the DEVONthink database on my local drive, index everything, and also use iOS with it. I’ve had no problems with databases up to 60GB. It’s a wonderful way to use DT when you want to freely access the files using different apps.

However, I’ve been indexing for years, and I’ve learned a lot from unpleasant, unexpected experiences. User error has been the cause of the problems, but indexing is inherently tricky, especially on multiple devices with both OSX and iOS. I recommend started slowly and experimenting before trying anything new.

[BLUEFROG]

Devonthink has crashed a few times

Have these crashes been reported to our support system?

PS: Regarding indexing, @Greg_Jones is the forum’s real-world authority in indexing, as he has used indexing (almost, if not completely) exclusively for years. I’d poke around the forums for his posts on the subject.

[Ronin]

No.

I guess they had something to do with indexing a folder that also syncs to the cloud? Unsure.

[Greg_Jones]

@Ronin

I guess they had something to do with indexing a folder that also syncs to the cloud? Unsure.

Indexing a local folder that also syncs to the cloud, such as Dropbox and iCloud Drive, has never been a problem for me. In my experience it’s when the indexed folder is not local (only on a cloud service) and/or the database is on a cloud service that problems can arise. I also expect that local drives that are not mounted, as mentioned earlier in this thread, can be problematic but I’ve not tested that configuration myself.

[FROBGOBLIN]

As mentioned before, Greg is the local (pun intended?) authority on indexing here (usually, after I have run into issues in the past, I’ve come across a post by Greg or get a response that sorts me out), and if I understand him correctly this time, indexing a folder on your computer that is also syncing with Dropbox (for example) is no problem.

My experience has been the same—no issues at all with mixing indexing and data synced to Dropbox.

There are some interesting conundrums you can create for yourself this way, though. For example, let’s say that two databases index the same file, but you delete the file in one database. That’s OK, surprisingly, as long as you have both databases open at the same time (I haven’t tried it yet with one closed and one open, but that seems like a bad idea).

But, let’s say that you fiddle with a file using another app on iOS (maybe a text editor synced with a file in Dropbox), then mess around with the file in DTGO on iOS (remember, it is indexed, but on the computer), and then mess around with the file on OSX with DT. I’ve generally found this to be a stupid thing to do, with unexpected results, (apparently) depending on settings dealing with sync conflicts and the quality of the apps that get enmeshed in this sync nightmare (lots of sync conflicts in the text editor I used to use on iOS before DT got markdown support).

How about a local folder not synced to Dropbox, indexed into a database in DT, synced through Dropbox, and then opened on another computer? Interestingly, the indexed folder (which originally only existed on the first computer) is indexed and created on the second computer. Wow. Am I understanding what happened correctly? You are now “indexing” the same folder that exists simultaneously on two different computers that are only synced through the DT app—a kind of double indexing. Obviously, a recipe for trouble if you are not careful (for example, mucking about with the files on one computer while forgetting that DT is syncing them to another one), but also an elegant solution for a user intent on indexing and syncing between two computers (me). Naturally, backups and lots of testing in tiny baby steps is recommended before jumping into a workflow like this.

But, the gist of my message is that indexing can be rather amazing in its ability to accomodate unconventional workflows, and reliable to boot.

[BLUEFROG]

Naturally, backups and lots of testing in tiny baby steps is recommended before jumping into a workflow like this.

Agreed… and not just regarding indexing, but any unfamiliar task. Don’t always assume you know how things work. That’s why we have these forums, the built-in documentation, and our support system.

But yes, with regards to indexing, it is better to be even more thoughtful as the ties to the filesystem are much tighter now.

[padillac]

The main downside to indexing that I’ve found, is that if you move / rename a file, then DEVONthink can’t find it, and the record is essentially broken. This is mostly important if you use DEVONthink for any sort of organizational purposes – grouping, tagging, aliases, or copy DEVONthink permalink.

Indexing works in cases where I’m using DEVONthink exclusively for search purposes, and doing all of my organizing in the Finder. I treat indexed data in DEVONthink as temporary – it might not be there tomorrow.

So if you want a powerful search on top of the files you’ve organized in Finder, indexing is great. If you want to do any sort of organization in DEVONthink, you’ll want to import.

[Greg_Jones]

@padillac

The main downside to indexing that I’ve found, is that if you move / rename a file, then DEVONthink can’t find it, and the record is essentially broken. This is mostly important if you use DEVONthink for any sort of organizational purposes – grouping, tagging, aliases, or copy DEVONthink permalink.

Indexing works in cases where I’m using DEVONthink exclusively for search purposes, and doing all of my organizing in the Finder. I treat indexed data in DEVONthink as temporary – it might not be there tomorrow.

So if you want a powerful search on top of the files you’ve organized in Finder, indexing is great. If you want to do any sort of organization in DEVONthink, you’ll want to import.

Interesting, as over the years I’ve found the opposite to be true. It’s the re-organizing of content in the Finder that the user has indexed that causes records to be broken. For me, DEVONthink has never has a problem finding records that the user has re-organized inside of the app, and I index 100% of the documents in my macOS databases. DEVONthink 3 has even better integration with the Finder, by automatically consolidating/de-consolidating content, where DEVONthink 2 required the user to do same manually or with scripts.

[padillac]

@Greg_Jones

Interesting, as over the years I’ve found the opposite to be true. It’s the re-organizing of content in the Finder that the user has indexed that causes records to be broken.

Yes that’s what I meant. If you’re using Finder for organizing, then the records will break.

If you want to index and not break records, you must never rename or move files outside of DT, ever.

To elaborate:

1. I index a file
2. I tag it in DT
3. I rename it in Finder
4. The DT record is now broken

[cgrunenberg]

@padillac

If you want to index and not break records, you must never rename or move files outside of DT, ever.

As long as filesystem events are supported (e.g. by internal or external discs), renaming should never be an issue and moving inside the same enclosing indexed folder shouldn’t be an issue either. The next maintenance release will fix some issues of the filesystem event handling, e.g. I just tried both operations successfully in the Finder. Not only was the index automatically updated in DEVONthink, the record was definitely still the same (same UUID).

[BLUEFROG]

If you are experiencing that, it is unusual and incorrect behavior.
Here are two independently indexed groups…

In the Finder, the file is moved from One to Two… and this is essentially immediately reflected in DEVONthink.

The file is then renamed in the Finder, and again the change is reflected and the link is unbroken.

[padillac]

I’m guessing this is new in DT3? Because I asked for this behavior years ago, would periodically test it out, and the eventually gave up on it ever happening. Glad to hear that it’s now been implemented

[BLUEFROG]

Glad to hear that it’s now been implemented

Yes, it’s new in DEVONthink 3… but also be mindful of the new ramifications.

Stumped - Delete indexed items without deleting original files?

I don’t think a straightforward response was indicated here, and I now find myself with at least one of the problems indicated here. I have a folder on dropbox and I intended to index one or two files within the folder. However, the entire folder got indexed. The folder contains my entire dissertation and related files and for obvious reasons I want to be very careful here. I don’t however want the entire folder in the db I am working in. The file is big. Am I to understand there is no way to re…

[ZeppoVA]

Since you linked to the thread I just started about understanding this new behavior in DT3 …

I have used indexing across multiple DBs and with multiple entries inside of a single DB for a few years. Knowing about the “breakage” and how DTPO acted when moving or renaming helped me approach the organization, use and maintain the files consciously and pragmatically with no unexpected issues. But - understanding how DTPO dealt with indexed files using Empty Trash is what allowed me to index and delete indexed stuff as needed while maintaining the integrity and security of the external files.

The automatic indexing and name changing etc in DT3 is nice and helpful.

The dilemma I now face is understanding how DT3 behaves and how it affects my existing DBs. DBs migrated from DTPO have both indexed and managed files spread throughput the groups and subgroups, as well as duplicates and replicants of indexed files. And some indexed items are moved within the DB to another indexed group for organization needs.

I’ll be happy to explain the why of that (even though it is more suited to explaining over beer to on the way to fishing hole), but the challenge is knowing what DT3 does to indexed stuff upon Empty Trash - especially with multiple indexed items of the same external file or folder, replicants, and when a file is indexed in more than 1 DB. It seems context sensitive but allows deletion of the source file without confirmation that t is doing so.

Bluefrog – where are the new ramifications explained ? The " file or group in an indexed group " context does not seem to help understand what DT3 does with multiples of an index (duplicates or replicants or indexed in multiple DBs). Fixing broken links is easy but, to be clear – it is the deletion behavior that is most important.

[BLUEFROG]

Some things to consider:

Duplicating an indexed file into a non-indexed group in a database generates an incremented file, e.g. file-1.png, in the Finder location.

Duplicating an indexed file into another indexed group in a database generates a new non-incremented file in the Finder location.

Deleting a duplicated reference file deletes the incremented file from the Finder (without warning, if the original is in an indexed group).

---

Reindexing a file into multiple locations will not generate new files. It merely adds references.
Replicated indexed files follow the same pattern of behavior as multiply indexed files.

Deleting a multiply indexed or replicated file will preserve the original file in the Finder.

However, this can lead to confusion if you delete such an indexed file outside the original location.
In this example, renamed.jpg was indexed into all three groups (including Three, which is not indexed).

Deleting the file from the Two and Three groups and emptying the Trash preserves the file in the Two group in the Finder. However, in the database the file is shown in the One group still.

[Silva]

I have a question. If tags associated with the files are deleted in DT, will they also be removed in Finder?