Maybe I’m doing something wrong, but I cannot understand the logic after the Synchronize command.
After reorganizing my files and folders in Finder, I would like to synch my DT database to rebuild the links to the new position.
When I choose the Synchronize command on a selected folder, the Log window says that some items have been removed.
So, instead of looking inside the folders to find the new position for the files in the Finder (no name has been changed), it seems that DT has deleted what it didn’t find at the original position.
Am I doing something wrong? Having to reimport everything manually would defeat the purpose of the Synchronize command.
Indexed content in a database retains Paths to the Finder items that have been indexed.
Deleting or renaming the Indexed Finder items will break the Paths held in the database, with resulting loss of information. Likewise, moving Indexed Finder items within the Finders hierarchy can break Paths with resulting loss of information in the database.
So Index capture of Finder items to a database places strictures on what can be done in the Finder without loss of information in the database. (One of the reasons why I generally prefer Import capture.)
Thanks, Bill. I wonder if the path of indexed files could be made “alive”. I’ve seen other software doing so (for example NoteBook), maybe because they don’t index the actual path, but some information found in the disk’s btree. I don’t know. But I would find less “dangerous”, since moving files (or renaming them) is a common practice.
As an alternative, I would find nice if DT asked if you want to remove the links, or look for the moved files when synching. This is how, for example, Logic or Kontakt do, when you move their audio folders.
Indexing would still be my preferred practice, since saving space seems to me still a good practice. Even if I understand the advantages of importing in some cases (for example, when annotating PDF files, and you don’t want to edit the original).
Maybe adding a “Select” button in the Info pane, for manually re-linking broken links, could help. As of now, the only solution I can see to re-link a renamed/moved document is dragging the document again in DT. The “Select” button could help preserving position, labels, etc. of the document in the original DT group, something that dragging would reset instead.
I’m just facing a practical example of why I would like renaming/moving files would not remove indexed items from DT.
I’ve collected several documents in a Finder folder called “Research”.
While developing my research, I had to add at least a second field of investigation. I would prefer to separately collect the materials relevant to each field in a different folder, inside “Research”:
/Research/Theme A/
/Research/Theme B/
However, it seems that, if I move all documents currently in “Research” to “Research/Theme A/”, all corresponding indexed items in DT will be broken. Resynching would result in the removal of all these items from my database.
Is there a way to reorder my files in the Finder, while leaving items intact in DT?
You need to move the files in Finder, then resync. Don’t move them in DT.
An alternative is to use one group that indexes a Finder folder hierarchy, and use replicants in a another, separate, group structure that reflects your current research needs: index the folder, make a group structure that reflects your research needs, and then replicate the indexed files to those groups. Rearrange the research groups & replicants as needed. (Replicants take almost zero room - they are links - so what’s happening here is a link to a link.) A label or a tag or a naming convention can be used to show you which groups contain replicants and which contain “original” indexed items. This way, once you’ve indexed your folders from Finder, you can forget about that hierarchy. Also, annotations point to the indexed document, not the replicant. So if the replicant is deleted, the links in the annotation are still valid.
Korms, thank you very much. However, I was probably not clear enough in describing my issue. You say:
Moving a file in the Finder breaks the link from DT. Thereafter, synchronising removes the item from DT.
I cannot find a way to reconnect an item with a broken link, unless I re-index the file. This has the consequence of resetting all editing already done on the DT item (labelling, changing name, positioning in an unsorted list).
Maybe there is some passage of your detailed explaination that I didn’t get, and the solution is there but I cannot still understand it.
I’m trying to reword my question, in the hope to be a bit clearer. This thing is puzzling me, and I cannot find a solution.
I index files (instead of importing them), because I need the files freely available in the Finder for other programs/uses.
Indexed files are then reordered in DT, and I also sometimes:
change their names in DT (to make them more polished: for example, if the original file name is “02-Discussion on the Sky” and I don’t need the other chapters for my currente research, I may change it to “Discussion on the Sky”;
label them;
link them to other files in DT.
Despite my care to build a solid hyerarchy for my documents in the Finder, as the research grow the structure might become clearer and more articulate, and I might need some break-up for my files in a folder. So, I could move half of my hundered files from /Research/ to /Research/Mining/, and the other half to /Research/Geology/.
This operation breaks all links in DT. I therefore:
lose the link to the original files from DT;
lose all links from other documents;
when synching, lose the instance itself of the file in DT.
The Info > Path field is not editable; nor there is a “Select/New path” button, to reconnect the broken link.
Reindexing the files causes:
the loss of the original custom order in DT;
the loss of labelling and custom naming;
a loss of internal links from other document;
I’ve experienced moving “indexed” documents in other apps. I’ve seen two behaviours:
automatic re-linking of the moved file (e.g., Circus Ponies NoteBook);
Question asking either to search for the files in the connected disks, or manually indicating the new file/folder’s position (Logic, Kontakt).
None of these options seems to be available in DT. It is strange, since these re-connecting tools appears as very trivial.
I’m in a serious empasse, since I’ve started some researches in DT, and I’m spending a bit too much time repairing links and rebuilding my databases. A hint for a workaround would be really appreciated.
Paolo,
As far as I know, the links created by DT’s indexing do not persist as do those that link files to their apps.
As a general rule for indexed folders, only modify the original file and folder names (in Finder), not the indexed “copies” in DT. It means hitting Synchronize often (or attaching the Synchronize script to the folder in DT from the info pane; this will set of a synchronisation every time you select that folder).
To avoid losing files & folders that you move around in the finder, make sure they always remain inside the folder you indexed. If you move them out, they will no longer be visible to DT. It does mean adapting your file hierarchies so that you index the highest possible enclosing folder — indexing ~/Documents is fine if that is relevant to your needs; DT can probably handle more than you can ever throw at it.
What you describe only seems to work when there is the same folder structure in the Finder and in DT. But having the same folder/group structure in DT and the Finder makes little sense to me, since I see DT as an alternative way of organizing documents, often mixing files from different areas of my disk.
For example, the document “Theatre studies” could stay in the /Essential Refererence/ folder in the Finder. It could be both in /Shakespeare/Research/ and /Wesker/Research/ in DT.
I could later discover that I have another version of that book, and would like to rename the older one to “Theatre studies (first edition)”. How could I syncronize it, since I don’t have a mirror of the /Essential Refererence/ folder in my DT database?
Maybe I start to understand Korm’s suggestion: have an indexed mirror of your disk in DT, then replicate all needed files from that DT mirror, not Cmd-Opt-dragging from the Finder. The full disk mirror would be a layer between the research folders and the data on disk.
I don’t think that korm was suggesting that you index the entire disk as that would not be productive. At a minimum, it would make the searching and AI tools of DT useless. I believe what he was suggesting was to index a high level folder, or folders in the Finder (e.g. ‘Research’) and all the Finder’s sub-folders will be indexed as well (‘Research>Research Project A’, 'Research>Research Project b", etc.). You might want to put all indexed folders in a top level group in DT (e.g. ‘Indexed Files’). You would then replicate the folders and/or documents as needed throughout the DT database. As has been mentioned, replicating a document in DT takes no disk space-all replicants point to the same file, which in this case is the indexed file on disk.
Now if you edit the name of an indexed document in DT, the new document name will appear in the Finder automatically. If you edit and save the content of an indexed document in DT, the changes will appear in the document in the Finder. If you make changes to the contents of indexed documents outside of DT, those changes will be reflected in DT. What you do not want to do is change the name of a document in the Finder nor move the document in the Finder as the link will be lost in DT (as you have experienced).
As a final thought, you may have surmised by now that many of us that have been using DT 2.0 for some time do not use the index feature extensively. At a minimum, we do not use it for dynamic files that are renamed, moved, or otherwise accessed frequently from the Finder. Personally, I only index files that other applications expect to be in a specific location, such as iTunes podcasts. I can, and do, access my Word documents, Excel spreadsheets, etc. from within DT. Also, documents in the DT database that are opened in the original application (e.g. Excel) will appear in the application’s ‘File>Open Recent’ menu, so as long as I am working with a small set of documents, I don’t need to return to DT to access these documents.
Absolutely true. Index only the Finder hierarchy that is relevant to the database.
True here, also. I index documents that belong to my Federal clients, for security reasons. Otherwise, I import.
If at all possible, import. DT databases can be large and still be efficient. Compare the size of the Finder folders you’re thinking of importing to the advice in this thread and others related to the search string “+database +size”.
I was just about asking the same question as I am facing the same very basic issue. so I found this ongoing discussion here.
result to me: there is definitely at the moment no way to keep DT-structure valid when you re-order your filesystem independently of DT.
I was also exploring to what extent this can be circumvented when - as mentioned/offered here – the original file-/folder structure (with its »living« replicants/duplicates etc) in DT is manipulated through DT (– which would make things more complicated; but maybe a price you need to »pay« using the index-methd). BUT (as far as I can see) this only works as long as you rename files/folders through DT and completely stops working when you create new ones and then (re-)move existing folders.
now, I think there are pros and cons discussed elsewhere in the forum on indexing vs. synching. but to me they result in the notion that both systems are/should be valid. I want to be free to stick with indexing. besides Paolos concern for size (which I share) there is always the issue that for several reasons you want to be able to work with your file structure outside of DT (one reason being that its not the file system, and that it can´t process ALL the information manipulation/retrieval needed, so you want to be able to work with the same files through other apps also at all times).
so, this brings me back to Paolos more concrete, constructive and unadressed questions/proposals of solving this:
I think that DT is such a powerful tool, and it is tailored to be THE main information-sorter/-structuring-tool that it should have these two dimensions at the same time:
openess/transparency to the basic file-system
sustainable projection of the information-structure generated witin DT (meaning also: keeping your work-input alive)
PS: just practically came another issue/scenario where all this is of concern.
I have ( – because of space-issues – ) a lot of my mediafiles on an extra drive, like probably some folks (especially those with notebooks). in this way they are reflected in DTpro. hopefully I will (outgrow) this problem one time.
– more generally this an issue which just reflects that file structures also change along the expectabel hardware-paths (besides other things, like the need to arrange it for reasons of structural development.)
at the moment I would have to chose between boxing my files (I know it is a soft “boxing” with DT… ) or losing hours, days and weeks of work and unreconstructble information structure.
… just to add to the more “existential” issue Paolo raised and give some further scenario-evidence.
I’m following the dialog regarding “live” indexing with increasing interest and support. Thanks to lerone, ptram and others for their thoughts. I index extensively and carefully prune and modify the target folder structures on disk. It would be very helpful if DTPO adapted to changes on disk automatically, as suggested above and in other threads.
… just as a reaction and to add to – or sort out, if you have it – the involved larger scale discussion:
I take korms comment here mainly as a support for transparency between DTpro and the file system (or »liveness« if that is preferred).
– like him I think this is a prerequisite for everyone to flexibly and sustainably develop their own individual approach to information structuring with the help of tools like DTpro.
I am saying that just to keep the issues and stakes clear.
– because I personally actually like the approach ‘tags as groups’ (and left leap as it was abandoning it). that is as long as the groups/tag-information is also back-channeled into the wider metadata-domain beyond DT, which is a thing DTpro thankfully does (with OpenMeta).
after all the group/replicant-structure is one thing (besides the wordbased AI) which sets DTpro apart in my eyes from other information management systems (and you don´t find it in PersonalBrain, and personal Brain is also a little obstrusive in terms of its tag vs. parent-structure … etcetc.). and there are other discussions in the forum asking for developing tags further…
all that said I personally agree very much that metadata have to play an independent role from groups, stacks and so on…
[see: http://www.devon-technologies.com/scripts/userforum/viewtopic.php?f=4&t=7291]
so, there is a lot involved, also in terms of the architecture, philosophy and general usability/sustainability/transparency and such of such a formidable tool like DTpro…
bottom-line still: improvement in the reference-system making it more live/endurable/flexible vis-a-vis the file system, like Paolos proposed, would be a good and appreciated thing
) or losing hours, days and weeks of work and unreconstructble information structure.