Syncing folders between DT-Databases? (this should be easy... I guess...)

hello community,

– pre-roll:
• I would have thought this is easy. but either I am overlooking the obvious solution, or it is not that easy in the first place.
• I´d also expected to find this question / issue in the forums as it seems such an obvious case, but couldn´t find anything on it (maybe because ‘sync’ is such a crowded keyword :slight_smile: )

– question / issue:
• I am looking for a way to sync one folder of a database with a ‘doublette’ in another db. simply that.
• one potential complication (though I don´t think so): the contents in the ‘original’ DT-folder are partly stemming from indexed folder (i.e. folder synced with the filesystem)

I am grateful for any help. I guess there should be threads or other documentation on this very issue – but as said: I can´t find them. so your help and input and any hints and leads are appreciated!

thx, oliver

Databases are either synced or not. There is no option for syncing individual groups within a database. (For the sake of completeness: DTTG supports shallow sync, but also does not support syncing individual sections of a database.)

2 Likes

ok. thanks Blanc.

I think there is some technical weight to ‘sync’ as a term.
and maybe part of the issue I am facing is getting 'database sync’ out of the equation.

I can e.g. automatically update the indexing status of a single folder – such that is ‘kept in sync’ with the resp folder of the filesystem.
I understand this is not how ‘sync’ is used as term in DT terminology.

I want to achieve something likewise with the contents of two folders in two different DT-databases.
there are so many options to process and move files in specified folders (with intelligent rules and/or scripts) that I thought just keeping the contents of two folders identical (‘in sync’) should be doable within DT…

… so, you are saying this is positively not possible within automattion (– outside of DT sync)?

In addition to @Blanc’s sage advice, you mention indexed folder in the mix. I’m not sure if they are now indexed or previously indexed. No matter. If you put all these files into a folder in the macOS file system, then index into both DEVONthink databases. If necessary, you can use macOS apps (Chronosync, rsynch, Hazel, et. al.) to sync if the folders are in different devices. This might achieve what you are trying to achieve (if I understand what you are asking).

Also… remember that the correct nomenclature in DEVONthink is “Groups” which has a specific meaning. In the macOS file system, the collection of files are indeed “Folders”.

2 Likes

Do you mean “folders” as in Finder (then it has nothing to do with DT) or do you mean “groups” as in DT?
In the latter case, I’m not sure why you’d need the same group with identical content in two databases. And it’s not clear to me, if you modify said group in only one database (in which case a smart rule might help) or in both.

1 Like

I wouldn’t dare. My DT installation makes me coffee in the morning and grooms the dog.

I think it would be helpful if you could provide more details; keep an eye on the detail @chrillek mentioned, too (group vs folder).

What is in the groups? Is the content imported, indexed, or mixed? If indexed, where from? Does the second device have access to the same source?

3 Likes

thanks to everyone leaning in here. indeed, as @rmschne hints, I appreciate the collective and individual wisdom that is shared here. and once again I realize that there are a lot of pitfalls in the descriptive space between intention, expression, DT nomenklatura and inuitive user lingo…

… indeed that reinstills some hope in me that my question still might be regarded as solvable, too!

so, I try to make up for anything that didn´t really come across transparently in my first attempt to describe what I am after (– though I still would think in the end it not a very exotic case; see below)

my bad. while some indexed folders are involved I am in the end indeed talking about DT internal(!) groups. they just happen to also hold some (referenced) files – i.e. replicants – from other folders that are indexed.
the process is like this: I index some file folders, but also create new MD-files within those folder from within DT – and they are flawlessly ‘synced’ with the original OS-folders.
now I selectively replicate files from these ‘base groups’ (linked to indexed folders) within my ‘global’ DT-database to other / different folders for topical sorting / grouping in that very same DB.
that´s why I label / characterize the groups in question ‘derivative’ here.
in the end, or: in the next step, I want these ‘derivative’ groups to be mirrored ‘forward’ into – different, selected – ‘special’ DT-databases, of which I have a whole bunch.

yes, my basic ‘starting’ group in the ‘global’ DT-database is an indexed folder. but then I distribute (replicate) the files into ‘derivative’ groups. thus the folders I want to ‘mirror’ into other databases are themselves not really indexed folders but those ‘derivative groups’ from within my global ‘inbox’ / DB. (but still their contents are linked to / derived from indexed folders in the very same db.)

but as the content is distributed from indexed folders to other ‘derivative’ / curated groups (the ones I want to mirror or ‘sync’ to another DT-db) any syncing on OS-level wouldn´t help / doesn´t apply for the scenario – though I understand the thinking behind that kind of strategy. but the groups I want to mirror do not exist in their specific composition outside of the (global) DT-database (even if their contents originally comes from an indexed folder).

in the end it only needs to be ‘one way’, resp. I could very well live with that kind of restriction. it would already be enough of an achievement if I could mirror my ‘derivative’ groups from the ‘global’ database into the ‘special’ databases. and while it would be even better, if the set-up would also allow for the other way round (allowing me to change folders in the ‘specialized’ databases and mirror-back the changes to the ‘global’ database), it is not strictly neccessary.
– but my initial hope was that I could achieve such a mirroring of individual groups via smart rules, – just as you say. but, then, I don´t see an easy way for that currently – and surprisingly.

my workflow / architecture makes this neccessary. I am working with a particular and rather modular scheme of databases, some ‘global’ some ‘specific’. and my whole scenario (plus the way I use the immensely powerful search and AI in DT) relies on some folders being present in the global and some specific databases.
maybe to give a shorthand for the rationale forcing my hand: I deal with a lot of very different contexts. but in my collection process I want to deal with only one ‘global’ ‘inbox’ – because everything else would drive me nuts… :-). so I need a way for automated transmission of the sorting within my global DB into my multiple sub-DBs (which I only open / use according to work context)

I hope I could clarify my case a little; and also retrofit my case into the correct DT-lingo / -nomenclatura / -schematics.
I also hope I could clarify why a sync on the OS level wouldn´t solve my case.
I see that it is not very easy explain or understand, given the different descriptive levels involved (something additionally complicated by the fact that indexed folders are involved, which are in some way (via replication) the initial sources of the ‘derivative’ DT-groups in question and to be ‘mirrored’ into still other databes)
– but I also still think the underlying question is a real simple one, and be phrased very simple in correct DT-lingo: how can I automatically ‘mirror’ one group from DB ‘A’ to => DB ‘B’, and keep both identical down the line?!

thanks again for any shared wisdom, as for any practical guidances!

By syncing database A with database B using the sync-mechanism internal to DEVONthink (and which is explained in detail in the handbook). You cannot sync a single folder in lieu of syncing the whole database.

2 Likes

@Blanc – I think there is a profound misunderstanding in here. ‘the sync-mechanism internal to DEVONthink’, as far as I know, applies to different devices. I did try to give an extensive description. But nowhere do I talk of different devices. I talk of different databases (on the same device)

I went quite some lengths to explain that I am not really talking about the technical / nomenclatura ‘sync’ process of DT. this is - inter alia - why I substituted the (intital) colloquial ‘sync’ with ‘mirroring’ in my re-description.

my whole question revolves around the rather simple but baffling situation that you can seemingly tell DT (via smart rules) that any file x in folder A in DB 1 which has been [ labeled / opened / keyworded / etc. ] in (this or that) way shall be moved / duplicated to folder B in DB 2, or be systematically modified in this or that way … but still there is (seems) no easy, obvious (to me) way to simply make automation keep all files in folder A in DB 1 ‘in line with’ (in ‘mirror-state’ / ‘in-sync with’ / ‘the same’ / etc pp.) as to folder B in DB 2 (on the same device).

I apologise, I was trying to respond whilst doing too many other things at the same time and misunderstood your intentions.

To the best of my knowledge, there is no simple way to do what you are looking to do. In fact, I had attempted to do something similar some while back when I tried to make certain groups available to the rest of my family. It turned out not to be worth the effort.

One-way mirroring could be done by deleting group A in database Z, followed by duplicating group A in database X to database Z. Those steps could be performed by a smart rule and scheduled. Two-way mirroring would require scripting; things become complicated when you delete files or edit them in both groups, though.

You may want to reconsider how you have split up your databases. Because groups can be replicated within but not across databases, your problem would be easily solved if you were working with one rather than many databases.

thanks, @Blanc, for your second look into this!
also, I do understand we are all prey to overstretch / multitasking / simple misunderstanding at times.

first of all I read your last reply as a kind of vindication as to the surprise to hit a wall with something one would deem covered by the very sophisticated combinatorial set of smart rules that DT does provide…

…then, I hear you on reconsidering my DB architecture. just that in my case this is already the outcome of some deliberation and elaboration over the years. (besides the point maybe, but still a thought, that DT with it’s DB options kind of invites people thinking around sets and modular uses of DBs - think work space / environment presets -, so that I’d think these kind of cases should be covered… just a thought…)

… but on to the practical and pragmatic:
as I said, one-way mirroring would already help me a lot!
so, could you elaborate a little in terms of the smart rules used and the way this set-up would work?
… I kinda get the general idea but would not be able to replicate this right away…

thanks again!

You didn’t mention whether you want the same files or the same records in two different databases.

If you index a folder into more than one database, then you have the same files in each database. But you won’t have the same records.

  • Test.md in database A and Test.md in database B will only share their textual content.

  • Record properties are not “synced” across databases, i.e. if you set a red label in database A it won’t show up in database B.

    • Exception: Finder comments are “synced” because they are written into the actual files.

So, if you are only interested in having the same files in more than one database then used indexed folders.

It’s probably not useful at all but this script syncs records whose kind:any.

As there’s no way to tell whether a record’s property has changed we can only “sync” all properties of all records in a given indexed group.

The problem is: How to trigger it? :slight_smile:

-- This script syncs records whose kind:any. It doesn't sync groups.

property theDatabaseNames : {"_temp", "test"} -- set names of databases
property theIndexedFolderPaths : {"/Users/user/Desktop/Test/"} -- set names of indexed folders

tell application id "DNtp"
	try
		set theDatabase to current database
		set theDatabase_Name to name of theDatabase
		if theDatabase_Name is not in theDatabaseNames then error "Please select one of these databases:" & linefeed & linefeed & my tid(theDatabaseNames, linefeed & linefeed)
		
		repeat with thisIndexedFolder_Path in theIndexedFolderPaths
			set theRecords to (contents of theDatabase whose path starts with thisIndexedFolder_Path)
			
			repeat with thisDatabase_Name in theDatabaseNames
				if thisDatabase_Name ≠ theDatabase_Name then
					set thisDatabase to database thisDatabase_Name
					repeat with thisRecord in theRecords
						set thisRecord_Path to path of thisRecord
						set theResults to lookup records with path thisRecord_Path in thisDatabase
						if theResults ≠ {} then
							tell item 1 of theResults
								set aliases to aliases of thisRecord -- Wiki aliases
								set creation date to creation date of thisRecord -- The creation date
								try
									set custom meta data to custom meta data of thisRecord -- User-defined metadata of a record
								end try
								set exclude from search to exclude from search of thisRecord -- Exclude record from searching.
								set exclude from see also to exclude from see also of thisRecord -- Exclude record from see also.
								set exclude from Wiki linking to exclude from Wiki linking of thisRecord -- Exclude record from automatic Wiki linking.
								set label to label of thisRecord -- Index of label (0-7)
								set locking to locking of thisRecord -- The locking
								set rating to rating of thisRecord -- Rating (0-5)
								set state to state of thisRecord -- The state/flag
								set tags to tags of thisRecord -- The tags
								try
									set thumbnail to thumbnail of thisRecord -- The thumbnail
								end try
								set unread to unread of thisRecord -- The unread flag
								set URL to URL of thisRecord
							end tell
						end if
					end repeat
				end if
			end repeat
			
		end repeat
		
	on error error_message number error_number
		if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
		return
	end try
end tell

on tid(theInput, theDelimiter)
	set d to AppleScript's text item delimiters
	set AppleScript's text item delimiters to theDelimiter
	if class of theInput = text then
		set theOutput to text items of theInput
	else if class of theInput = list then
		set theOutput to theInput as text
	end if
	set AppleScript's text item delimiters to d
	return theOutput
end tid

Pete – this is of course some meaningful finegrain, and I think this will help people in simimlar situations understanding the issues and options.

In my case all this, I would think, flatly falls on the “records” side.
But it´s not by choice or decision, but rather because of the constellation of the matter.
To illustrate again, what I tried to convey in my second description in particular:

• I have an indexed folder A in DB 1 containing the file(-record)s x,y, z
• … now I sort y and z into folder B in the same DB 1
• … finally, I want this folder B to be identical (“mirrored forward”) with folder C in DB 2
– because of this there is no (direct) indexing in relation to folder C in DB 2. And because of this, I would think, the only way to achieve the parity of B and C would be duplicating / copying and moving records from B to C.

[edit] … this is also why I wonder whether any reference to the indexed folder A is really needed at all in any script / automation dealing with it? like:
property theIndexedFolderPaths : {"/Users/user/Desktop/Test/"} -- set names of indexed folders
– initially I just mentioned the original indexed folder in DB 1 to give a complete overview of the set-up and flow of things. but in the end the relevant folders to be ‘moved’ (worked on) are maybe technically and in light of your differentiation just DT-records holding folders (my folder B)…?!

– does that make sense?

Thanks, gracious Pete!

… your scripts are always useful :slight_smile:

… but, looking at the script and my answer to your first comment here, I wonder, whether this script really is fit for the particular setting? I would expect that some reference like
property theIndexedFolderPaths : {"/Users/user/Desktop/Test/"} -- set names of indexed folders
is not needed, as it is not the indexed folder per se that should be copied / held identical between the two databases (see my prev. answer)…?!

no matter to me. I think this is where my trouble with the smart rules start… that there is no relevant / easy / non-ambivalent property change to grab or to cling to…
but I am 100% fine with a re-rinse of the process, even if it´s not elegant technically

oh, it could be folder B (the original folder in DB 1) is re-indexed; it could be database 2 opens which would mean there is one step / trigger initiated ‘by hand’, which is also not elegant but fine to me; it could also just be done ‘by hand’ … does any of those make sense to you

thanks again for getting your hands dirty here…!

Why?

2 Likes

That, I think, is the answer Jim. Even though I have to admit that in being the answer it is at the same time not the answer :smiley:

2 Likes

I’m trying to imagine a scenario in which this would be useful. :thinking:

  1. I already tried to give an answer above, Jim.
    I am not sure you read it. if so, let me know what kind of take you have on ot, or on which aspect I should elaborate

  2. I personally think, in cases where users gave some elaboration on motivation and set-up it is not productive to just flat out question their way of doing their things. at least given that I already partly answered the question it reads that way… I’d expect some more constructive approach / tone in forums, especially from moderators

I personally think this kind of ‘banding up’ and commenting over the head of someone who brings forward an issue / question is not nice (and I am saying this nicely). I also do not think it encourages people on the forum to see it as safe / helpful space.
… in the end this is even more irritating as it is documented that is not clear on whose side misunderstandings rest.

I am trying to talk issues. not question peoples motivation. or their way of going about. and I work on the premise of respecting others people motivations and knowledges (especially in their own affairs)

I’d wish such specific threads with user requests for help / advice in the DT forum would only be used by people interested in the issue and in helping (including understanding the OPs perspective instead of just lazily question it).
the others (who feel bugged) can sinply pass on w/o losing anything.
(that is, if this is not about documenting / displaying authority / superiority)

I think this is not a good example of a forums general culture, tone & style. and it rather displays a latent (or not so latent) aggresion.
– but everybody can have their own opinion on that. I want to document mine here, in light of the last two ‘comments’ / ‘answers’.

Understanding the “why” of something and being willing to question the efficacy of a particular process is often very useful in determining the direction of responses. Sometimes the issue to be solved isn’t the one presented.

2 Likes