script to "auto replicate" all similar items to a

Dear all,

I hope the title says it all. This script takes advantage of devonthink’s “see also” command (it’s actually named compare in the applescripting), placing replicants of all items similar to the currently selected item into their own new folder. I use it whenever I find an apparently “orphaned” or oddball article–this automatically find it some friends. I then rename the group to the appropriate name, delete any unwanted replicants, and then use that folder as a target for DevonThink’s “classify” box.

Hope it is useful!

best,

erico



---Autoreplicate all similar to new group
--- Script by Eric Oberle for Devonthink Pro
---To use this script, click on a record and then run the script.  The script will automatically create a group in the enclosing folder of the selected item that will contain a replicant (alias) to all items that devonthink considers similar to the new group.  You can then delete those replicants as you see fit.  Great for an initial sorting of a topic that hasn't had its own group yet, but for which you have collected a few articles or documents....

tell application "DEVONthink Pro"
	set cuPos to {}
	using terms from application "DEVONthink Pro"
		try
			set cuPos to (selection of think window 1)
		on error
			display dialog "please make sure you have selected an article"
			return
		end try
		
		
		if cuPos is {} or (kind of item 1 of cuPos is "Group") then
			display dialog "please make sure you have selected an article"
			return
		else
			if parent of item 1 of cuPos is not {} then
				set import_location to first item of parent of item 1 of cuPos
			else
				set import_location to root of current database
			end if
		end if
		
		set group_name to "similar to: " & (name of first item of cuPos)
		set the_comment to "original url was " & (URL of first item of cuPos)
		set similar_recs to compare record (first item of cuPos)
		set new_destination to create record with {type:group, name:group_name, comment:the_comment} in import_location
		
		replicate record (first item of cuPos) to new_destination
		repeat with this_rec in similar_recs
			replicate record this_rec to new_destination
		end repeat
	end using terms from
end tell


Hi, erico:

Thanks – I appreciate your idea – it’s neat!

Haven’t yet tried your script but certainly intend to use it.

This is a great tool to help the user to initiate new groups of related content for better database organization.

Either for organizing ‘orphan’ documents, i.e. ones that Classify doesn’t yet know how to handle, or for creating new groups that reflect the user’s particular needs or interests, the script removes some drudge work that otherwise the user would have to do.

“Telling” DT Pro about relationships between items that are perceived by the user will make the artificial intelligence features work better for that user. This script can make it much easier for the user to “tell” DT Pro about such relationships.

Hi, erico:

I copied your script to the clipboard, pasted it into Script Editor and compiled it, then saved it as “New Group of Similar Contents” in ~/Library/Application Support/DEVONthink Pro/Data/. In DT Pro’s Scripts menu I selected Update Scripts Menu.

Then select a document and invoke the script. There’s still user intervention required to review and throw out items that don’t meet the “focus” that’s wanted for the new group, but the script gives a good running start.

Then rename the new group and move it to an appropriate location in the organizational scheme.

My reactions:

[1] Sometimes both Classify and I can’t decide where to place a new document, with the result that it doesn’t get organized. erico’s script can help remove the ‘block’ and get the document organized.

[2] The script really did help me put together a set of reference material that I wanted to organize under a new project group. I selected an article that I knew would be in one of my “project references” groups, invoked the script and moved the new group into my project folder. Repeated this on another article (different topic but related to the project). Very quickly I had the beginnings of my project references set. Some interaction (manual labor) such as throwing out stuff that I’m not interested in for the project might be needed. And of course I’ll probably add additional material as I work on the project.

Note: One of the strengths of DT Pro is the ability to replicate files. Replicants add very little size to the database, and replicants can be filed in multiple locations. When a document is replicated there’s still only one document, no matter how large it may be. The only additional overhead in the database is a bit of metadata, just a few bytes per replicant. So I can create a new organizational structure in my database for a new writing project, then replicate into that project organization the references that I want to use without making the database larger.

Think about these interesting possibilities:

  • When I’m finished with a project, I could export the project material from my database and delete the project files without deleting those project reference files from my database. Or I can leave the project files in the database, as the only significant size increase would result from the new material I wrote.

  • Having exported the project material as above, I could create a new DT Pro database and import that exported material into it. Now I’ve got a new database just for that project. I can archive it, or distribute it to another DT Pro user as a Web site (if organized like the Tutorial database), a download or on a CD or DVD.

Bill,

I’m glad you like the script. I certainly find this script helps a lot in the situation you describe—the straggler items for which I haven’t gotten around to making a new folder. This way they have friends, and the next “classify” action that matches it will probably have a very good chance of landing in the new folder. And indeed, it shows the power of the replicants in dt while not require the user to deal with what (IMHO) is annoying about them, the need drag them around with command-option-drags.

Because you mention that some “clean up” needs to be done, I’m posting an altered script that does two things slightly differently…First, it prompts you for the name of the new folder, so you don’t have to rename it by hand. Secondly, I’ve also put into the script both the line to move (rather than replicate) the original item into the new folder. You can comment out the one you don’t want by putting two minus signs in front of it. The script below, in otherwords, “moves” the original item in. Obviously, for those who prefer the earlier version, the older one is above. :unamused:

And just so I don’t post a script with thinking of a new feature for which to lobby christian :imp: , I might mention that it would be nice if there were a way in DT to tell the compare command either a) how many items were desired 2) what the cut-off score should be, or 3) to return a list of items along with their score. I find this script usually yields a few too many for my taste, and I’d like to control that.

But that’s the problem with devonthink…the more it shows us what it can do, the more we want…:wink:

best,

Eric


---Autoreplicate all similar to new group
--- Script by Eric Oberle for Devonthink Pro
---To use this script, click on a record and then run the script.  The script will automatically create a group in the enclosing folder of the selected item that will contain a replicant (alias) to all items that devonthink considers similar to the new group.  You can then delete those replicants as you see fit.  Great for an initial sorting of a topic that hasn't had its own group yet, but for which you have collected a few articles or documents....

tell application "DEVONthink Pro"
	set cuPos to {}
	using terms from application "DEVONthink Pro"
		try
			set cuPos to (selection of think window 1)
		on error
			display dialog "please make sure you have selected an article"
			return
		end try
		
		
		if cuPos is {} or (kind of item 1 of cuPos is "Group") then
			display dialog "please make sure you have selected an article"
			return
		else
			if parent of item 1 of cuPos is not {} then
				set import_location to last item of parent of item 1 of cuPos
			else
				set import_location to root of current database
			end if
		end if
		set group_name to "similar to: " & (name of first item of cuPos)
		display dialog "Please Enter Name for New Group" default answer group_name
		if button returned of result is "Cancel" then
			return
		else
			set group_name to text returned of result
		end if
		
		set the_comment to "original url was " & (URL of first item of cuPos)
		set similar_recs to compare record (first item of cuPos)
		set new_destination to create record with {type:group, name:group_name, comment:the_comment} in import_location
		
		
		---either comment out the next line or the following one, depending on whether you wish to replicate or move the originally selected record into the new folder
		---replicate record (first item of cuPos) to new_destination
		move record (first item of cuPos) to new_destination from import_location

		
		
		repeat with this_rec in similar_recs
			replicate record this_rec to new_destination
		end repeat
	end using terms from
end tell




``` :unamused:  :laughing:  :laughing:

Eric:

Thanks for the second script also.

Comments on how I would use one or the other script:

  • For dealing with “stragglers” the second script is the more appropriate, setting it to move rather than replicate the item to the new group. Now one doesn’t have to manually delete the original “straggler” item.

  • For dealing with project reference groups, which may be temporary and so shouldn’t interfere with subsequent Classify suggestions, I would use the original script (or the “replicate” option of the second script) and replicate the items to a new group. Then I would select the Info panel for that new group and tell DT Pro to ignore it for Classification determinations.

Thanks for making the second script convenient to set up as two separate scripts, one for creating a group containing only replicants and one for actually moving a “straggler” into the new group (accompanied by replicants of the similar items). It’s also nice to be able to name the new group on the fly.

And you are right: every time we see a new potential for DT Pro, we start thinking about all the further extensions of it that would be great to have. Which is why DT Pro keeps getting better. We may have to wait for some of those possibilities, though – some might require another generation or two of processing power, for example. :slight_smile:

I just fixed the bug in the second version of the script that would move all replicants of the originally selected item to the new group. Now it only moves the selected replicant.

Secondly, I wanted to note here that after messing around with this further, I discovered the “score” attribute to record. But unfortunately, after an applescript “compare” this attribute is not set. I’m not sure that is the intention of the parameter, but well, it seems like it might be…

finally, I’m attaching a script I use for cleaning up folders that contain too many replicants. It is essentially an “inverse delete”: it deletes all replicants in a folder that do not contain at least one of the phrases that you give it. It works well if you end up with a 100 replicants and want to now weed them down by using keywords. Unfortunately, it is a bit slow because applescript cannot “subtract” one found set from another, so it has to look at every record separately… Nevertheless, its a lot faster than what I can do by hand.


---delete replicants in the current folder not containing given strings 
--- Script by Eric Oberle for Devonthink Pro
---To use this script, click on a group or a record and run this script. It will prompt you for phrases that all replicants in the current folder must contain.  All other replicants will be deleted, non replicants should be left alone.  This script should not delete any other items, but please be careful.  This script is useful for filtered out big lists of replicants created by my "replicate similar to new group" command.  

tell application "DEVONthink Pro"
	set cuPos to {}
	using terms from application "DEVONthink Pro"
		try
			set cuPos to (selection of think window 1)
		on error
			display dialog "please make sure you have selected an article or a group"
			return
		end try
		
		
		if cuPos is {} or (kind of item 1 of cuPos is "Group") then
			set target_group to item 1 of cuPos
		else
			if parent of item 1 of cuPos is not {} then
				set target_group to last item of parent of item 1 of cuPos
			else
				set target_group to root of current database
			end if
		end if
		display dialog "Please enter phrases  which all replicants must contain at least one instance. Separate phrases by commas " default answer "a,the"
		if button returned of result is "Cancel" then
			return
		else
			set filter_list to my split_into_list(text returned of result, ",")
		end if
		
		set all_recs to get children of target_group
		
		set the_count to 0
		show progress indicator "Scanning records…" steps count of all_recs with cancel button
		repeat with this_rec in all_recs
			set delete_this_one to true
			if number of replicants of this_rec is greater than 0 then
				repeat with this_filter in filter_list
					this_filter = " " & this_filter & " "
					if (plain text of this_rec contains this_filter) or (comment of this_rec contains this_filter) or (name of this_rec contains this_filter) then
						log "**record " & name of this_rec & " is no-delete"
						set delete_this_one to false
					end if
				end repeat
				if (delete_this_one is true) then delete record this_rec in target_group
			end if
			if cancelled progress then
				hide progress indicator
				return
			end if
			set the_count to the_count + 1
			step progress indicator
			
		end repeat
		hide progress indicator
	end using terms from
end tell


on split_into_list(the_text, split_on)
	set oldDelim to AppleScript's text item delimiters
	set AppleScript's text item delimiters to split_on
	set return_list to every text item of the_text
	set AppleScript's text item delimiters to oldDelim
	return return_list
end split_into_list