Concordance and smart groups

I’m indexing some texts for research. I paste parts of the text into different RTFs, and name them like this: 1-2-17. Which means text 1, chapter 2, part 17.

Then I make smart groups: I make a new group, name it (example) “creativity”, use Shift-Apple-I to open the information dialog. Then add a script, the “Any word.scpt”

Great, so now DT lists all parts where creativity is.

But when I open the Concordance window, it lists that there are 20 occurances of the word “creativity”. When I click the word, it only lists 10.

The reason is that it counts the lists under the smart groups as another one. Why is this? It doesn’t make sense to me?

I thought the parts that were listed under smart groups were “Replicates” (well, they must be because when I edit one, the “original” changes too.

But why does DT count a word twice when it’s replicated? So if I replicate a word into several groups, it’s occurance bounces up.

What is it that I don’t understand here?

=) Oyvind

Oyvind, the Concordance is part of DT Pro’s code that not only lists all the words in the entire database and their frequency of occurrence, but knows which documents they are in and how they are organized in your database.

So when you replicate a document and place it into 2 or more groups DT Pro’s Classify function will know about that in making suggestions for location of a new item, and might suggest location of the new item in one or both of those groups, depending on how similar to the replicated item (and other content in it’s group) the new item may be.

Why make a smart group for “creativity”? You could instead do a search for it (which doesn’t result in DT Pro having to manage an organizational context) or even select the word “creativity” and press the Option key, which will pop out a drawer listing documents that contain the word.

Remember that when you create a smart group it’s difficult to control the specificity of the search in your organizational structure. The smart group will be database-wide, and may be pulling in documents that contain the term from other than the set you are interested in. And when you create a smart group you will change the frequency of use of the term in the database.

But if you instead use Tools > Search it’s easy to specify the group you wish to search for the term. And the search result will tell you how many documents contain the term. And the search will not change the frequency of use of the term in the database.

Example: I was associated for some years with Lynton K. Caldwell and coauthored several publications with him. I’ve got several references to him in my database. If I do a search I find 27 results for his name. When I open the Concordance I find 79 “Lynton” terms, plus a few more occurrences as all uppercase or all lowercase.

Now I create a group and replicate the search results into that group. I go back to Concordance and update it. I’ve just doubled the frequency of use of “Lynton” and the other variants.

And that’s the way it is. :slight_smile:

The Concordance wasn’t designed with the purpose in mind of helping some poor soul determine the frequency of use of terms in different Shakespeare plays. That could be done document by document, but with some drudgery, or more easily by creating a separate database for each play. Instead, it was designed to help DT Pro’s search and artificial intelligence features.

The reason I make smart groups for certain words (and terms) is that it’s so much faster to just click the smart group, and it updates.

If I should click or write “creativity” each time I need to search for it, it would take much longer time.

The other thing is that I made my research look simpler than it actually is. :slight_smile: I have about 30 smart searches, about 10 of them are persons, then there are some terms (or phrases. I use “phrases” script I found in another forum here), and some are just single words.

I could of course search for all these “manually” each time I need to see them, but it takes too much time.

It’s also useful for me to see the list with all the smart searches, and the number behind them.

I understand why the Concordance list lists words more times when they are replicated.

Maybe what I would like to see then are better smart groups: Smart groups that behave like smart groups in other apps. When I make a smart group in iTunes, it doesn’t replicate my music. If you see what I mean.

I haven’t used DT for long, and I’m sure I will find ways to organize things along the way.

When first learning about DT Pro, while still back on Windows and experimenting with dtsearch (totally unrelated windows-only software), one of the things that made me decide to go Mac and use DT (yes, DT was acually a prime reason for moving to Mac) was the Concordance tool.

I was a bit disappointed to later find out that the concordance is not, in fact, truly a database wide concordance as a user would understand.

Is there any way that in a future version that a way to make the Concordance actually show true concordance information for the database (as a user would want to use it, as delineated by Bill above)? It might seem like a small change, or of negligible value to some, but is useful in certain types of textual work and anaylsis. Thanks.

Just let me know which improvements you would like to see (small and/or concrete suggestions preferred :wink:). Thanks.

Concordance is the feature that I use the most. I’d like to see an option to make it case sensitive/not case sensitive should the user want to.

I would like a concordance that does not change when I make smart groups. Or the opposite: Smart groups that doesn’t change the concordance.

In other words: I would like to replicate a document a million times (or less) without all the words in that document getting a million hits extra in concordance.

:slight_smile: Oyvind

I will be posting my suggestions about the concordance, as well as what I see as related suggestions, in the “requests and suggestions forum”. Just since that’s where requests/suggestions are supposed to take place, and so others who have similar thoughts may add on to the thread there.