Tags should be equal citizens when searching!

The new tagging feature is nice but I find it a bit lacking still, particularly with respect to searching. My two complaints are

  • Tagged items don’t show up individually in search results (when searching for a tag)
  • Tag terms seem to be indexed separately from other words

Here’s an example of the workflow I’ve used that led to this discovery:

Came across an article online called “The Problem With User Stories.” I clipped it to DTP, and added the “Agile” tag to it. The article itself doesn’t include the word “agile” anywhere. If I use the search bar to search for “agile,” it shows me the one group named “Agile” representing the tag. I can click in there and see all the items below it. However I would like the item to show up in the normal list of results, not just be stuck under this group.

I found that I could workaround it by opening up the inspector for the item, copying the tags and pasting them into spotlight comments. Now doing a search for “agile” shows that article as a separate item in the search results.

However, if I change the search to “agile stories” then the item disappears. This is why I think that the search must be doing something different for tags than for the normal content of a document. If I search for “problem stories” then it correctly shows up, and it shows up even if I search for “stories” and some word that appears in the body but not the title.

Basically when I tag something I see it as an annotation of the original document. When I search for a word and a document is tagged with that word, I expect it to show up in the results just as it would if it had that word in the title.

This would only be a minor annoyance if my workaround were effective…but the fact that a search for “agile stories” won’t reveal this document, when it has “Stories” in the title and is tagged with “Agile,” to me that’s a serious breakdown in DTP’s search capabilities.

…and there seems to be a bug which doesn’t allow smart groups to search for tags, even though it appears an option when editing/creating smart groups.

First let me say that I think you continue to do a fantastic job developing DTPO!!

I’m not experiencing that problem with searching on tags via smart folders. However, I would really like to see searching on tags become much more easily available - for example, as an option in the search drop-down menu and in the advanced search window, having the option to put a “create new smart group” icon in the toolbar, and the option to have a keyboard shortcut to edit a smart group.

I still don’t understand why people think there is an advantage to using groups, subgroups, and replicants over tags - although I could certainly understand someone with a very large database deciding that they are not going to go back and change from one approach completely to the other. However, going forward it seems like one approach (groups) gets increasingly difficult to manage over time (with stored information growing exponentially), while the other (tagging) doesn’t get much more difficult to manage at all.

Because I am very interested in testing out the tags approach, and because I suspect that I ought to be able to manage a database with a single Library group that contains all of the tagged items, I am beginning to go through the process of converting my approach to storing and retrieving information in DTPO. One of the difficulties I have encountered thus far is that all of the groups in a branch are inherited as tags when I choose a group at the end of that branch. In the group/folder way of organizing things, some groups aren’t really meaningful tags as much as they are indicators of logical divisions of information - like signposts to help direct the browser. It would be nice if there was an easy way to turn off inheritance for the purpose of tagging - rather than having to undo it for each folder individually. Although I suspect that one really needs to have a very flat database in order to fully and cleanly switch over to the tags approach.

This might be silly, and may not matter, but when I save new information into my Inbox, tag it, and drop it into my Library group, it has an address that starts out "Tags / " - I would like it to have an address that starts out "Library / " because I don’t want to see the Tags folders in search results unless I specifically request it. However, I can’t imagine that there will be many times when I am simply looking for all of the things that are tagged with a certain word or phrase. Even if I do, I could (and probably would) use a smart group - that is, I would be a lot less likely to look for all the files that are tagged with “article” than I would to look for all those that are tagged with both “article” and “Organizational Learning”. Which reminds me, it would be great if we could indicate that we don’t want the smart group searches to be case-specific.

I have another - unrelated to tags - suggestion/request: It would be great to have a module or application that enabled a database that was an expansion of Sheets to include a Form view - kind of a Bento-light. There are some types of information that are much easier to capture and review that way. I think it would be completely reasonable to have that be an add-on for a fee if it had some nice usability - I don’t think it needs to have the glitz that is present in Bento.

Thanks again for great continued development on DTPO 2!!!

I like to second Padillacs concern, especially in relation to the adverse effects of having tags treated seperately in terms of search operations.

I too would like to find indexed documents when searching with the search-bar whenever I put in terms which are either in the content-body or in the tag-section of it.

(PS: I want to point out that I am really happy with the principal effort to include tags into DT; actually this and the inclusion of OpenMeta made me return to DT after almost 2 years of abstinence!)

There used to be a proto-version of this functionality, long since abandoned. Individual records in Sheets could be summoned up as separate documents, which had a minimal Form look to them. There may be others who have good information on why that particular branch was left to wither. :neutral_face:

hello,

as I more and more like »the folder is equivalent to tag« idea and think the system-folder-structure should be a primary source for meta-info, I am quite happily using this more and more extensively.

… so, I would really like to know, whether this feature request is on the radar of the Devon-team?!
not being able to use tags (parent folder) as a side-by-side-criterion for structured searches really spoils a lot of potential…

so, wonder whether this issue, brought up by different people, is acknowledged? thanks!

Here’s why your workaround using Spotlight Comments failed when you modified the search query to look for two terms, but only had one term in Spotlight Comments:

DT search treats the components of an “All” search, such as Name, Content, Comments, etc. as individual “fields”.

It found the term you had placed into Spotlight Comments, and displayed an item in search results. But when you added a second term, that item disappeared.

Why? Because your query was of the form word1 AND word2. But none of the “fields” in your database contained both terms, so the result was null.

I understand…that’s not how I want it to work though. If a document has the word “stories” in the title, and I tag it with “agile”, then I want it to show up in a search for “agile stories” (with no quotes). I understand that it doesn’t work, and why. Hopefully at some point in the future it’ll behave the way I want.

hello, to support that:

one of the good things DT introduced was to take parent-folder serious as meta-information and treat them equal-to/as tags.

it loses a lot(!) of its worth though, if - for example - I put some file named or containing “tagging” in the folder “information” and looking for “information tagging” I don´t find that file.

– it basically severs the folder sort and(!) the tag structure from the rest of the intelligent information retrieval structure of DT.

– what would be the advantage of that?

hey,

as others I am using tagging via DevonThink, and as a matter of fact as my prime tool for this by now.
as mentioned by others already, along with Ammonite and Punakea/Leap this gives some wonderful benefits – that is, as long as tagging doesn´t get a seperate process from normal info-sorting/retriveing in DT.

… so I wonder whether this issue/request to streamline tags into the normal search/indexing within DT is heard/considered, as I do not know of any DT-voice on this up to now?!?

as tags (folders) are simply an additional information, it should in my view really be integrated in a powerful info-organizer – or the search function loses some serious value within DT in regards to the integrated tagging system. it also beats user-expectations, when the user is searching in “all” available information categories – and tags are bluntly not part of this.

Yes, tags are additional information, and it is important that they be searchable, either independently or in combination with other query terms and filters. As well, one should be able to create smart groups based on the inclusion of tags in the query.

But because Tags usually have names that may be common in document contents, other metadata or in the Names of documents, they need to be searched in a unique way. Otherwise, a search based on a Tag Name would probably pull a number of unwanted results if tags were included in ‘All’ searches.

That can be done in the full Search window (Tools > Search), but not in the limited Query field in the toolbar of view and document windows. Use the Advanced button in the full Search window to enter a variety of query filters, including Tags, dates, date ranges, item Kind, etc.

I’ve attached a screenshot of a query in the Advanced button. Those with sharp eyes may note that I’m using a beta of the next maintenance release for the DEVONthink applications. Comparisons for tags includes ‘is not’ as well as ‘is’, and date comparisons include ‘is’, ‘is less than’, ‘is less than or equal to’, ‘is greater than’ and ‘is greater than or equal to’.

So this search pictured in the screenshot is set to filter for items that have a specified tag, but exclude items that have another specified tag, that have creation dates falling within a specified date range, and are images. (Don’t be confused by the way creation dates are expressed in the screenshot. I like that format, and have set it for short dates in System Preferences.)

As the full Search window is used, all I would have to do to convert the search into a smart group would be to click the ‘+’ button to the left of the query field.

Remember, though, when using the Advanced button, to press Reset to clear it if it isn’t wanted for subsequent searches.

How about that? :slight_smile:
Advanced button search.jpg

hey Bill,

thanks for adressing this! that is appreciated.

… I guess what I wanted to express is that the evaluation of “probably unwanted results” might not be the standard (or encompasing) case for people actively using tags. with “additional” I meant “integral” not “separate”, as in my experience that is the way people actually use tags: they add them as ‘core characterization’ or as the most important (“key”)words that actually match (or summarize) the content.

to give a case: when I put some article in a folder/group “atomic energy” (or tag it like that), that doesn´t contain that string originally (either because it´s not part of the text or because it is another language), I sometimes exactly want it to appear when I search for “atomic energy” alongside all those documents that contain the string “atomic energy”.

as it is, I would need to go all the way to do an “full search” and then each time build a complex search query. (in which case, just besides I would have to correctly guess/remember the exact tag-name; e.g. putting “energy” there would leave out the offer of the “atomic energy”-tag, as there is no “contains” but only “is”/“is not” and I have to match the right string from the very start of it)

of course, I take the case that sometimes people - depending on the situation or their usage scenario of tags – don´t want the integral search. but given that all other programms I know making use of tagging allow for an integrated search, I would assume there is a reason for that, i.e. that the other scenario is at least equally probable.

practically and to resolve that, in my view, at least an option to define “tags” (and folder groups) as integral part of the search should be there in the end.

there should be one customization/usage way, which allows to just search for a string like “atomic energy” (or just “energy” for that matter), which actually brings up documents containing “atomic energy” either in their text-body or as a tag/group label.

so that is my opinion as someone using and cultivating the usage of tags since some time and trying different pieces of software on this.
– but, again, thanks for taking this all serious for now.

Your scenario on tagging documents about atomic energy reminds me of my frustration about serious problems with assigning keywords (the problem is the same for tags): subjectivity and consistency, when I was director of a computer information center many, many years ago. Back then, searches of our computer tapes could only be done on keywords.

Subjectivity: Different persons given the task of assigning keywords (or tags) to the same document will invariably differ in their interpretation of the important features or content of the document. The more complex the document, the worse this issue becomes. Worse, the same individual reading similar documents will display the same issue, depending on the individual’s reaction to the content of each document.

Consistency: Should the keyword (or tag) be ‘atomic energy’ or ‘atomic power’ or ‘nuclear energy’ or ‘nuclear power’? This is an example of a keyword or tag that should contain at least two terms, as we wouldn’t be happy with a search in a large database that simply queried for ‘atomic’ and ‘power’, as (at least in my main database) the search results would include hundreds of irrelevant results. Because the literature often varies in the use of terms, the person doing the tagging is likely to be inconsistent in applying the same tag always to similar documents. If in the process of tagging we’ve jumped about among those 4 tags noted above, a search for any one of them, such as “atomic energy” won’t pick up the similar documents that were not tagged that way.

Indeed, tags can be very useful to allow searching of a collection of documents that are in different languages - provided that the tags are applied correctly and consistently.

Frankly, I almost never bother to tag the hundreds of new items that I add to my databases every week, as I enter them. Most of those new items are interesting to me and I do categorize them early on into an appropriate topical database and a group within that database.

Here’s why: to do a good job of tagging a priori, as the item is entered into a database, could take a good deal of time and effort. But I concluded long ago that a priori tagging is a poor investment of my time and energy. Nor (except in an exceptional case) do I want to import tags created by someone else, when I import documents. Why should I trust their tags?

That doesn’t mean that I think tags are not useful and important. When I’m actually working on a research project is the stage when I’ll spend a good deal of effort searching for relevant material and often using See Also to see suggestions for other possibly interesting items. That’s when I use tags, to help pull together resources that are useful in various ways for that project. In the old days, I used to create new groups holding resources for a project and replicate items into those groups. Tagging is less work to accomplish the same objective. Just as I used to delete those project groups containing replicants of resources when the project was finished, I erase the tags created for a similar purpose, when the project is finished. That’s because I found that the resource groups (or tags) created for the purposes of one project are not likely to be useful for the next project, and might even be impediments to approaching my resources from a different perspective.

Tags are important to me for various other uses than in projects. For example, I’ve got a tag set for the writings of a colleague, which also includes a critiques of his work that I plan to respond to, and some of my preliminary notes on a response. As these documents are differently grouped in my database, I created a hierarchical tag group to pull them together for future reference.

I use tags more extensively in my financial database than any other, although that may change with the added capabilities of date range searches (in a forthcoming update of DEVONthink), as smart groups based on date ranges will replace some of my existing uses of tags in that database.

I’ve been lovingly building my main database for years, even before the initial appearance of DEVONthink. I prune it occasionally, removing obsolete or otherwise unsatisfactory references, and use tags to mark items for possible removal.

I usually exclude groups from tagging (File > Database Properties), so I see only tags that I’ve created (or, in one instance, imported). As you can guess, I don’t use a very large number of tags (which helps me apply them consistently).

Disclaimer: If you don’t agree with my perhaps eccentric views about tagging (although the literature of information science doesn’t disagree with me), feel free. Carefully done tagging can distill the information content of documents, and can while away many an hour in hopefully productive ways. :slight_smile:

Of course I agree on the difficulty with subjectivity and consistency. this is one of the problem creating tagging-systems. but then this problem only replicates the very problem that DevonThinks otherwise deals so well with: changing and variable terms included in the text-corpus itself.

also, diminishing these generally valid observations, DT is potentially quite a good software to deal with the problems you describe (which makes it even more tragic that later on in use/search it is so handicapped). what DT has in its favor here is:
– the possibility to create tag hierachies accessible and visible within the folder/group pane allows for good ways to consistently use a personally structured tag-system
– a flexible auto-complete also helps here while tagging (which, as mentioned passes out in the advanced search, at least as a real open/flexible auto-complete or flexible/wildcard string-search is not offered)
– a possibility for theoretically assigning synonyms/“aliases” within the tag-info window (on “theoreticcally” see below)

  • then within DT it is fairly easy to administer and stremaline excessive tag-systems recurrently and progressively (changing tag-names, unify tags).

all this would help for building a real effective tag-system within DT, if it then could be used as an integral (not set-apart, “additional”) system, that is if the tag-system would – at least optionally! - be transparent to the other information-spheres within DT (text, metainformation and so on).

take the case of assigning aliases to a tag, which potentially is very powerful to create systems of synonyms, therby reducing a lot of the fizzling-out-effects you describe: as it is now (given non-transparency of different information in the general search), there is no way to access the tag itself as well as its “aliases” from the same search window. so it becomes useless as alias as no one really knows how to build an effective search using this possibility, being thrown back to a separate “tag” search for which you need exact strings and whch doesen´t encompass the “alias”-field (as well as no other information domain).

the same remains true for a general use of tags as described earlier – a description not adressed by contrasting it with other, more reduced useage-scenarios which do not rely on tags in a more extensive manner.

of course I am aware some people use tags in a specific manner other than the one I describe – as your personal system. the whole point is, the way the tag-system is used should be up to different scenarios – also to those which use tags as part of the core-information – as virtually all other tools using tags allow for/do!
after all, what is the point in integrating a tag system within the very structure of DT in such a prominent way (even putting it in the interface of the windows and setting it on par with other groups) when afterwards tags are seperated off and not retrievable in an easy and intuitive way?? or what is the use featuring tags prominently, but not allowing for a tag-system/-use chosen by the user but prescribed by a specific understanding of tag-systems very singular to Devon Think?

You made some very good comments.

DEVONthink has evolved a great deal over the past nine years and there are plans for major future enhancements.

One of the interesting things about tags in DT is that they are conceptually based on groups, and groups are at the core of the database.

In fact, if you work only with group tags, you can achieve a number of objectives you described. For example, especially in the Split view the hierarchical structure is readily visible.

As it is, also, if one uses the Groups & Tags panel - which also displays the organization of user-defined tags.

I often use the Groups & Tags panel to assign a tag to multiple selected documents and/or to a desired group location.

The Groups & Tags panel is a great way to check the contents of any group or the items that have been assigned a tag. Double-click on a tag, for example, and a view window opens that displays the items assigned that tag.

For some reason, this panel doesn’t seem to get as much attention as it deserves. It’s conveniently available via a keyboard shortcut to toggle it on/off, and I make a lot of use of it.

As the panel floats above other windows, it can be used to drag clippings or files from another application directly to a chosen location in a database.

This may be important. Did you realize that in the full Search window you can choose the Tags group of a database, or a selected tag, on which to base the search query and so filter the tagged items via a query?

Yes we agree on that. and I would hope and look forward that some coming version would make it possible – to return to my usage example and request – to do something like typing “atom energy” in one search field and (either by default or through user preferences/options) and find all documents that contain the strings “atom” and “energy” either in their text body or in some other field (tags, metainformation, alias, iptc, etc).

that is, a search for “atom energy” should also – at least in one possible configuration – bring up all documents containing that string including those tagged “atomic energy” or those tagged “nuclear energy” with the tag containing the ‘alias’ “atomic energy”. which is something you would expect DT as hyper-intelligent search tool to do already, but which is impossible at the moment – though it is a very simple and close by scenario and user expectation ( – I would hold).

so that is my wish/suggestion boiled down to a useage scenario.

(… hoping that meanwhile in the discussion we might have enriched the conceptual understanding of possibilities of tag-groups within DT a little.)