Your point is well taken. I have no doubt that this is not the best or most efficient way to go. It started with something very simple that I wanted to accomplish, and then over the years, it has blown up into a massive over-engineered monster that I am now very used to and somewhat stuck with. It doesn’t surprise me that I am basically trying to recode (without any talent) something quite complicated that already exists and inevitably doing it badly. I don’t have a coding background, which I’m sure has contributed to a lot of poor decisions. If I knew enough to make use of what exists, I would, gladly.
The main thing the smart groups are intended to group are other groups. If I am writing a proposal, I start the group name with ‘psl’. If the group contains a presentation, I add “.presON’ somewhere in the tag. The tag name contains info about year, organization, person, and other attributes, so when I create the (albeit long) tag, the script checks whether an appropriate smart group exists for each attribute under a main PROPOSAL group and creates one if it doesn’t. Honestly, I love the functionality, even though one could argue it is overly cumbersome to include all the attributes in the tagname. It works for me and I can search for proposals and presentations by year, org, person, etc. The second part of the smart group looks for documents that have a specific set of tags that I define for each prefix (like “psl”) so that they will always show up in the smart group, but the main thing is to group other groups.
If this kind of thing interests you, I would LOVE to get more into the weeds and get some advice on how to improve it. However, at this point, it works well enough except for cracks like the latency that have gotten worse over time. Since it is so complex and I have no desire to argue that anyone else should emmulate it, I am trying to find the path of least resistance to improve it if possible, and accept that there are many ways I could have done it better otherwise.
It appears you have essentially tried to create a system trying to cover all possible permutations of groups and tags. Even if there were created by impromptu scripting, I think it has left your setup in a state where many of the smart groups were one-offs and/or are no longer pertinent.
Given the complexity you’ve already built in and your familiarity and reliance on the setup, I would recommend building in parallel, not trying to improve the current setup in situ. However, you may not have the time and the inertia of your current system could slow or unduly influence a new approach. Those are consideration only you can respond to.
What if you simply freeze the current state of your database? If you start over with your obviously highly sophisticated system in a new database, what would happen?
You would have a new database with the same system, but it would be faster. And if you couldn’t find something in one database, you could find it in the other. Would that be a viable option?
Edit: Ah, sorry, that’s probably what bluefrog already suggested.
I’m also a bit baffled by the statistics.
Do all these views have to be one-click?
Are you aware of the filtering options in DT’s sidebar? For example, this smart rule
seems completely unnecessary to me. You can just open the Info filter and select that label.
The filter affects whatever you’re browsing in the item list, but unlike a search it shows the hierarchy of filtered items instead of a flat list.
I get the sense that you could simplify things substantially by using a few broad categories as points of entry in combination with the Filter tool. (And remember that you can also sort the item list, for example by year.)
These entry points could be a combination of groups, smart groups and tags.
You’re already in the habit of creating these long, complex tags… At least it doesn’t sound like creating the tag itself is automated? If you just split up that long string of attributes into individual tags, you could use them to query/filter with ad hoc.
For example, you could have a proposals tag and add it to your favorites. Select it, then open the Tags filter. If you have tags for all the relevant orgs and people, you can easily filter by those. Or you could filter by a presentation tag (presON, if you like that terminology) to show proposals with a presentation. Or any combination of these.
Another point of entry could be a meetings tag (or group) – again you can use the same approach to quickly find meetings with any person.
The tag filter has a search field where you can type a few characters if you don’t immediately see the tag you want.
It’s also useful to keep in mind that the window’s main search bar only searches the filtered items.
Thank you for all the thoughts and suggestions! They are much appreciated. I will have some time to look at this more in depth this weekend and consider alternatives. In the meantime, I just want to confirm that everyone (seems) to be in agreement that there is no way to use a lot (10,000s) of smart groups within DT4 that is going to viable. Is this true? If I create 10,000 smart groups, it is always going to be a drag on performance? Is there any ability to toggle behavior so that the smart group only conducts the search when the group is opened?
I now understand that there is a cost to each smart group and I am working on a strategy to reduce the number of groups I rely on.
However, I am still trying to understand why it is slow for me to scroll up over groups that have a lot of smart groups even if the smart groups aren’t open or visible. Is there something about scrolling over a smart group that causes it to redo the search? It isn’t slow to scroll up past the highest level groups – only the groups that contain smart groups as immediate children.
So scrolling over this takes several seconds of stalling before I can scroll up in the sidebar, where each group contains 10 or so smart groups:
However, if I collapse this all under one parent group, I can scroll over it easily with no stalling. I am trying to figure out if the stalling issue is due to the smart groups or some other issue.
