It’s a logical way to get a hierarchy in Smart Groups too.
If smartGroup A relies on smartGroup B and smartGroup B relies on smartGroup A, then you have a circular reference which cannot be allowed.
Err… same for groups? Don’t allow it 
Perhaps I misunderstand your phrase in the title, “search in” - do you mean a smart group whose definition searches in another smart group? You can search in a group. You wouldn’t want to search in a smart group for the reason I gave above.
When you think about it, using one smart group (B) to search another smart group (A) is the same as duplicating the A group and adding one or more new search parameters to it. A simple example: if the A group searches for “Kind is PDF/PS” and if B were to search A for “Data Modified is Today”, then you could have just duplicated A and added the B parameter to it. No?
(OTOH, the business of circular search is a red herring - as Sophie observed, the case would be prohibited by the DT programmer.)
As far as I can tell, this isn’t entirely true. Right now when putting together a Smart Folder you have a choice of making “Any” of the parameters true of “All” of them true. But you can’t nest the different types.
I might want “All” of (“Any” of the items like this in Smart Group 1) and (“Any” of the items in Smart Group 2). Or perhaps I’m wrong and there’s a way to do this?
Tom S.
For those of you who use Sente, you know how smart groups can be really usefully implemented. (NB–I am writing as a still noobie DevonThink user).
In Sente I have the smart groups <status = “Need to find”>, <has tag “At Library”>, <has tag “Online at library”> and <missing tag “Online at library”>.
I can then nest these smart groups and immediately I have a list of everything I need on the library’s computers, and in another nested group everything I need on the library’s shelves!