Maybe this is possible already but I do not think so.
I would like to create my own ordering algorithm for a Smart Group.
The only way I think this might work would be to create a Smart Custom Metadata concept. The Smart Group could then be ordered by a custom relevance (Smart Custom Metadata) field in the same way that it can be sorted by Name
or Kind
today.
The custom relevance field would be of type Integer
but the value would be assigned to each document/file found in a particular search, based on its relevance quality, as determined by the algorithm.
An advantage of this Smart Custom Metadata approach is the custom field could be used in traditional searches as well. The traditional search would be originally ordered by relevance
but then could be re-sorted by the custom relevance field.
The critical factor for this request would be that the order of items displayed in the Smart Group would be determined by the custom algorithm.
For now, I would implement the algorithm in either something executed by an AppleScript or AppleScript its self.