I have the same issue. I can’t get DEVONThink to use tabbed windows. I can only open new windows. No matter if I try to open a new window with hot keys or vie the menu bar.
Even in full screen mode DEVONThink opens a new window instead of a tab.
Intel MacBook Pro 2020
macOS Big Sur 11.1
Devonthink 3.6.1