That’s not what the Take Rich Note / Append RIch Note services do.
As @brookter wrote, enable the services (a one-time setup):
- Open System Preferences
- Select Keyboard
- Select the Keyboard Shortcuts tab
- Select Services
- In the right-hand panel, scroll down to the Text section of services
- In the Text section of services, make sure that these services have a checkmark next to them:
DEVONthink Take Rich Note
DEVONthink Append Rich Note
Close System Preferences
Back in DEVONthink, go to a document and select some text with the mouse. While the text is selected do either of the following:
- Open the DEVONthink > Services menu, or
- Press control+mouse click to open the contextual menu, then chose the Services menu there
You’ll see theses services
DEVONthink Take Rich Note
DEVONthink Append Rich Note
Click Take Rich Note to make a new RTF note using the highlighted text. Click Append Rich Note to append the highlighted text to the most recent note that was created by clicking Take Rich Note.
There are default keyboard shortcuts assigned to these actions. Using these shortcuts you won’t need to use the Services menu:
Take Rich Note —> command-)
Append Rich Note —> command-@
Like all text services in OS X, there must be text selected in an application before the service will work. If you are selecting text in a plain text file, you’ll need to use the Take Plain Note / Append Plain Note services instead. If you are selecting text in another application (not DEVONthink), the services will work there, too.
The new note / appended note are created in whatever DEVONthink group has been set in DEVONthink > Preferences > Import > Destination.
It’s simple! Took me 50 times longer to explain than it takes to actually use the feature 
Another possibility, if you have a pre-existing document (RTF or plain text) is to open that document in its own window. In the document from which you want to copy text, select the text with the mouse, and then click-hold-drag that text to the document that is open in its own window.