required terms not in results

I’m not getting search results I want, and I don’t understand.

My DTP queries seem perfectly good, yet I get results that don’t seem to conform. Mostly, the results include many docs without terms I specify.

For example, a query that uses this structure (not the actual terms)

And I got many results without “bricks.”

What could explain this? What am I doing wrong?

What is your searchScope set to under the magnifying glass - All?

This particular search uses a set of 17 sites I’ve set up in a search set.

What’s the search scope (see “Options” tab)? E.g. are keywords/descriptions also searched? Did you have a look at the plain text display of the found results? Just open them in a browser window and choose Web > Display > Text.

I don’t see an “options” tab. I do see those options in the “search in” menu on the tab labeled “advanced.”

It’s set to search in everything. i.e, title, text, URL, keywords, description, and objects. In the “files” section just below, it’s set to search everything except OpenOffice.

I hope that’s what you mean.

Ah, wrong thread! You’re using DEVONthink, not DEVONagent. The highlighting of found occurrences does not yet support the NEAR operator (due to limitations of e.g. the WebKit and PDFkit frameworks of macOS)

No, I’m using DevonAgent. I also use DevonThink, but that’s not where these selections appear.

If, as you say, DevonAgent doesn’t support the “near” operator, what’s it doing in DevonAgent list of operators? See the attached screenshot of DevonAgent (yes, DevonAgent) help. “Near” is not only listed there but it’s been there for years.

It sounds to me like we’re working on different systems.

The search supports the operator, only the highlighting of found occurrences in documents doesn’t (and therefore too many words might be highlighted). Anyway, a real world example (search term plus screenshot of used options plus found URL) might be more useful to check whether there’s a bug or not.

I don’t care about highlighting. All I care about is whether the results conform to the query. As I said in my original question, I get results in which a necessary term doesn’t appear at all.

I’ll post a real world example of results, including a screenshot, a little later.

Thanks, this will be really helpful.

Hello again, Christian.

Here’s a demo of the trouble I’ve described.

  1. First, see the attachment; it’s a screenshot showing the search terms. It begins with the string.

"AtScale" NEAR ("IoT" OR "Internet of Things") OPT "d....

  1. URLs of the three highest ranking results. “AtScale” appears in none of these pages.

Thanks for your help.

I couldn’t reproduce this so far. Could you please post the complete search term? In addition, do you use a secondary search term?

Primary search term

”smart cities"

Secondary search term:

"AtScale" NEAR ("IoT" OR "Internet of Things") OPT "data analysis" OPT "hadoop" OPT "data warehouse" 

I created a new search set with these terms and added the above links to the “Sites” tab. However, no result is returned as expected. Which version do you use? Could you send me a copy of your search set?

It’s version 3.9.5.
smart (369 KB)

I assume you saw the search set I attached and the version number. Any news?

Christian is on vacation for one week. He will have a look at your file as soon as he’s back at his Mac.

Did you use this search set in your examples or did you modify its settings? Because the secondary query is used to match pages and doesn’t contain any required NEAR term, only optional ones:

data OR ("IoT" OR "Internet of Things") OPT "data analysis" OPT "hadoop" OPT "data warehouse" OPT "qlik" OPT ("tableau" NEAR "software") OPT "Ruthbea Clarke"

Thanks. That clarified it. I took your hint and put the whole query in the “secondary” query box and got much better results.

I was confused by this instruction:

"Secondary Query: When you enter something here, the primary term (the one entered or the default query) is only used for querying the search engines, not for accepting or rejecting pages. Without a secondary query, DEVONagent Pro uses the primary query for both querying search engines and post-filtering results.”

This phrase made me think that the primary query had some accept/reject function despite what the phrase after it said: “…primary term (the one entered or the default query) is only used for querying the search engines…”

If “querying” doesn’t mean to reject or accept, then what does it mean?

The primary search term is sent to search engines and the search engines return results based on this search term.

DEVONagent uses the optional secondary search term (or the primary search term if not specified) to match & filter pages on its own.