Is there a way to use the NOT operator without having it exclude entire documents? For example, suppose I have the file with the following two lines of content text:
The quick brown fox jumps over the lazy dog that is sleeping on the job. The fox is sleeping.
The closest I can come is by using the following syntax:
text:(fox NEAR/9 sleeping) NOT (fox NEAR/3 sleeping).
Ideally, if a file contained both lines 1 and 2, this type of search would return a result containing the first line in the document while ignoring the second line without excluding the entire document from the pool of results.
In reality, this search excludes all files containing a match with the second condition while returning any files containing only the first line/match. As a result, the example file above never shows up.
This is important because Iām working with nearly 100,000 periodicals in my database, and this would be tremendously useful in situations where I am looking for two terms whenever they occur in proximity, but also want to exclude a third term that is muddying and adding hundreds of useless results to my searchā¦ If I want to do a comprehensive study of a topic in newspapers from 1880ā1890, for example, I either have to weed through thousands of muddied file content results or else exclude a lot of files in favor of a more focused, exclusionary search. The ability to combine these and dynamically return file contents that match the entire search string could cut down on 90% of the work.
Hope this all makes sense. Happy to clarify if needed.
Yeah, I understand this is the logical result. This is why I prefaced the post with: āIs there a way to use the NOT operator without having it exclude entire documents?ā
As explained, Iām working with periodicals. Periodicals and newspapers contain many articles on a variety of topics. Searching for historical theological articles on the Old and New Testament provides lots of results but is often also muddied with results tied to the Old and New Covenant. Using āNOTā to exclude the term ācovenantā proves problematic because, in many cases, periodicals that address the discussion of the Old and New Testaments also use the term covenant elsewhere (often in a separate article within the same periodical). So the result is that rather than excluding only articles that address discussion of old/new covenants, I end up excluding entire periodical issues because some other article elsewhere in the file listed the term ācovenant.ā The problem is that this wipes out a lot of good results that I would want to keep. This is why Iām looking for some kind of way to exclude a term when it is in close proximity to others rather than universally.
For academic and archival research, this is a surprisingly common problem when trying to work with tens of thousands of multi-topic periodicals and journals. Having some kind of way to delimit the results with this kind of proximity-based specificity would enable better research and lookups for all sorts of things.
I.e. A NOT/15 boolean operator would be amazing to exclude any results where a term occurs within 15 words. In that regard, this would be a feature suggestion if there is no other way to achieve this currently.
Iāll take it that this is not possible. Do I need to make a feature request somewhere for consideration of NOT/n? I know a lot of apps have vote-based feature request pages.
Iām wondering if old AND (NOT(Covenant near/15 Testament))
might be doing what you want. That should find all documents containing āOldā where āCovenantā does not occur within 15 words of āTestamentā.
Itās a bit difficult for me to gather from your prose what youāre after - perhaps a short example of what you do and donāt want to match helps.
Thanks for the idea chrillek. Unfortunately, when I try this same syntax with test files to ascertain if it will work, it still excludes entire file results if they contain a proximity match between covenant and testament. This is not helpful, because one article in a periodical pdf might contain the result I want to exclude, while another article later in the same file contains the result I want to include. In other words, I want the boolean āNOTā to exclude in-file results from showing when they match the āNOT,ā without removing entire files that may have other in-file matches of what I am wanting to find.
In my case example, Iām wanting articles discussing the old/new testament without reference to old/new covenant because the latter are usually articles outside of the intended scope of research. If a periodical contains one of each kind of article, I donāt want to lose the good one just because the undesired one was present elsewhere in the file.
But then ((old OR new) AND testament) AND (NOT covenant)
should do the trick. Before, you were talking about ādistancesā between words.
Speaking in terms of boolean algebra: You intend to find those PDFs that contain term A, but do not contain term B. Thereās nothing about ādistanceā in this requirement, itās all about existence.
But maybe Iām still not getting it. Iām educated in logic and might fail translating from your description.
And perhaps it would be reasonable to split the periodicals into articles, to make things easier.
Yeah, āNOTā is all about existence in the entirety of the document, unfortunately. So the moment (NOT covenant) is added, all newspapers/periodicals where the term is mentioned even once anywhere will exclude the entire file result entirely.
Iām working with a database of around 100,000 multi-column historical periodicals and newspapers. It is virtually impossible to split them into articles because they donāt cleanly start and end on mutually separate pages (just like modern newspapers where articles can start mid-page, after another article). The amount of work necessary, even if they could be separated, would be astronomical.
So the moment (NOT covenant) is added, all newspapers/periodicals where the term is mentioned even once anywhere will exclude the entire file result entirely.
And this is the logical result of your search. Youāre telling the search engine you donāt want something to be matched but show it to me anyways. Thatās contradictory and would indicate a broken search engine IMHO.
((old OR new) AND testament) AND (NOT covenant)
This would match old testament or new testament and logically exclude documents with covenant in it.
Boolean operators are powerful things but they also donāt lead to a syntax that is intuitive for everyone. And for the kind of search you are referring to, they are required.
Does this look correct, excluding old covenantā¦?
I think @bluefrog and I just demonstrated that it is possible. Boolean algebra is the key to it. But frankly, Iām not sure that this is really what youāre after ā from your previous posts, Iād gathered that you want to include and exclude the same document in the search results simultaneously.
Thatās only possible if you put it in the same box as Schrƶdingerās cat. But even then ā lifting the lid will destroy the illusion.
It is about negating the following predicate. So, NOT (a NEAR/4 b) will hopefully match every document where a and b are not four words or less apart. Including those, Iād suppose, where neither or only one term appears, and of course those where the distance is more than four words. NOT (a AND b) will match all documents not containing a and b at the same time: either a, or b, or none of them. NOT (a OR b) will match all documents containing neither a nor b.
And so on.
All that I was really querying was the possibility of a proximity-based exclusion. The old and new terms merely narrow out the results from including stuff like peopleās last will and testament. In a sense, this begins to accomplish the same thing in a positive way. That alone wasnāt quite enough, which is why I was seeking a way to exclude additional terms in proximity to these without excluding entire files.
If itās not possible currently, thatās fine. Thatās all I was trying to determine in case there was another method that could work to narrow things down. The bigger the database, the more beneficial advanced filter techniques become.
Youāre fighting against how searches technically work versus your conception.
In your example, AND (NOTā¦) is going to disservice your results as it will exclude documents you hope to (not expect to) match.
If you use OR you will get more matching documents, positively matched ones, even if they also include covenant. IMHO this is better than missing documents.
Hereās a question, re-asked if need be:
You want to match old testament or new testament.
You donāt want to match documents with only covenant in them but only all the terms?
Given these documents, which one(s) should be matched?
Yeah, this is what I have been doing. The problem is that when I get, say, 900ā5000 resultsā¦ a lot of them could have been excluded if there was a way to exclude specific terms in proximity to my desired results.
You are correct that in your example, I only want to match document 1 in that scenario.
It gets more complicated, however, because I have additional periodicals in my database of two kinds:
Some have articles on the old or new testament that reference the Abrahamic covenant, or discuss the old and new covenants, which I donāt want to muddy my results. These entire documents would be fine to exclude and Iād be happy with using NOT ācovenantā to exclude these.
But then others can have an article on page 1 which could be about new marriage covenants, or some other use of the term covenant. Then a completely separate article on page 5 is all about the old and new testament. If I excluded the term covenant as in the prior example, the entire document would be excluded and I would miss out on seeing the article on page 5 just because a separate article on page 1 referenced the term covenant.
So, in the end, Iām forced to stick with AND and OR boolean logic, which is fine, but produces lots of additional hits for me to wade through.
Use this: ((new OR old) NEXT testament) OPT covenant
This matches Document 1 not Document 2. covenant will not be highlighted in the search hits. While that lack of highlighting gives you what you say youāre needed, @cgrunenberg would have to weigh in on whether thatās intended or not.
Iām not convinced that the search criteria is at all clear. Basically, they want to find old/new testament and covenant, provided that itās not old/new covenant and that the terms appear in the same article.
Each document can, they said, contain many articles. And there is no clear limit to an article, it seems. But the area of interest is the article. How would anyone write a search expression for that? I can understand that itās desirable, but feasible? I donāt think so. Even with regular expressions (and Iām a big fan of those) ā I donāt see a way.
Whatever search mechanism one uses, they operate on certain regions: a line, a paragraph, a page, a document. Arbitrary, not clearly defined parts of a document are not part of this.
I do understand why the not operator would not work thereā¦ because it would exclude the entire document when part of it may be desired (as with the attachment below). This is why I was querying the possibility of a feature request for an operator that, instead of searching for terms in proximity, excludes results with terms in proximity to others (as in article 1 below). It would essentially be the inverse of regular proximity operators.
