Please tell me I'm being stupid

DEVONagent Tips
1

HI

Like many souls here, I’m trying to get to grips with the DV suite of goods, and this week it is DEVONagent’s turn. So my aim is to try and keep track of what is being pumped out there that’s relevant to my topic, which for me is underwater aviation archaeology, with info on terrestrial AA also useful. So I I’ve set up two search sets:

  1. (aircraft OR plane) NEAR (restor* OR conserv* OR underwater OR archaeolog*). This is for news sites and academic sites only and is to be used to pick up anything new in the news or academia to do with aircraft restoration, conservation or any connection of aircraft to underwater or archaeology

  2. (aviation NEAR archaeology) OR (aircraft NEAR archaeology). This second one is more general, just searching for aviation archaeology everywhere.

I expected a load for both of these, the idea being that after that initial tsunami, I could then only get new stuff. The problem is that I seem to be getting a lot of stuff that is entirely irrelevant, especially with the first one, so that when I open them, nothing is highlighted, or if it is, it’s just one term. What have I done wrong?

2

Very interesting topic!

Your search term in the first case might be overly broad and catch a lot of junk. You might want to try NEAR/n or BEFORE/n with low n values. Also do a main search such as (aviation NEAR/2 archae* ) and then do a sub search in the results. Look into using scanners.

3

Many thanks for the reply. I actually don’t mind it being too broad, as long as what I’m finding is relevant, but for the first search string in particular, I’m finding stuff that has just one of the search terms e.g. aircraft, whereas the search was aimed at finding those pages that had at least one of each of the terms in parentheses reasonably close to one another.

And yes it is a fascinating topic!

4

Yeahhhh, I think I’m going to give DEVONagent Pro a miss for a while. I’ve tried for a while with it, but it just seems to overload me with irrelevant info, no matter how much I try and tweak it. Sorry.

5

Look at the search “fields” listed in the Advanced tab of your search set. If you are using a proximity operator but see only one of those terms in the downloaded pages, it’s likely because that criterion was met in an ‘invisible’ field such as keywords or description. Uncheck the fields you don’t want to search.

Unless the word separation distance is specified when using a proximity operator, term pairs 10 words apart will be included.

If the jargon of the field usually has term x before term y, using the BEFORE/n (with small n) operator will produce better results than NEAR.

Of course, if the jargon of the field usually pairs the terms x and y, a phrase search for “x y” will be more productive than using a proximity operator.

I find DEVONagent Pro searches often more useful than a Google search precisely because my searches yield fewer results, but well-filtered relevant results. And the results have already been downloaded to my computer for inspection.

6

Many thanks for your reply; I’m using your tips now. I’ll see how they do over the next few days and let you know how I get on.

7

Here’s how I’ve been using DEVONagent recently. All of a sudden DEVONagent has become WAY more useful to me than it has in recent years:

I want to search the web for useful information, but there’s so much stuff out there! I figured I can use DEVONagent to help me out.

The way I use DEVONagent, web sites and pages are divided into three categories:

  • pages that I know about and think might be useful
  • sites that I know about and think might be useful
  • pages and sites that I don’t yet know about that might be useful

I use three key search sets to find and use this information:

  • Pages
  • Sites
  • Search

Pages

This search set doesn’t use any plugins. It contains a list of URLs in the sites tab, all set to “Crawl”. It represents the subset of known web pages that I find potentially useful.

Sites

This search set doesn’t use plugins either. Instead, it has top-level sites listed in the sites tab, all set to “Search”. This lets me search those individual sites for new pages. It represents the subset of known web sites that I find potentially useful.

Web

This search set uses the DuckDuckGo and Google plugins, with no Sites specified. It represents the World Wide Web :slight_smile:

Search set focus and information availability

Each search set has varying degrees of focus and what I call “information availability” – the scope of possible information that they can find.

  • Pages – high focus, low information availability
  • Sites – medium focus, medium information availability
  • Web – low focus, high information availability

Pages can only find the pages that I’ve identified (although with follow links enabled, that’s not quite true… :slight_smile:). Sites can find web pages on any sites I’ve identified. Web can find pretty much any page on the internet.

How I use the search sets

Here’s the basic process I follow:

  1. Do a Web search. Add any interesting pages to Pages search set, and sites to Sites search set. Use the “Search” setting for each site in the Sites set. There’s probably a lot of junk you’ll toss in these results, though the more specific your query the less junk you’ll have.

  2. Do a Sites search. Add any interesting pages to Pages search set. You’re likely to find a higher percentage of useful results, because they’re limited to the domains you’ve previously identified as useful.

  3. Do a Pages search. Work with those results. These are going to be hyper-relevant because it’s limited to web pages you’ve identified as useful.

  4. Rinse and repeat. That’s why it’s called research :slight_smile:

The Pages search represents a known subset of the internet that I find interesting. The Web and Sites search help me find new pages to care about.

The trick

The trick is to save any and all useful sites to your Sites search set, and pages to your Pages search set, even if they’re not related to your current topic. With your Pages search set, you can zoom in on anything you like. Because DEVONagent caches pages, subsequent searches through the Pages set will be really fast. I have a Pages search set with 4500 links in it. The first run through it takes just under 5 minutes, but subsequent runs take only 8 seconds.

A note about archived pages and new results

DEVONagent has some features that ensure you only see results one time – the “Filter archived pages” and “Only new pages” search set configurations. I admit I haven’t gotten into that yet, but I think they will be useful. I would enable those only on the Web and Sites search sets.

8

Good contribution @padillac. It would be nice if DEVONtech would pin this to the DEVONagent forum for permanent reference – as a stand-alone thread without the other topic.

9

@padillac, that’s a great reply, thank you. I have been using the “Filter archived pages” and “Only new pages” search set configurations, plus also making sure “Ignore Cache” is left unticked, but still, so much completely irrelevant stuff.

10

After some use, I’ve ditched the Pages search set in favor of using Safari bookmarks. That lets me save bookmarks on iOS as well, which is nice. I still use the Sites and Web search to find more pages to bookmark.

11

Brilliant strategy.
I will print it out to study.
Thank you, padillac.

.