i have a specific question and can't seem to find the answer

DEVONagent Tips
1

first, i love DT and DA, and use them both every day. nothing helps me organize my research better!

my question:

i would like, with devonagent or devonthink, to go to an aggregator website like aldaily.com/ (arts & letters daily, google news is another example), and to just be able to search through (or archive!) all of the linked articles from that front page.

i tried to set devonagent to search aldaily.com on sites, and to follow 2 levels of links… didn’t work the way i wanted it to…

can anyone help?

2

I took a look at that Web page, which has hundreds of links. In turn, many of those links are to external newspapers and magazines that would have thousands of links. Furthermore, the page is constantly changing on a daily (perhaps even more frequent) basis.

So I have to ask what you want to accomplish.

In principle, every day you could tell DT Pro to create a download archive of the site to 2 levels depth. That would quickly fill up your hard drive (I suspect terabytes in just a few weeks) and would cover every imaginable topic that might be in, e.g., the newspapers and magazines that would be downloaded. Why? That’s not unlike trying to download the entire Internet.

More likely, you have some specific topical interests that can be defined in multiple DEVONagent search queries. In that case, create multiple DA search sets that use the URL for the site, and give each a query such as “Proust” or “Western Civilization” or Shakespeare" and so on. Set up those searches to mine the site with daily schedules and archive the search results. By so defining and searching for topics of interest, you can use this site as a source for information of interest to you.

3

so, when i said google news i made the scope for this tool too big. i was thinking of aldaily.com, which has one page of links to articles, updated once a day. i thought having it automatically archive articles that contain some key phrases would be nice :slight_smile:

another thing that would be nice is to be able to archive backarchives for academic journals (like policy review magazines at: policyreview.org/pastissues.html)

my goal is to have a pretty large database of articles/written pieces that i can process into evidence cards that look kinda like this:

I/L - CHINA’S RURAL POLITY IS KEY.

[b]CHINA’S ECONOMIC WELL BEING STILL RELIES ON A RURAL SAFETY NET SET UP BY LAND REFORMS THAT TOOK PLACE UNDER DENG.

Bardhan, 2005[/b] (Pranab Bardhan is Professor of Economics at the University of California, Berkeley, and co-chair of the MacArthur Foundation-funded Network on the Effects of Inequality on Economic Performance. He is Chief Editor of the Journal of Development Economics., “China, India Superpower? Not so Fast!” YaleGlobal, 25 October 2005)

China’s authoritarian system of government will likely be a major economic liability in the long run, regardless of its immediate implications for short-run policy decisions. In the economic reform process, the Chinese leadership has often made bold decisions and implemented them relatively quickly and decisively, whereas in India, reform has been halting and hesitant. This is usually attributed to the inevitably slow processes of democracy in India. And though this may be the case, other factors are involved. For example, the major disruptions and hardships of restructuring in the Chinese economy were rendered somewhat tolerable by a minimum rural safety net – made possible to a large extent by land reforms in 1978. In most parts of India, no similar rural safety net exists for the poor; and the more severe educational inequality in India makes the absorption of shocks in the industrial labor market more difficult. So the resistance to the competitive process of market reform is that much stiffer.

so, it requires a lot of research, organization, which DT has helped me loads with, i just wish i could “up my production.” :smiley:[/u]

4

The page I looked at was aldaily.com, which, with it’s long list of external links, is a pretty big window into the Internet. If that site were organized differently, it would be much easier to manage for archiving. It’s not the main content of articles that causes problems. It’s all those darned external links to newspapers, magazines and the like that make a 2-level search potentially enormous.

The other site you mention, Policy Review, is much better organized for your purpose. Two-level searches of the Archives link could pull results quickly. As I noted, one approach would be to create several new DA Search Sets that would each have a topical query.

Personally, I spend several hours a week going through the journals that I’m interested in and selecting material to place into DT Pro. In an online weekly journal such as Science Magazine I don’t want to download everything, but there are always some articles that fit with my reference collections.

You can of course save the DEVONagent digests to your drive (click on the Digest button then select File > Save As). That’s one way of producing summaries. And if you transfer the full HTML pages of articles from DA to DT Pro, DT Pro can also produce summaries. Note: You may wish to experiment with Sentence or Paragraph summaries.

5

thanks bill. i love devonthink :slight_smile:

keep up the good work!

6

Jeffromartin,

open the search sets panel via “Tools > Edit Search Sets…”. Then…

  1. Create a new search set
  2. Select the “Sites” tab and add the location “www.aldaily.com”
  3. Select the “General” tab and activate “Follow Links” (one level should be sufficient)
  4. Enter “more” (without quotes) in the field below the follow links slider
  5. Disable the filter for similar pages
  6. Set the default query to “*” (without quotes)

Now close the search sets panel, open a new search window, select the new search set and start the search.

BTW: To archive the results automatically, you might also activate the “Add results to DEVONthink” action in the search sets panel.

7

oh, there it is. thank you so much. that is exactly what i was looking for. :slight_smile: