SEC webpage downloads

[pkoak]

I would like to be able to download all the files pertaining to a particular company from the SEC website and put them into devonthink. I can’t find a way to do it. This is the webpage for a particular company:

http://www.sec.gov/cgi-bin/browse-edgar?company=&match=&CIK=inhx&filenum=&State=&Country=&SIC=&owner=exclude&Find=Find+Companies&action=getcompany

From this webpage there is a link to another webpage which in turn provides a link to the ultimate document. Does anyone know a way to download the documents either directly into devonthink or via devonagent. Idealy it would be nice to be able to isolate the download to particular documents e.g. schedule 14D’s.

Anyone help would be appreciated.

[cgrunenberg]

There are two possibilities. The first one is a script:

property pURL : "http://www.sec.gov/cgi-bin/browse-edgar?company=&match=&CIK=inhx&filenum=&State=&Country=&SIC=&owner=exclude&Find=Find+Companies&action=getcompany"

tell application "DEVONagent"
	set theHTML to download markup from pURL
	set theLinks to get links of theHTML base URL pURL containing "Documents"
	repeat with theLink in theLinks
		set thePage to download markup from theLink
		set theDocuments to get links of thePage base URL theLink type "txt"
		repeat with theDocument in theDocuments
			add download theDocument referrer thePage
		end repeat
	end repeat
end tell

And the second one is a search set:

1. Create a new search set and set the default search term to “*” (without quotes)
2. Add the URL to “Sites” tab
3. Enable following of links (see “General” tab) and enter the follow links term “Documents” (without quotes)
4. Select the “Linked Documents” scanner (see “General” tab)
5. Perform a search using this search set and afterwards run the Scripts > Results > Download Objects of Results

Both solutions should add 38 documents to the download manager.

[pkoak]

Thank you for your reply.

I have tried setting up a search set as suggested but can’t seem to get it to work. When I run the search with * as the query, I do not get any results although it does say "Files:37 under the query box.

Here is the Search Set:

What am I doing wrong?

[cgrunenberg]

Could you please post a screenshot of the Log tab of the search window or export the search set and send it to cgrunenberg - at - devon-technologies.com? Thanks in advance!