How to write the plugin for searches via google shops

DEVONagent Automation
1

Hello,

for the last hours I tried many different variations, but I can make it work.

Here are the basics I used for developing the plugin:

Next page link:
Page 2
https://www.google.de/search?q=tiffany&biw=1688&bih=977&tbs=vw:g&tbm=shop&ei=nZqYU9WlDPSL4gSK14H4BA&start=36&sa=N
Page 3
https://www.google.de/search?q=tiffany&biw=1688&bih=977&tbs=vw:g&tbm=shop&ei=nZqYU9WlDPSL4gSK14H4BA&start=72&sa=N
Page 4
https://www.google.de/search?q=tiffany&biw=1688&bih=977&tbs=vw:g&tbm=shop&ei=nZqYU9WlDPSL4gSK14H4BA&start=108&sa=N

Sems that direct links have two different structures:
Direct Link to result Link:
http://www.google.de/aclk?sa=l&ai=CO9OLnZqYU5jbEOSpywOZ6YGoDsm5sIUFgf2_974BqOC7jkQICRAdIN7Nzx4oH1Du0r9oYJXqoYKwB6AB0-rr_gPIAQepAtJL49P3LLw-qgQlT9Bj3exJIVLxyfnC1Pz780SMxx65LDta6Drd2s2t4Bwa2ewkZMAFBaAGJoAHlZWUAZAHA-AS1LmftI6Zgt7ZAQ&sig=AOD64_0qY-y99dRcjmx0pwb5LOa_O0cSUQ&ctype=5&rct=j&q=&ved=0CNQDENQpMCA&adurl=http://de.shadestation.com/Tiffany-Co-Sunglasses-TF4055B-80014L-61.html%3Faff%3Daff10%26currency%3DEUR

AND

https://www.google.de/shopping/product/6666849172759907436?sclient=psy-ab&q=tiffany&oq=tiffany&pbx=1&bav=on.2,or.r_qf.&bvm=bv.68693194,d.bGE,pv.xjs.s.en_US.-Ep_RCG9YGU.O&biw=1688&bih=977&tch=1&ech=1&psi=xXOYU_bCOqrO4QTV0YHYBA.1402500038949.7&sa=X&ei=nZqYU9WlDPSL4gSK14H4BA&ved=0CL0DEPICMB4

Based on that I tried several versions, test connected but with a search term no pages were downloaded.

Can you switch on the light for me?

Regards

Joern

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Description</key>
	<string>Searches the Web using Google Shops</string>
	<key>EncodingUrl</key>
	<string>UTF-8</string>
	<key>EngineUrl</key>
	<string>https://www.google.de/search?q=_agentQuery_&amp;biw=1688&amp;bih=977&amp;tbs=vw:g&amp;tbm=shop&amp;ei=nZqYU9WlDPSL4gSK14H4BA&amp;start=_agentOffset_&amp;sa=N</string>
	<key>Identifier</key>
	<string>shops.google.de</string>
	<key>Info</key>
	<string>220 Google Shop Plugin</string>
	<key>Keyword</key>
	<array>
		<string>shop</string>
	</array>
	<key>LinksMatching</key>
	<string>*/shopping/product/*</string>
	<key>LinksNotMatching</key>
	<array>
		<string>*.google.*</string>
		<string>*search?q=cache:*</string>
	</array>
	<key>Name</key>
	<string>220 Google Shops</string>
	<key>OffsetPerPage</key>
	<integer>1</integer>
	<key>Operators</key>
	<integer>100</integer>
	<key>ParseLinks</key>
	<true/>
	<key>ResultsPerPage</key>
	<integer>59</integer>
	<key>Start</key>
	<integer>36</integer>
	<key>Version</key>
	<string>1.0</string>
</dict>
</plist>
2

The main issue is that ParseLinks is enabled but DEVONagent can’t automatically handle Google’s shopping links yet. Therefore this has to be removed and a LinksMatching key is necessary. Here’s a simple draft:


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>CrawDelay</key>
	<real>0.1</real>
	<key>Description</key>
	<string>Durchsucht das Web per &lt;Google Shopping&gt;.</string>
	<key>EngineUrl</key>
	<string>https://www.google.de/search?q=_agentQuery_&amp;biw=1688&amp;bih=977&amp;gbv=1&amp;tbm=shop&amp;ei=1UWZU-eFDuuV7Abs7YDQCw&amp;start=_agentOffset_&amp;sa=N</string>
	<key>Identifier</key>
	<string>com.devon-technologies.google_shopping.plugin</string>
	<key>Info</key>
	<string>Google Shopping Plugin</string>
	<key>LinksMatching</key>
	<string>&amp;adUrl=</string>
	<key>Name</key>
	<string>Google Shopping</string>
	<key>OffsetPerPage</key>
	<integer>20</integer>
	<key>Operators</key>
	<integer>59</integer>
	<key>ResultsPerPage</key>
	<integer>20</integer>
	<key>Start</key>
	<integer>0</integer>
	<key>Version</key>
	<string>1.0</string>
</dict>
</plist>
3

Hello Christian,

while discussing the PDF" Export issue" with the support team we found this entry in the console.log related to the plugin in this discussion.

“2014-06-25 12:46:38.363 DEVONagent[62368:707] Unknown key CrawDelay in plugin /Users/jweber/Library/Application Support/DEVONagent/Plugins/google shopping.plist”

Is there anything to do with this key?

regards

Jörn

4

Just a typo, it should be CrawlDelay.