Question about Corporate Web-Sites

[cube_rat]

Let's say one wants to create a site that crawls the web looking for specific things, like let's say product information, from the top 20 manufacturers (this criteria for "top" could be sales, revenue, whatever) within a certain geographical area. The results would provide the user with a set of links to queried products on the various corporate sites.

Is there some sort of agreement one must have with the corporate sites to scour their information to be used in an organized way on another site? Or is a corporate's www content open season for crawling and display in one area other than the corporate site? Gawd, I sound so ambiguous, sorry.

I know, I know this already being done by NUMEROUS web-sites. Nothing new technology wise, however, there's one specific area of data on corporate sites that is not being crawled and organized for the end user looking for a specific thing. Hard to believe, isn't it? It's amazingly true. I'm 100% sure I'm not the only one thinking about this either.

 

[wabmester]

Can you imagine if google needed permission from every site they spidered? Ouch.

Check out the robot exclusion protocol, sort of a web handshake agreement that says "don't spider me."

http://www.robotstxt.org/wc/exclusion.html

 

[cube_rat]

Thanks, Wab! I'll be researching...

 

[cube_rat]

I like, I like

If you need stronger protection from robots and other agents, you should use alternative methods such as password protection.