Join Early Retirement Today
Reply
 
Thread Tools Search this Thread Display Modes
Question about Corporate Web-Sites
Old 10-13-2005, 06:14 PM   #1
Thinks s/he gets paid by the post
cube_rat's Avatar
 
Join Date: Jul 2005
Posts: 1,466
Question about Corporate Web-Sites


Let's say one wants to create a site that crawls the web looking for specific things, like let's say product information, from the top 20 manufacturers (this criteria for "top" could be sales, revenue, whatever) within a certain geographical area. The results would provide the user with a set of links to queried products on the various corporate sites.

Is there some sort of agreement one must have with the corporate sites to scour their information to be used in an organized way on another site? Or is a corporate's www content open season for crawling and display in one area other than the corporate site? Gawd, I sound so ambiguous, sorry.

I know, I know this already being done by NUMEROUS web-sites. Nothing new technology wise, however, there's one specific area of data on corporate sites that is not being crawled and organized for the end user looking for a specific thing. Hard to believe, isn't it? It's amazingly true. I'm 100% sure I'm not the only one thinking about this either.
__________________

__________________
fuzzy? cute?
cube_rat is offline   Reply With Quote
Join the #1 Early Retirement and Financial Independence Forum Today - It's Totally Free!

Are you planning to be financially independent as early as possible so you can live life on your own terms? Discuss successful investing strategies, asset allocation models, tax strategies and other related topics in our online forum community. Our members range from young folks just starting their journey to financial independence, military retirees and even multimillionaires. No matter where you fit in you'll find that Early-Retirement.org is a great community to join. Best of all it's totally FREE!

You are currently viewing our boards as a guest so you have limited access to our community. Please take the time to register and you will gain a lot of great new features including; the ability to participate in discussions, network with our members, see fewer ads, upload photographs, create a retirement blog, send private messages and so much, much more!

Re: Question about Corporate Web-Sites
Old 10-13-2005, 06:17 PM   #2
Thinks s/he gets paid by the post
wabmester's Avatar
 
Join Date: Dec 2003
Posts: 4,459
Re: Question about Corporate Web-Sites

Can you imagine if google needed permission from every site they spidered? Ouch.

Check out the robot exclusion protocol, sort of a web handshake agreement that says "don't spider me."

http://www.robotstxt.org/wc/exclusion.html
__________________

__________________
wabmester is offline   Reply With Quote
Re: Question about Corporate Web-Sites
Old 10-13-2005, 06:28 PM   #3
Thinks s/he gets paid by the post
cube_rat's Avatar
 
Join Date: Jul 2005
Posts: 1,466
Re: Question about Corporate Web-Sites

Thanks, Wab! I'll be researching...
__________________
fuzzy? cute?
cube_rat is offline   Reply With Quote
Re: Question about Corporate Web-Sites
Old 10-13-2005, 06:31 PM   #4
Thinks s/he gets paid by the post
cube_rat's Avatar
 
Join Date: Jul 2005
Posts: 1,466
Re: Question about Corporate Web-Sites

I like, I like

If you need stronger protection from robots and other agents, you should use alternative methods such as password protection.
__________________
fuzzy? cute?
cube_rat is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to answer questions. ???With a question or assignment????? dex Forum Admin 2 02-25-2007 08:56 AM
Web sites for the RVing Lifestyle audreyh1 Life after FIRE 5 01-30-2007 12:29 PM
Techie question - Low cost web host/registration dex Other topics 6 11-22-2005 07:16 PM
Saving a Web Page - Need a PC Expert! Cut-Throat Other topics 9 10-31-2005 12:01 PM
New cost of having a web presence cute fuzzy bunny Other topics 22 05-30-2004 06:43 PM

 

 
All times are GMT -6. The time now is 04:15 PM.
 
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.