_________________ Paul Laudanski, Microsoft MVP Windows-Security
CastleCops: [de] [en] [wiki]
MikeMiles Lieutenant
Joined: May 29, 2003
Posts: 231
Posted:
Mon Aug 11, 2003 5:29 am
ia_archiver is Alexa's bot. It's primary mission is to make a huge archive of the net. It feeds the waybackmachine at http://www.archive.org/ . The results of its indexing usually do not show up for about six months. Alexa also runs a third-rate search engine at http://www.alexa.com/ which is powered by Google with some of the archive pages peppered in the results showing up as "related searches." Alexa also takes the information it indexes off of everyone's websites and sells it to other people.
IMO this bot is a bandwidth hog which doesn't return that much traffic for the amount it eats. They sell your webpages' content and images (copyrighted material) to others without your permission and without giving you a commission. This bot has always been on my ban list. If you're an Amazon affiliate, their TOS forbids you from banning it because they use it to do spot checks for their affiliate program. Some here like this bot, but I don't.
The second one crawls HTML documents, pictures, video, and audio. It's a bot used for FAST Web Search: http://www.fastsearch.com/us/products/fast_web_search/crawler_faq which powers the All The Web Search engine and a bunch of small search engines primarily located outside the U.S.
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
