You are missing our premiere tool bar navigation system! Register and use it for FREE!

NukeCops  
•  Home •  Downloads •  Gallery •  Your Account •  Forums • 
Readme First
- Readme First! -

Read and follow the rules, otherwise your posts will be closed
Modules
· Home
· FAQ
· Buy a Theme
· Advertising
· AvantGo
· Bookmarks
· Columbia
· Community
· Donations
· Downloads
· Feedback
· Forums
· PHP-Nuke HOWTO
· Private Messages
· Search
· Statistics
· Stories Archive
· Submit News
· Surveys
· Theme Gallery
· Top
· Topics
· Your Account
Who's Online
There are currently, 64 guest(s) and 0 member(s) that are online.

You are Anonymous user. You can register for free by clicking here
Nuke Cops :: View topic - Google crawler [ ]
 Forum FAQ  •  Search  •   •  Memberlist  •  Usergroups   •  Register  •  Profile •    •  Log in to check your private messages  •  Log in

 
Post new topic  Reply to topicprinter-friendly view
View previous topic Log in to check your private messages View next topic
Author Message
MikeMiles
Lieutenant
Lieutenant


Joined: May 29, 2003
Posts: 231


PostPosted: Wed Jun 04, 2003 3:54 pm Reply with quoteBack to top

sharlein wrote:
Is this normal behavior or did I do something wrong?

Yes, it's normal behavior. The 216.* IPs are Google Deepbots and the 64.* IPs are Freshbots. The Freshbots do the same thing on my site. Sometimes they access only one page and other times they will index a bunch. They come around quite frequently.

Google makes one full crawl of the entire web about every 6 weeks with their Deepbots. From this crawl, the pages go into their permanent index. Depending on how its programmed, the bot may do all your site in one visit or come back multiple times over a number of days.

The Freshbots are sent out daily. They're like little scouts whose mission is to find new sites and look for new information on already indexed ones to help keep Google's database fresh during the periods between the full crawl. They appear to prefer new pages over updated ones except on heavily linked sites whose content is constantly changing. The pages crawled by these little guys go into a temporary index and will usually be dropped in a few days. Google is always experimenting on ways to improve how it indexes sites, and they will use Freshbots to test out their new code.

If your pages are getting indexed okay by the Deepbots then don't worry about it. If your pages are not, then you should be concerned. Freshbots are like a bonus. The pages they crawl though have a short life span in Google's index.
Find all posts by MikeMilesView user's profileSend private message
Display posts from previous:      
Post new topic  Reply to topicprinter-friendly view
View previous topic Log in to check your private messages View next topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2005 phpBB Group

Ported by Nuke Cops © 2003 www.nukecops.com
:: FI Theme :: PHP-Nuke theme by coldblooded (www.nukemods.com) ::
Powered by · TOGETHER TEAM srl ITALY http://www.togetherteam.it · DONDELEO E-COMMERCE http://www.DonDeLeo.com
Web site engine's code is Copyright © 2002 by PHP-Nuke. All Rights Reserved. PHP-Nuke is Free Software released under the GNU/GPL license.
Page Generation: 0.148 Seconds - 202 pages served in past 5 minutes. Nuke Cops Founded by Paul Laudanski (Zhen-Xjell)
:: FI Theme :: PHP-Nuke theme by coldblooded (www.nukemods.com) ::