You are missing our premiere tool bar navigation system! Register and use it for FREE!

NukeCops  
•  Home •  Downloads •  Gallery •  Your Account •  Forums • 
Readme First
- Readme First! -

Read and follow the rules, otherwise your posts will be closed
Modules
· Home
· FAQ
· Buy a Theme
· Advertising
· AvantGo
· Bookmarks
· Columbia
· Community
· Donations
· Downloads
· Feedback
· Forums
· PHP-Nuke HOWTO
· Private Messages
· Search
· Statistics
· Stories Archive
· Submit News
· Surveys
· Theme Gallery
· Top
· Topics
· Your Account
Who's Online
There are currently, 312 guest(s) and 0 member(s) that are online.

You are Anonymous user. You can register for free by clicking here
Nuke Cops :: View topic - Blocking bots from indexing site question [ ]
 Forum FAQ  •  Search  •   •  Memberlist  •  Usergroups   •  Register  •  Profile •    •  Log in to check your private messages  •  Log in

 
Post new topic  Reply to topicprinter-friendly view
View previous topic Log in to check your private messages View next topic
Author Message
mddu
Corporal
Corporal


Joined: Feb 07, 2006
Posts: 57


PostPosted: Wed Jul 12, 2006 12:10 pm Reply with quoteBack to top

I've searched the forum and couldn't find the answers to my question so here goes.

Is it possible to block said bot (i.e. yahoo, google, aol etc) by banning the ip that they come from in Sentinel?

If I do that will it block real visitors who search one of those sites and find my url and click on that link from the search engine?

They're accessing admin only modules and it's frustrating me. I don't care if they index stuff that's viewable to members and guests. I just don't want them indexing things that I'm only allowed to view.

Any help appreciated.

Forgot to add that I'm running php nuke 7.8
Find all posts by mdduView user's profileSend private messageVisit poster's website
perfect-games
Site Admin
Site Admin


Joined: Jun 18, 2004
Posts: 217


PostPosted: Wed Jul 12, 2006 12:54 pm Reply with quoteBack to top

well the best advice i can give is not to block them, becuase google.com etc will contain nothing but you been banned pages.

but i would make a robots.txt

and for example to block them from your admin files do the following

create a file robots.txt or edit current one if you have one. (usally kept in root of the domain where mainfile.php is located

and use the following:

User-agent: Googlebot-Image
Disallow: /admin/
Disallow: /admin/modules/
Disallow: /admin/case/
Disallow: /admin.php

add more as required also change the User-agent to what you want it to block.

hope that helps

Steve
Find all posts by perfect-gamesView user's profileSend private messageSend e-mailVisit poster's website
spottedhog
Captain
Captain


Joined: Apr 30, 2004
Posts: 561


PostPosted: Wed Jul 12, 2006 12:55 pm Reply with quoteBack to top

You posed an interesting yet complicated question....

OK, the first part is, if you block the IP address of the search engine bot, it will not block a web user when they click on your website link within the search engine.

If you wish to get fully serious on what you ask, you may need to fully look at Sentinel, and possibly other ways... see below.

If your website is on a Unix/Linux server, you can work with 2 different files: .htaccess and robots.txt These are located in the root directory of your Nuke install.

The robots.txt is a file that bots are "supposed" to view first to determine what they can and cannot do. However, the vast majority of bots do not bother to read what is in the robots.txt file. These are the bots that are "bad bots".

The .htaccess file is a file where you can do many things within it. Part of that is it can deny IP address from accessing the website. It can also redirect your personal list of "bad bots" to some other website. In this method the .htaccess file views the User-Agent name of the person or bot coming into your website. If you have a "bad bots" redirect list in the .htaccess file, and if that User-Agent is in your list of "bad bots", it will redirect that "bad bot" to the url you put in the .htaccess code.

A word of caution..... the .htaccess file is something very powerful and if things are not coded properly in it, it will shut down access to your website.

The bottom line of what I think you were asking is, sure you can ban the IP of the bad bot, however, that bad bot may have access to more than one IP address. It is probably best to block the User-Agent if at all possible.

By the way.... I just hit the highlights of this particular issue. I left out many of the details...

_________________
SMF-Nuke admin

SMF and PHP Nuke integration is ready! Take a look at it by clicking on the link above.
Find all posts by spottedhogView user's profileSend private messageSend e-mailVisit poster's website
Display posts from previous:      
Post new topic  Reply to topicprinter-friendly view
View previous topic Log in to check your private messages View next topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2005 phpBB Group

Ported by Nuke Cops © 2003 www.nukecops.com
:: FI Theme :: PHP-Nuke theme by coldblooded (www.nukemods.com) ::
Powered by TOGETHER TEAM srl ITALY http://www.togetherteam.it - DONDELEO E-COMMERCE http://www.DonDeLeo.com - TUTTISU E-COMMERCE http://www.tuttisu.it
Web site engine's code is Copyright © 2002 by PHP-Nuke. All Rights Reserved. PHP-Nuke is Free Software released under the GNU/GPL license.
Page Generation: 0.112 Seconds - 465 pages served in past 5 minutes. Nuke Cops Founded by Paul Laudanski (Zhen-Xjell)
:: FI Theme :: PHP-Nuke theme by coldblooded (www.nukemods.com) ::