I've searched the forum and couldn't find the answers to my question so here goes.
Is it possible to block said bot (i.e. yahoo, google, aol etc) by banning the ip that they come from in Sentinel?
If I do that will it block real visitors who search one of those sites and find my url and click on that link from the search engine?
They're accessing admin only modules and it's frustrating me. I don't care if they index stuff that's viewable to members and guests. I just don't want them indexing things that I'm only allowed to view.
Any help appreciated.
Forgot to add that I'm running php nuke 7.8
perfect-games Site Admin
Joined: Jun 18, 2004
Posts: 210
Posted:
Wed Jul 12, 2006 12:54 pm
well the best advice i can give is not to block them, becuase google.com etc will contain nothing but you been banned pages.
but i would make a robots.txt
and for example to block them from your admin files do the following
create a file robots.txt or edit current one if you have one. (usally kept in root of the domain where mainfile.php is located
add more as required also change the User-agent to what you want it to block.
hope that helps
Steve
spottedhog Captain
Joined: Apr 30, 2004
Posts: 566
Posted:
Wed Jul 12, 2006 12:55 pm
You posed an interesting yet complicated question....
OK, the first part is, if you block the IP address of the search engine bot, it will not block a web user when they click on your website link within the search engine.
If you wish to get fully serious on what you ask, you may need to fully look at Sentinel, and possibly other ways... see below.
If your website is on a Unix/Linux server, you can work with 2 different files: .htaccess and robots.txt These are located in the root directory of your Nuke install.
The robots.txt is a file that bots are "supposed" to view first to determine what they can and cannot do. However, the vast majority of bots do not bother to read what is in the robots.txt file. These are the bots that are "bad bots".
The .htaccess file is a file where you can do many things within it. Part of that is it can deny IP address from accessing the website. It can also redirect your personal list of "bad bots" to some other website. In this method the .htaccess file views the User-Agent name of the person or bot coming into your website. If you have a "bad bots" redirect list in the .htaccess file, and if that User-Agent is in your list of "bad bots", it will redirect that "bad bot" to the url you put in the .htaccess code.
A word of caution..... the .htaccess file is something very powerful and if things are not coded properly in it, it will shut down access to your website.
The bottom line of what I think you were asking is, sure you can ban the IP of the bad bot, however, that bad bot may have access to more than one IP address. It is probably best to block the User-Agent if at all possible.
By the way.... I just hit the highlights of this particular issue. I left out many of the details...
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
