To whom this may concern:
I would like to apologize for any inconvenience we may have caused. We
are committed to ensuring that MSNBot behaves appropriately. MSNBot has been designed to be polite and obey the Robots.txt exclusion protocol so this is quite possibly a bug on our part. If you would be willing to
share with us your robots.txt file as well as your log file that would
be help us identify the issue and correct it.
Thank you very much and we apologize for the inconvenience.
Eytan Seidman
MSN Search
MikeM, here's what I have in robots.txt and .htaccess. htaccess seemed to work for a couple of days, then it was back again. The php ban worked all of one day. robot.txt doesn't seem work at all with this bot, although according to MSN if you disallow MSNBOT that will stop it indexing.
The htaccess is barebones. I didn't use Options or RewriteEngine because until now I haven't had a problem with a bot. I don't want to block legitimate visitors from MSN, so I've blocked the specific URL. This is the only URL the bot uses on my site.
Quote:
<Limit GET POST>
order deny,allow
deny from all
deny from 131.107.163.59
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
deny from 131.107.163.59
</Limit>
AuthName www.elementalmagick.us
AuthUserFile /my virtual path/my password file
AuthGroupFile /my virtual path/my service group file
The last 2 lines above were changed for public posting and are not the way they are written in htaccess.
The php work around is posted earlier. I tried an experiment with the workaround, adding it to files where I found the bot sitting and terminating the bot's session to force it to reaccess the files and (hopefully) find it cannot. Did not work. This thing is worse than a roach.
Am I not doing something right or is this bot buggy?
MikeMiles Lieutenant
Joined: May 29, 2003
Posts: 231
Posted:
Thu Jun 26, 2003 3:07 pm
I'm glad they wrote back. I hope you send Microsoft the info they want so they can fix the bot. I don't know why your .htaccess and php ban would work and then suddenly not unless the bot changed IPs on you. Others have said this bot will do that when you try to ban it.
Your robots.txt looks okay to me. I don't use the limit rule but how about trying it this way:
<Limit GET POST>
order allow,deny
allow from all
deny from 131.107.163.59
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
If you see this bot jumping to another IP in the same range, you can use partial IPs like this: 131.107. or 131.107.163. (with a trailing dot). Just realize you'll be cutting off a lot of other people too. That's why banning a good bot that's acting really weird should be done temporily until it goes away or is fixed. Most of the bad bots you can permanently ban and not effect others. You're lucky if this is your only roach. My roaches are mostly spybots. Unfortunately, I had to add a few more of them to my ban this last week.
ulissesnelson Lieutenant
Joined: Apr 09, 2003
Posts: 188
Posted:
Sun Jun 29, 2003 1:59 am
crap i posted another topic first and then i saw this,seems that if they attemp to fix the bot they dont fixed. i just installed the ip_tracking after a hard change in my nuke site ,and i get nuts when i checked the ip_traking to see if was working ok,that f*** ip appeard and in 1 minute ,he hited 100 times my statistics module.
Code:
User (A\D) IP Address (A\D) Host Name (A\D) Last Viewed (A\D) Hits (A\D)
admin ***.**.**.** pr5-ts.telepac.pt 2003-06-29 11:00:44 17
80.142.155.25 p508E9B19.dip.t-dialin.net 2003-06-29 10:51:17 4
131.107.163.49 131.107.163.49 2003-06-29 10:41:21 100 <--there he is
ladysilver Lieutenant
Joined: Apr 07, 2003
Posts: 278
Location: USA
Posted:
Sat Jul 26, 2003 6:57 pm
Here's another odd one: I was checking my referrers when I came across a couple of entries for XXXX:+++++++++++++++++++++++++++++++++++
The nuke referral module and the database couldn't provide any more info so I pulled the server log files and got this:
The whois leads to Verizon. Any ideas what info this ip was trying to get?
Zhen-Xjell Nuke Cops Founder
Joined: Nov 14, 2002
Posts: 5939
Posted:
Sat Jul 26, 2003 7:00 pm
I'm going to bed, its been a long day, I just had to say nice avatar.
_________________ Paul Laudanski, Microsoft MVP Windows-Security
CastleCops: [de] [en] [wiki]
MikeMiles Lieutenant
Joined: May 29, 2003
Posts: 231
Posted:
Sat Jul 26, 2003 8:34 pm
No idea why the X's and +'s. If it's Verizon, it's probably a regular surfer. Which pages do your logs show him accessing (if any).
ladysilver Lieutenant
Joined: Apr 07, 2003
Posts: 278
Location: USA
Posted:
Sun Jul 27, 2003 10:49 am
Thanks, Zhen-Xjell.
MileMiles, it looks like it was a normal index page call. I thought the referring URL was odd and I was wondering if someone did this thinking it would obscure their url in the referrer list and I wouldn't be able to trace them, or if it was something else.
_________________ In a world without fences or walls, there is no need for Gates or Windows
ladysilver Lieutenant
Joined: Apr 07, 2003
Posts: 278
Location: USA
Posted:
Sun Jul 27, 2003 10:51 am
BTW - I heard nothing back from Microsoft after I sent them the log info, but their bot is staying off my site.
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
