Strange visitor

Lieutenant Joined: Apr 07, 2003 Posts: 278 Location: USA

This may sound odd, but this week I have a visitor who seems to be sitting in forums or news all day. It is the same IP, and I have tracked it, but I've done nothing because whoever or whatever it is just seems to be sitting. I can't find any evidence of attempted hacking or any changes to my site. I though at first it might be a sessions problem and I deleted the session, but it pops back up again in a few minutes. I'm sure it's a bot, but what it can find on my fairly new site to occupy it for hours at a time is beyond me.

I'm ready to just block it and see what happens, but before I do that I was wondering if anyone has an idea what it could be.

Lieutenant Joined: May 29, 2003 Posts: 231

ladysilver wrote:

I've done nothing because whoever or whatever it is just seems to be sitting.... I'm sure it's a bot, but what it can find on my fairly new site to occupy it for hours at a time is beyond me.

It's very atypical for a bot to just sit. They usually come to your site and check out a page or two and leave right away or stick around and index a bunch pretty fast. People usually keep multiple browsers open. It could be someone who is checking out your site and then using another window to do something else giving you the impression he is sitting idle when he's not.

What's the IP? Bots usually use pretty distinct IPs. If you want, I'll look it up to see if it's a known bot.

Posted: Sat Jun 21, 2003 7:35 am

You may also want into looking around for an add-on called who's online.

It lets the admins know who's online by IP, the registry, and length of visit. Its an admin only tool and its drop and load.

Heres the copyright:
Who is online Admin module v3.0 by Jack Kozbial
http://www.InternetIntl.com

Lieutenant Joined: Apr 07, 2003 Posts: 278 Location: USA

Thanks for answering & for helping. I have the Who's Online admin module and usually bots show up as "googlebot" or "commercial". This one is "unknown domain".

The IP is 131.107.163.59, which resolves to Microsoft, and I thought it might be the msnbot, though I've not listed my site with MSN. But there is no systematic spidering, just long sessions in places that look pointless to spend hours (at least to me).

A small sample from my June 19 log:

131.107.163.59 - - [19/Jun/2003:16:36:25 -0500] "GET /modules.php?name=Private_Messages&file=index&mode=post&u=3&sid=4693e0df0e6d6037144dc7f73d643edd HTTP/1.1" 200 43655 "http://www.elementalmagick.us/modules.php?name=Forums&file=viewtopic&p=54"
131.107.163.59 - - [19/Jun/2003:16:38:06 -0500] "GET /modules.php?name=Forums&file=posting&mode=quote&p=54&sid=4693e0df0e6d6037144dc7f73d643edd HTTP/1.1" 200 43637 "http://www.elementalmagick.us/modules.php?name=Forums&file=viewtopic&p=54"
131.107.163.59 - - [19/Jun/2003:16:50:49 -0500] "GET /modules.php?name=Forums&file=viewtopic&p=54&sid=4693e0df0e6d6037144dc7f73d643edd HTTP/1.1" 200 67697 "http://www.elementalmagick.us/modules.php?name=Forums&file=viewtopic&p=54"
131.107.163.59 - - [19/Jun/2003:17:04:10 -0500] "GET /modules.php?name=Forums&file=posting&mode=quote&p=34&sid=4693e0df0e6d6037144dc7f73d643edd HTTP/1.1" 200 43637 "http://www.elementalmagick.us/modules.php?name=Forums&file=viewtopic&p=54"

I have pages of logged visits from this URL, all exclusively from this past week, all stretching for hours and showing it looking at a few forum posts. I've logged a very few visits to the calendar and to other pages, but mostly it stays in the forum and news. Lol, this website is new (up since April), so there is not that much there that even the slowest reader couldn't digest in an hour.

If I terminate session, which I have done to see what happens, withing 60 seconds it's back again. It's only this URL - everybody else comes and goes normally in my Who's Online.

When I started writing this, it was off. Now a quick look shows it back again. It's just weird.

Lieutenant Joined: Apr 07, 2003 Posts: 278 Location: USA

Ah well, I went ahead and blocked the IP. If it turns out it was some nice person who likes my site well enough to want to spend 14 hours on it, I'll just risk getting my ears burned off in an email. Very Happy

Lieutenant Joined: May 29, 2003 Posts: 231

Yup, that IP belongs to Microsoft and most likely is not a person. Microsoft doesn't have it's own bot to feed MSN search, they actually use someone else's. They have developed their own new bot though and have been doing prototype testing on the web.

Others have also noticed a spider coming to their sites from different Microsoft IPs without a UA or referrer like was done to you. Most of the time, it doesn't check the robots.txt before proceeding. So most people have been banning the little sucker. Afterwhich, it started giving out fake referrers to some folks. I don't know if these bots belong to Microsoft or someone else using their network. The prototype is supposed to give the UA MSNBOT and respect robots.txt, but it's only like in the last few days where a couple folks have started to even see that UA name in their logs. This may be the result of getting banned too much.

At any rate, you might want to flag that one as a temp ban because this bot uses a wide range of Microsoft's IPs. When you ban one or more of theirs, you'll be banning other visitors.

Private Joined: Apr 07, 2003 Posts: 45

Just from what I have read, Microsoft is really gearing up to take on Google for the top search engine on the net. Maybe its a beta bot Very Happy

Posted: Mon Jun 23, 2003 8:33 pm

This is interesting.

The same URL (131.107.163.57) visited my site for over an hour today. I went into the forums admin area and noted that it had visited every single forum.

Couple of hours later: it's back again...kinda spooky in a way Exclamation

Lieutenant Joined: Apr 07, 2003 Posts: 278 Location: USA

I think I will contact Microsoft and see if they can tell me what this is. If I hear anything useful (unlikely, but there is always hope), I'll post it to this thread.

Lieutenant Joined: Apr 07, 2003 Posts: 278 Location: USA

No word from MSN yet - figures - but the little toerag got around .htaccess and I found him/her/it back on the site again. I don't have access to the Apache setup so I can't configure the setting to make it recognize .htaccess (if that is the problem), so I've tried a work-around.

I added this to my_header. php file and it seemed to kick him off:

$banned_ip = array();
$banned_ip[] = '131.107.163.59';

foreach($banned_ip as $banned) {
$ip = $_SERVER['REMOTE_ADDR'];
if($ip == $banned){
echo "You have been banned!";
exit();
}
}

Anyone know for sure if this will work across the site where I have it? Should I add it to header.php or is this likely to be enough to do the trick by itself?

Posted: Tue Jun 24, 2003 5:20 pm

That should work. i have killed alot of bots with a similar script. Heres the deep dish on the IP:

OrgName: Microsoft Corp
OrgID: MSFT
Address: One Microsoft Way
City: Redmond
StateProv: WA
PostalCode: 98052
Country: US

NetRange: 131.107.0.0 - 131.107.255.255
CIDR: 131.107.0.0/16
NetName: MICROSOFT
NetHandle: NET-131-107-0-0-1
Parent: NET-131-0-0-0-0
NetType: Direct Assignment
NameServer: DNS1.CP.MSFT.NET
NameServer: DNS2.CP.MSFT.NET
NameServer: DNS1.TK.MSFT.NET
NameServer: DNS1.DC.MSFT.NET
NameServer: DNS1.SJ.MSFT.NET
Comment:
RegDate: 1988-11-11
Updated: 2002-12-05

TechHandle: ZM39-ARIN
TechName: Microsoft
TechPhone: +1-425-936-4200
TechEmail: noc@microsoft.com

OrgAbuseHandle: ABUSE231-ARIN
OrgAbuseName: Abuse
OrgAbusePhone: +1-425-882-8080
OrgAbuseEmail: abuse@microsoft.com

OrgNOCHandle: ZM23-ARIN
OrgNOCName: Microsoft Corporation
OrgNOCPhone: +1-425-882-8080
OrgNOCEmail: noc@microsoft.com

OrgTechHandle: MSFTP-ARIN
OrgTechName: MSFT-POC
OrgTechPhone: +1-425-882-8080
OrgTechEmail: iprrms@microsoft.com

Let us know if he gets through again.

Lieutenant Joined: May 29, 2003 Posts: 231

ladysilver wrote:

No word from MSN yet - figures - but the little toerag got around .htaccess and I found him/her/it back on the site again.

Those who have had the little sucker visit said he was persistent. They also wrote to Microsoft about its odd behavior (different from yours though) but never got responses back. This is the email address Microsoft has listed on their site for anyone who has questions or problems with their bot: MSNBOT@microsoft.com .

Lieutenant Joined: Apr 07, 2003 Posts: 278 Location: USA

Well, my work around didn't work. It was back on my site again today for several hours, also found it in the error logs looking for "user.php".

Instead of blocking the IP, I've specifically disallowed MSNBOT in robots.txt, which according to Microsoft at http://search.msn.com/msnbot.htm should stop it unless it's buggy. Rolling Eyes

Why would their bot be any different?

Let's see if it's fixed this time (toes and fingers crossed). Very Happy

Posted: Wed Jun 25, 2003 8:28 pm

Is it always the same IP? If thats the case, you may want to run the IP blocking script banning it altogether.

Lieutenant Joined: May 29, 2003 Posts: 231

ladysilver wrote:

Instead of blocking the IP, I've specifically disallowed MSNBOT in robots.txt, which according to Microsoft at http://search.msn.com/msnbot.htm should stop it unless it's buggy.

Well, he kinda sounds buggy if he's staying on the same pages for hours. What did you put in your .htaccess file that he got around it?