It’s a bit like the annoying kid at school who everyone tells to push off, but they keep coming back just in case you want to talk to them. MSNbot, on seeing my “bugger off” in robots.txt, deserved thanks to its full-on approach to spidering that comes in somewhere between a DoS and a handy revenue generation trick for your ISP, now comes back to double-check the robots.txt file anywhere between every couple of hours and every couple of minutes. Get the message, MSNbot! ‘Op it!
Seriously, though, so many web admins are denying MSNbot spidering in robots.txt that it could find itself with large holes in the index. Or maybe that’s the plan? Make holes in the index, *where there would be sites run by savvy admins*. Leading to more sites run by non-savvy, or bandwidth-ambivalent admins. i.e. people trying to run IIS and Big Corps.
But now I’m just being paranoid.
