It’s a bit like the annoying kid at school who everyone tells to push off, but they keep coming back just in case you want to talk to them. MSNbot, on seeing my “bugger off” in robots.txt, deserved thanks to its full-on approach to spidering that comes in somewhere between a DoS and a handy revenue generation trick for your ISP, now comes back to double-check the robots.txt file anywhere between every couple of hours and every couple of minutes. Get the message, MSNbot! ‘Op it!

Seriously, though, so many web admins are denying MSNbot spidering in robots.txt that it could find itself with large holes in the index. Or maybe that’s the plan? Make holes in the index, where there would be sites run by savvy admins. Leading to more sites run by non-savvy, or bandwidth-ambivalent admins. i.e. people trying to run IIS and Big Corps.

But now I’m just being paranoid. has moved

I’ve now got this site running on my new server at Bytemark which I’ve been quite impressed with, so far. The only thing that I’ve noticed as a little slow is compilation of new stuff (I’m using Gentoo, so that’s obligatory) but what do you want with 64Mb?! That, of course, can be upgraded… Even Tomcat seems to run OK (no real testing done as yet).

Check Bytemark out if you want full root access to a well-connected server at a decent price.