It’s a bit like the annoying kid at school who everyone tells to push off, but they keep coming back just in case you want to talk to them. MSNbot, on seeing my “bugger off” in robots.txt, deserved thanks to its full-on approach to spidering that comes in somewhere between a DoS and a handy revenue generation trick for your ISP, now comes back to double-check the robots.txt file anywhere between every couple of hours and every couple of minutes. Get the message, MSNbot! ‘Op it!

Seriously, though, so many web admins are denying MSNbot spidering in robots.txt that it could find itself with large holes in the index. Or maybe that’s the plan? Make holes in the index, *where there would be sites run by savvy admins*. Leading to more sites run by non-savvy, or bandwidth-ambivalent admins. i.e. people trying to run IIS and Big Corps.

But now I’m just being paranoid.

0 Replies to “msnbot”

  1. – – – = = = H i ! _ G a y s _ T h a n k _ y o u _ f o r _ s u c h _ a n _ i n f o r m a t i v e _ w e b s i t e . . V e r i _ i n t e r e s t i n _ a n _ e a y _ t o _ c o m p r e h e n d – T h a n x ! – – = = A l l , N i c e _ s i t e , _ I _h a v e _ b e e n _ s e e i n g _ s o m e _ r e a l l y i n t e r e s t i n g _ c o m m e n t s . . . _ g o i n g _ t o _ t e l l _ m e _ f r i e n d _ l o u i s e _ a b o u t _ t h i s _ s i t e ! ! = = = – – –

  2. Good morning, I am new to this site. I have just learned about this site. I am going to read on and it’s very interesting to know

  3. Pingback: Diet Pills Death
  4. Pingback: wild flowers
  5. Pingback: mortgage
  6. Pingback: 886
  7. Pingback: chase credit cards
  8. Pingback: asian tits
  9. Pingback: bontril
  10. Pingback: insurance leads
  11. Pingback: rome hotel
  12. Pingback: italy hotels
  13. Pingback: australia betting
  14. Pingback: germany 2006 odds
  15. Pingback: spread betting
  16. Pingback: betting sports
  17. Pingback: uk life insurance
  18. Pingback: w hotel
  19. Pingback: auto insurance
  20. Pingback: american express
  21. Pingback: visa platinum card
  22. Pingback: amoxil
  23. Pingback: experian credit
  24. Pingback: plan b pill
  25. Pingback: hotel deals
  26. Pingback: cipro xr
  27. Pingback: best annuity rate
  28. Pingback: centex mortgage
  29. Pingback: Randy Sizemore
  30. Pingback: denver hotels
  31. Pingback: debit card
  32. Pingback: fat blockers
%d bloggers like this: