msnbot

It’s a bit like the annoying kid at school who everyone tells to push off, but they keep coming back just in case you want to talk to them. MSNbot, on seeing my “bugger off” in robots.txt, deserved thanks to its full-on approach to spidering that comes in somewhere between a DoS and a handy revenue generation trick for your ISP, now comes back to double-check the robots.txt file anywhere between every couple of hours and every couple of minutes. Get the message, MSNbot! ‘Op it!

Seriously, though, so many web admins are denying MSNbot spidering in robots.txt that it could find itself with large holes in the index. Or maybe that’s the plan? Make holes in the index, *where there would be sites run by savvy admins*. Leading to more sites run by non-savvy, or bandwidth-ambivalent admins. i.e. people trying to run IIS and Big Corps.

But now I’m just being paranoid.

0 Replies to “msnbot”

  1. – – – = = = H i ! _ G a y s _ T h a n k _ y o u _ f o r _ s u c h _ a n _ i n f o r m a t i v e _ w e b s i t e . . V e r i _ i n t e r e s t i n _ a n _ e a y _ t o _ c o m p r e h e n d – T h a n x ! – – = = A l l , N i c e _ s i t e , _ I _h a v e _ b e e n _ s e e i n g _ s o m e _ r e a l l y i n t e r e s t i n g _ c o m m e n t s . . . _ g o i n g _ t o _ t e l l _ m e _ f r i e n d _ l o u i s e _ a b o u t _ t h i s _ s i t e ! ! = = = – – –

  2. Hello, a really interesting experience to visit your website. For sure i will come back soon. greets to all !

  3. I’d just like to thank you for taking the time to create this internet website. It has been extremely helpful

  4. Good morning, I am new to this site. I have just learned about this site. I am going to read on and it’s very interesting to know

  5. I finally decided to give you a little feedback ! well you got it! i love your site !!! no , really, its good…

  6. Nice site, I have bookmarked your site yet and I will come back again ! You have a gratest site!

  7. This is a great page. And the contents are really that worth reading. I will add this to my own library

  8. I used this site to get information for that i had in my class. This is an excellent site for this information :)

  9. I just surfed in and found your site, I really enjoyed the visit and hope to come back soon. nice Site!

  10. Greetings Very good web site. I loved it. Found invaluable information. Just what I was looking for :-)

  11. You are invited to check out some relevant information dedicated to- Tons of interesdting stuff!!!

  12. Pingback: Diet Pills Death
  13. Pingback: wild flowers
  14. Pingback: mortgage
  15. Pingback: 886
  16. Pingback: chase credit cards
  17. Pingback: asian tits
  18. Pingback: bontril
  19. outlawing circumnavigate.prohibited?beloved.spectrography contenders.- Tons of interesdting stuff!!!

  20. Pingback: insurance leads
  21. Pingback: rome hotel
  22. Pingback: italy hotels
  23. Pingback: australia betting
  24. Pingback: germany 2006 odds
  25. Pingback: spread betting
  26. Pingback: betting sports
  27. Pingback: uk life insurance
  28. Pingback: w hotel
  29. Pingback: auto insurance
  30. Pingback: american express
  31. Pingback: visa platinum card
  32. Pingback: amoxil
  33. Pingback: experian credit
  34. Pingback: plan b pill
  35. Pingback: hotel deals
  36. Pingback: cipro xr
  37. characterize sinfulness calibrated,benchmark starch stopcock – Tons of interesdting stuff!!!

  38. substrate convincers propeller multiword Dadaistic dabbled?Fredericksburg maintenances

  39. Pingback: best annuity rate
  40. Pingback: centex mortgage
  41. Pingback: Randy Sizemore
  42. Pingback: denver hotels
  43. Pingback: debit card
  44. Pingback: fat blockers

Leave a Reply