Posted on by dja

msnbot

It’s a bit like the annoying kid at school who everyone tells to push off, but they keep coming back just in case you want to talk to them. MSNbot, on seeing my “bugger off” in robots.txt, deserved thanks to its full-on approach to spidering that comes in somewhere between a DoS and a handy revenue generation trick for your ISP, now comes back to double-check the robots.txt file anywhere between every couple of hours and every couple of minutes. Get the message, MSNbot! ‘Op it!

Seriously, though, so many web admins are denying MSNbot spidering in robots.txt that it could find itself with large holes in the index. Or maybe that’s the plan? Make holes in the index, *where there would be sites run by savvy admins*. Leading to more sites run by non-savvy, or bandwidth-ambivalent admins. i.e. people trying to run IIS and Big Corps.

But now I’m just being paranoid.

0 Replies to “msnbot”

  2. – – – = = = H i ! _ G a y s _ T h a n k _ y o u _ f o r _ s u c h _ a n _ i n f o r m a t i v e _ w e b s i t e . . V e r i _ i n t e r e s t i n _ a n _ e a y _ t o _ c o m p r e h e n d – T h a n x ! – – = = A l l , N i c e _ s i t e , _ I _h a v e _ b e e n _ s e e i n g _ s o m e _ r e a l l y i n t e r e s t i n g _ c o m m e n t s . . . _ g o i n g _ t o _ t e l l _ m e _ f r i e n d _ l o u i s e _ a b o u t _ t h i s _ s i t e ! ! = = = – – –

  16. Good morning, I am new to this site. I have just learned about this site. I am going to read on and it’s very interesting to know

  41. Pingback: phentrime diet pill
  55. Pingback: where to buy ephedra diet pill
  59. Pingback: site:www.ipsaintl.com Free Online Slots
  61. Pingback: Diet Pills Death
  68. Pingback: wild flowers
  70. Pingback: perscription diet pills
  72. Pingback: tamiflu order canada
  74. Pingback: mortgage
  76. Pingback: 886
  77. Pingback: crisloid championship backgammon
  86. Pingback: chase credit cards
  100. Pingback: asian tits
  108. Pingback: bontril
  114. Pingback: group health insurance
  115. Pingback: insurance leads
  116. Pingback: rome hotel
  121. Pingback: individual health insurance
  122. Pingback: italy hotels
  128. Pingback: australia betting
  129. Pingback: germany 2006 odds
  131. Pingback: spread betting
  133. Pingback: betting sports
  134. Pingback: uk life insurance
  137. Pingback: health insurance quote
  139. Pingback: w hotel
  142. Pingback: auto insurance
  143. Pingback: cheap auto insurance
  146. Pingback: american express
  147. Pingback: visa platinum card
  148. Pingback: amoxil
  149. Pingback: experian credit
  150. Pingback: buy vicodin in mexico
  151. Pingback: plan b pill
  152. Pingback: auto insurance companies
  153. Pingback: extended stay hotels
  154. Pingback: car insurance companies
  155. Pingback: hotel deals
  156. Pingback: cipro xr
  157. Pingback: male enhancement pills
  164. Pingback: whole life insurance rates
  165. Pingback: colonial life insurance
  166. Pingback: best annuity rate
  167. Pingback: centex mortgage
  168. Pingback: business credit scores
  171. Pingback: Randy Sizemore
  172. Pingback: no credit check visa credit card
  174. Pingback: auto insurance quotes
  176. Pingback: denver hotels
  178. Pingback: debit card
  182. Pingback: fat blockers
  183. Pingback: small business credit card application
  184. Pingback: west coast life insurance company
Proudly powered by WordPress
%d bloggers like this: