19 May 2003

Google barred from NYT FREEDOM-TO-TINKER -- The New York Times forbids Google to index its site at the same time the New York Times complains it cannot outperform bloggers on some news items. Go figure--here's the robots.txt file # robots.txt, nytimes.com 4/10/2002 # User-agent: * Disallow: /96 Disallow: /97 Disallow: /98 Disallow: /99 Disallow: /00 Disallow: /01 Disallow: /1996 Disallow: /1997 Disallow: /1998 Disallow: /1999 Disallow: /2000 Disallow: /2001 Disallow: /2002 Disallow: /2003 Disallow: /2004 Disallow: /2005 Disallow: /learning Disallow: /library Disallow: /aponline Disallow: /reuters Disallow: /cnet Disallow: /partners Disallow: /archives Disallow: /indexes Disallow: /guests Disallow: /events Disallow: /features Disallow: /reference Disallow: /specials Disallow: /services Disallow: /thestreet Disallow: /weather Disallow: /RealMedia Disallow: /nytimes-partners Disallow: /books Disallow: /nytstore Disallow: /financialtimes Ed Felten reflecting on The New York Times whinging that Google doesn't give their stories top billing. x_ref119

Quick Links on Printwash GARRINGREEN HOTSPOT -- When Andrew Orlowski slammed "blog noise" in The Register, he created a seismic event in the blogosphere. Some Quick Links show the extent of the ripple. Michael Hall: Let them eat Internet appliances Michael Hall: Above the fold? Where's the fold? Larry Lessig: MediaCon: the Internet Threat Dave Winer comments Dave Winer: If you want to be in Google, you got to be on the Web Bill Brandon: "Once in a while, you get to see the moving finger as it writes. This is one of those times." Bruce Umbaugh: "Doc is having an important conversation." Eric Norlin "really couldn't care less" Fernando Pereira: "The central issue is knowledge, not publishing tactics. Google is an amplifying instrument that makes more obvious the edges of knowledge networks." Dan Gillmor: Google and the visible Web Flemming Funch: "Maybe we shouldn't tell the big corporations about this. I enjoy my splendid Google rankings."  Scott Gray: why blogs percolate Robert Scoble: get rid of webloggers who aren't authoritative Josh K: Blogarama Dan Gillmor: Third Superpower Tuttle SVC: As Google goes The Happy Tutor thinks hyperlinks subvert hierarchies Team Murder is all for voluntary exclusion from search engine results Jim Moore and Second Superpower Neil McIntosh: The Blog Clog Myth Doc Searls: It's positive example time Doc Searls: More on Printwash x_ref26121

Busy weblogs.com slows Googlebot GARRINGREEN HOTSPOT -- While watching the performance of Weblog.com's busy Radio server, I have formed the following conclusions. During whole swaths of the afternoon in Europe, my local copy of Radio cannot upstream. I think that Googlebot cannot crawl my Radio Weblogs site either during these peak times. Sometimes days pass before my local Radio can upload dsily site summaries. Once again, this delays Google's indexing of permalink content. Compared to its indexing service of Topgold, Google lags at least one week behind the speed it crawls a dedicated URL. x_ref119