Nathan Wallace: RSS is sticky

Nathan's right, the problem is the stickiness of RSS and the fact that the aggregators automatically poll the servers every hour (some are really bad and poll more often).

Here, let's talk about a population of 1000 people. Let's assume these are all developers and are all interested in MSDN. I would assume their usage pattern might break down something like this:

20% will visit at least once a day
40% will visit at least once a week
20% will visit at least once a month
20% will not visit in any one month (assuming these folks visited before but just aren't revisiting)

These are made up stats for the purposes of explaining the RSS bandwidth issue.

So, if my math is correct, per month we'll see the following hits:

200 x 30 (visits per month) = 6000
400 x 5 = 2000
200 x 1 = 200
200 x 0 = 0

But, what happens when these folks use an RSS news aggregator?

Let's assume all 1000 will subscribe to the RSS feed.

Now the usage model changes quite a bit

20% will leave their aggregator on all day long pulling down the feed once per hour (200 people x 24 pulls a day x 30 times per month) = 144,000
20% will leave their aggregator on at least one hour a day, pulling down the feed at least once (200 x 1 x 30) = 6,000
30% will leave their aggregator on at least a few hours a week (let's say five hours) (300 x 5 x 5) = 7,500.
10% will only pull down their feeds once per month (let's say five hours a month) (100 x 5 = 500).
10% won't turn their aggregator on at all. 0.

So, that comes out to 8,200 hits for HTML vs 158,000 for RSS. Again, if my math (and theories/assumptions) are correct.

This gets worse over time because on most sites HTML traffic will go down as people move away (at least until the site reposts interesting content that'll bring back more traffic) while RSS just grows and grows even if new content doesn't get posted because people subscribe and don't move away.

Another problem is aggregators. I'm hearing some are pinging too often. How about we do a survey of what the defaults are on your favorite aggregator? Here, I'll start:

1) NewsGator: every 60 minutes.

ClickZ: the blogger on the payroll.

Funny thing is, am I really blogging on the payroll? I blog only on my own equipment and only at night when I'm at home and away from work. Thanks to Steve Rubel for the nice quote. Speaking of which, Steve was here for dinner a week ago with his fiance, who's a school teacher in New York City. Great guy and sure is changing the world of PR.

IBM's Ed Brill pays me a compliment. "Scoble isn't paid nearly enough for his contribution -- no, creation -- of a community around the Microsoft technologies he espouses."

Oh, oh, the competition is onto me. Thanks Ed, appreciate the compliment. Like your blog too, although I don't understand people who think Lotus Notes is great stuff. But, I agree with you about Dan Gillmor's book.

While on vacation I did keep up with reading my feeds. I figure that in the past five days there've been somewhere around 6,000 to 8,000 items published. I scan them all (reading probably at least part of 75% of those). Here's a tip: write better headlines and you'll get more readers. Joel Spolsky's headlines are the worst. His RSS items are headed by just a date and tell you nothing about what he's writing about. It probably doesn't matter for him, though, because everything he writes is a "must read" item. But for the rest of us who aren't as smart as Joel we gotta write better headlines (particularly if RSS is going to be chopped the way it is on my link blog).

Anyway, when I got home my "blog this" folder had 315 items in it. Yeah, that's a lot, but looking through my link blog I find I'm constantly inspired, prodded, challenged, and increasingly informed about what's going on in the technology industry. And it sure is more efficient to skim 315 items than it is to skim 6,000. At least the noise level is very low -- no cat photos, I promise!

Tejas Patel notes that Google wants to get into instant messaging. He also takes an inventory of what Google's done already.

Adam Bosworth, who now works at Google (he pioneered Microsoft's XML efforts) is working on "stuff mom can use."

And Joe Beda left Microsoft last week to go to work for Google (he worked on Avalon stuff for Longhorn).

Hmmm, what is Google up to? (I have my guesses but don't want to give these smart guys any more ideas than they already have).

Steve Maine: what the hell happened to blogs.msdn.com?

RSS is broken, is what happened. It's not scalable when 10s of thousands of people start subscribing to thousands of separate RSS feeds and start pulling down those feeds every few minutes (default aggregator behavior is to pull down a feed every hour).

Bandwidth usage was growing faster than MSDN's ability to pay for, or keep up with, the bandwidth. Terrabytes of bandwidth were being used up by RSS.

So, they are trying to attack the problem by making the feeds lighter weight. I don't like the solution (I've unsubscribed from almost all weblogs.asp.net feeds because they no longer provide full text) but I understand the issues.

I know of a major broadcaster that refuses to turn on RSS feeds because of this issue too. We need smarter aggregators and better defaults.

I only pull down RSS feeds once per day -- right before I start reading my feeds.

But, clearly, RSS is losing some of its advantages. More and more sites are not providing full-text feeds. I can't fight this one alone.

Hope you had a nice long labor day vacation. Mine was fantastic. Went to Boston/Montreal/Quebec.

On Saturday, had a very geeky day. First, got another famous-geek-sits-next-to-Scoble-by-random-chance story. On Saturday morning our friends and Maryam went to breakfast in the South End of Boston before hitting the road to go to Montreal. The place we chose was crowded, so we sit at the counter. I notice the couple next to me are redesigning something. How do I know that? Cause they have a bunch of pages with lorem ipsum text (filler text). So, I start up a conversation.

Turns out it's Jason Pontin, former editor in chief of Red Herring magazine, and is currently editor in chief at MIT's Technology Review.

How weird is that? Turns out they are redesigning the magazine and switching to Adobe's InDesign. He tells me that they are working on a story for a future issue about the coming war between Google and Microsoft.

Anyway, once up in Montreal we had one of the more fun geek dinners. Peter Stathakos has pictures and a writeup. The group gave us a great walking tour of Montreal. Lots of fun.