Some notes from Roger's workbench.
Put Aggregators on a Diet. As the host of thousands of weblogs on Weblogs.Com, Dave Winer has to deal with one of the big scaling issues with syndication feeds: Once an aggregator subscribes to a feed, it could be checking the file multiple times a day, even when the site hasn't changed in years.
For example, Java.Weblogs.Com hasn't been updated since 2001. A single user who subscribes to its RSS feed could be requesting that 13K file a dozen or more times a day. If the site has 20 subscribers, they could potentially be using 144 megabytes of traffic a month requesting that file.
There are a number of ways to address this problem, and feed providers probably need to adopt all of them.
Charles Miller has written a tutorial on supporting conditional get requests in HTTP. These requests enable a client to avoid receiving a file that it already has been sent in the past, using ETag and Last Modified headers. A client can find out in a 200-byte request that the feed has not changed.
Looking at my logs, it appears that a bunch of popular aggregators support conditional requests, including Bloglines, NetNewsWire, Newz Crawler, NewsGator, RssBandit, SharpReader, and Wildgrape NewsDesk. Radio UserLand does not.
When I request a feed from Weblogs.Com with an HTTP viewer, it looks like Frontier, the server hosting the site, does not send a Last-Modified header. If it did, the traffic burden on the site from moldy RSS feeds should be reduced considerably.
In the feed itself, Dave recommends the adoption of a new XML-based redirect that indicates the feed will never change again by redirecting it to nowhere.
If you're going to point a dead site's feed to nowhere, why not simply delete it? Any decent aggregator will eventually dump a feed that results in "file not found" errors.
Though some aggregators may not support these elements yet, as bandwidth consumption becomes more of an issue, they are likely to adopt an established part of the RSS format. Also, this makes it possible to occasionally check one of these feeds to see if the publisher resumes activity in the future. [Workbench]
