Teeny Tiny But Cool New Feature

I've been meaning to put this in -- Smart Add -- since, well, forever.  What I did was make it so that if you try to add an RSS feed on Feedster's Add page and its already there then it lets you see every post for that blog (if you want to).

Try It

10:20:48 PM  | Permalink:   | comment []  |  IM Me About This   

Proposing a New Standard of Blogging Currency: The Zawod

Someone just pointed out to me that the FuzzyBlog is now worth 30,000 on Blogshares.  Go figure.  I didn't have a clue.  Now I see that Jeremy Zawodny is worth 60,000 and Scripting.com is worth 280,000 and I was thinking that "Gee I always wanted to be 1/2 as popular as Jeremy" so how about this unit scale:

1 Zawod = 60,000

So I'm 1/2 a Zawod right now and Dave is roughly 4 /13 Zawods.  Comments?

9:12:52 PM  | Permalink:   | comment []  |  IM Me About This   

Is There a Market for This?

I just witnessed David Weinberger's problems getting desktop Linux working for himself and I was wondering if there's a market for Home User Linux Support.  Yeah you can always call RedHat but I wonder if people actually do that.  I'm thinking also of myself.  With the help of pr00fster I got my dual monitor graphics card running but I still don't have the resolutions I want (one screen is 1280x1024 and the other is capable of 1600x1200 but only works at the lower res) and I don't have a window manager installed that's really dual monitor friendly*.  So I'm wondering if there's a market for this.  Personally I'd promote it just by answering questions on the different desktop Linux newsgroups and mailing lists and having a signature that marketed yourself.

Thoughts?

*But I do have to admit that it is a hoot and a half when I get a dialog box split between the two screens.

9:05:44 PM  | Permalink:   | comment []  |  IM Me About This   

Feedster Has Been Throttled !  Sigh...

Its not often that you find me face down on the couch sound asleep at 3 pm on a Sunday afternoon -- but that's where I just was.  Basically I spent from 6 am to 3 pm (yeah 9 **expletive deleted** hours) dealing with a huge amount of incoming performance issues.  My previous entries describe that ***expletive deleted*** experience.  So what we had to end up doing was limiting incoming aggregators to no more than 25 queries per rolling 5 minute window*.  Why those numbers?  Not really sure except that 5 is prime and I kinda like it.  And once I had that then the 25 queries were natural.  And these seemed reasonable.  If you hit us with more than this then we're going to deliver you a message saying to email us for a enhanced (and currently non-existent) license key.

If these limits get in the way, etc, drop me a line (or preferably IM me) and we'll work it out.  And I know people bitched about it when Slashdot did this and for the first time I can see their perspective.  A constant flood of aggregator requests does amount to a server attack.  This is a big issue for RSS performance in the months and years to come.  I'll add some thoughts on these to our developer wiki.  What's that you say?  Didn't know we had a wiki?  Well that's coming probably tomorrow.

Oh and I have to give a big thank you to www.Scottah.com and "Mike Down Under" (Australia) who did their very best to keep me sane.  Its just astonishing how helpful it is to have someone you can rant to while something bad is going on.  And while all this was going on, Mike and I engaged in pair programming via IM on a fairly tricky little string processing algorithm that we needed and I was just too fried to work on.  Mike ran with it gratis and a few things came out:

• Mike's good -- I basically kibitzed rather than did anything substantiative
• I'm hugely grateful
• Mike's interested in starting his own web dev firm in Australia and if anyone needs a good developer in that part of the world drop me a line I'll hook you up
• I'm encouraged by using IM as a pair programming medium.  I've never seen anyone talk about this and I was surprised how well it worked.  Since I work pretty muchy exclusively from home and am always on IM, this could be very good for me.

*Now we did the lazy ass coward's approach to stateful http tracking and if you're geeky then you understand that statement and the requisite limits. 

5:49:43 PM  | Permalink:   | comment []  |  IM Me About This   

Similarity Searching Now Available !

If you do a Feedster search now, you'll see a link in the result set "similar posts" (next to "cached") and, when selected, it uses the selected article to return all other articles like it.  Right now this is, unfortunately, hard wired to english.  And it operates by content analysis not by link analysis.  We had actually planned to release this later this week but we did a massive update dealing with the DDOS issues and it just happened to sneak out into the wild when we moved stuff from our dev box to our production box.  Ah well.

1:17:53 PM  | Permalink:   | comment []  |  IM Me About This   

Scaling RSS : The Feed Vendor's Point of View

Feedster is essentially a "feed vendor" w/ more than 50% of incoming searches coming from aggregators.  I'm seeing that Dave and Phil are talking about scaling RSS now and I thought it would be smart to kick in my .02 from the Feedster perspective.  This morning was difficult to near impossible to get to Feedster due to a combination of factors -- one seemingly intentional DDOS from a cloaked IP address AND a huge number of aggregators that were each sending in queries and when they didn't get the results fast enough, what did they do?  Why they sent them again for heaven's sake!  And then again!  And again!  And right now if you send us an RSS query, we're going to send back a offline page.

So now I'm implementing "RSS Throttling".  Sigh.  And while I haven't isolated down the user agents yet, when I do find them, I'll have a real serious desire to send back a scolding to the aggregator like "I'm sorry -- this aggregator is a greedy child trying to resend queries too frequently.  Bad Aggregator!  Why don't you try (well mannered aggregator here)".  And I'd hate to do this.

Something for aggregator vendors to keep in mind -- when you're sending out a feed request, don't assume that because the URLs are different that the resulting IP address is different.  And its just plain rude to hammer webservers.

12:57:40 PM  | Permalink:   | comment []  |  IM Me About This   

Unhappy Feedsterness

Well it is a Sunday so at least that is in our favor.  Since about 6 am-ish this morning we've been getting what effectively (scratch that -- it is) amounts to a DDOS (distributed denial of service) against our server cluster from a user or users with masked IP addresses.  And we just haven't been able to stop it since their damn IP addresses are masked.  Here's a snipped from our logs:

206.49.81.28 - - [20/Jul/2003:10:24:09 -0500] "GET /ads/index.php/serve/1 HTTP/1.0" 200 9057 "http://www.feedster.com/search.php?hl=en&;ie=UTF-8&q=oracle&sort=date" "Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt)"

www - - [20/Jul/2003:10:24:34 -0500] "GET /rss.php?q=feedster&sort=date HTTP/1.0" 200 713 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"

Note the missing IP address.

Any thoughts ?  Help !!!

11:30:51 AM  | Permalink:   | comment []  |  IM Me About This   

To All Feedster Advertisers

This morning we got essentially ddos'd by having gotten severely hit by the Google bot, the Inktomi bot and several others that all decided to arrive concurrently.  And a malformed robots.txt file did no do its job.  This means that your metrics, particularly for our CPM customers, are, well, porked (that would be the polite way of saying what I feel right now).  Not to worry -- we'll make this up to you.

-- Scott

8:20:23 AM  | Permalink:   | comment []  |  IM Me About This