Eric Hartwell's InfoWeb

InfoScraper
Tools and techniques to extract information from web pages and newsletters

Home

Clippings

"Data! data! data!" he cried impatiently. "I can't make bricks without clay."
— Sherlock Holmes to Dr. Watson in "The Adventure of the Copper Beeches" by Arthur Conan Doyle.

"I like deadlines," cartoonist Scott Adams once said. "I especially like the whooshing sound they make as they fly by."

"There is nothing like that feeling of spending days and days banging your head against a wall trying to solve a programming problem then suddenly finding that one tiny obscure and seemingly unrelated piece of the puzzle that unlocks the solution. Oh yeah!"

- Chris Maunder, CodeProject Newsletter 28 Jan 2002

"Management at eSnipe, which is me, is also feeling the pain of the 2002 bear market. So rather than pout about it, I bought some stuff on eBay that I really didn’t need, but made me feel better."

- Tom Campbell, president of eSnipe

Wednesday, October 16, 2002

10:54:01 AM
Email translator update

Based on a few weeks' experience, I've come to realize that while email-newsletter-to-RSS is a fantastic idea, a more-or-less-direct regular expression approach is too cumbersome and fragile.

It's better to build a generic filtering and parsing system, and use custom regular expressions to tag the content types for each newsletter.

item (title + text + link)
list (multiple elements, each possibly with its own link)
story (multi-line text block, may have no link or more than one link)
ignore
end-of-item

Another problem is that I ended up with a bazillion separate rules and RSS files. It would be much cleaner to have a single rule that looks up the parsing info from a table based on the particular email source.

The RSS specification (1.x, 2) only allows one channel per file.

Finally, it would be convenient to post the results directly to Radio Userland.