Schooliblit

Saturday, March 13, 2004

Wading in the Deep Web
In search of the deep Web

The next generation of Web search engines will do more than give you a longer list of search results. They will disrupt the information economy.
- - - - - - - - - - - -
By Alex Wright
March 9, 2004 | When Yahoo announced its Content Acquisition Program on March 2, press coverage zeroed in on its controversial paid inclusion program, whereby customers can pony up in exchange for enhanced search coverage and a vaunted "trusted feed" status. But lost amid the inevitable search-wars storyline was another, more intriguing development: the unlocking of the deep Web.
Those of us who place our faith in the Googlebot may be surprised to learn that the big search engines crawl less than 1 percent of the known Web. Beneath the surface layer of company sites, blogs and porn lies another, hidden Web. The "deep Web" is the great lode of databases, flight schedules, library catalogs, classified ads, patent filings, genetic research data and another 90-odd terabytes of data that never find their way onto a typical search results page.
...
The deep Web contains some 500 times more data than the surface Web; but to regard the deep Web as simply a bigger and better version of the current Web is to overlook the essential feature of databases, which is structure. Most of the deep Web is structured or semi-structured data, as opposed to the sea of flotsam HTML that bobs across the surface Web.

This is something that librarians and information scientists have known about for a long time, but knowledge of the "deep web" or the "invisible web", as some call it, is beginning to move into the mainstream, and with that knowledge are coming new ways of searching and of accessing all that information. This is just more of what I've remarked on a couple of times recently, and it puts another line under that question: how do we teach people to navigate and evaluate information as the flood gets ever deeper?
6:53:50 PM

[comment []];[]