Sipping from the ocean
This morning's news on NPR led off with a long piece on technology and the use of data mining techniques on the collection and processing of information in the war on terrorism. An interview with one of the FBI agents who had some but not all of the information on Middle Eastern men attending flight schools was featured.
The story is tied to a feature on the NPR web site called Homeland Security in the Information Age.
The points made in the story and my elaborations could probably take several days and several hundred pages, but the following are just a couple of the high points. They have a lot of meaning for knowledge management systems in law firms and in business:
The FBI's inability to use and connect the pieces of information they had was apparently due to an old tech failure that I call the "database mentality". Early efforts at information processing were based on the database model. The imbued prejudice in favor of databases was due to the infatuation with the power of sifting, filtering, and sorting the information contained in the archive. The weakness of the database mentality was (and is) the enormous amount of time and effort that it takes to squeeze information in its raw state into the database so that analyzing it can begin. Sometimes odd-shaped pieces have to be forced into square holes. The FBI agent described a common failure: she could input the single word "flight" but not the phrase "flight schools". The ability to process information was limited by the imagination of the database architect, who restricted his fields to single-word entries and could not think of any reason to do otherwise.
The need to know, and know quickly, what is in a newly encountered lode, such as a laptop computer, means that the tools that the miner uses must be able to process the contents in their raw state. Systems which depend upon the conversion of data from one format to another before it can be made useful lose a lot in time, expense, and accuracy. For information processing, software which can read and extract from a large number of formats is much more useful than that which can only recognize one or a few.
One of the key elements of any information processing system is retrieval. Storage of terabytes of electronic information can itself be overwhelming if the tools for finding and retrieving the information are inadequate. Note the use of the commonly-encountered phrase "drinking from a fire hose".
"An analyst from South America would get 1,400 messages a day," [says a U.S. Representative and former Naval Intelligence officer]. "There are huge vacuum cleaners in the sky sucking in information for the foreign intelligence agents, but sometimes we get so far behind in collecting information that we find it too late, as we did with Sept. 11."
All of this information has always been in existence in the wild, but it has not been collected in any kind of information reservoir until the use of computers and the internet to permit computers to be connected in networks. The technology that creates the reservoir can find ways to either direct the flow in meaningful ways or to permit the user to sample freely at will, but it requires the use of particular tools that give the maximum flexibility to the user to do so. The paradigm for useful access for the internet -- the ultimate collection of oceans of information, miscellaneous and directed, random and hierarchical, related and unrelated -- is Google. There are several programs such as dtSearch which take the same approach to the location and retrieval of nuggets of information from source material in its raw state on hard disks and local and wide area networks.
7:06:35 AM 
|
