Roland Piquepaille's Technology Trends
How new technologies are modifying our way of life

jeudi 3 avril 2003

This article starts with a quote from Douglas Adams.

The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong, it usually turns out to be impossible to get at or repair.

It is true that machines are becoming more complex and "intelligent" everyday. Does this mean that they can exhibit unpredictable behavior like HAL, the supercomputer in 2001: A Space Odyssey? Do we have to fear our PCs?

The last book by Thomas M. Georges, Digital Soul: Intelligent Machines and Human Values, explains how our machines can develop neurosis and what kind of therapy exist.

Autonomous, goal-seeking machines that can reprogram their own goals and subgoals could, in effect, develop "minds of their own" and set off in unpredictable directions. If they create goals that make no sense whatsoever to us, then we may see those choices as "crazy." If you think that nutty people can wreak havoc, just imagine the potential for chaos when a supercomputer in charge of some critical aspect of our lives gets confused about its goals and purpose in life.

A neurotic behavior is usually associated with conflicting instructions sent to our brains.

So what would be the machine equivalent of a neurosis? Imagine that you are driving down a highway in your car and, slowly at first, you begin to apply more and more pressure to both the gas and brake pedals simultaneously. You notice the car's reaction, as it "tries to cope with" the conflicting forces that are simultaneously trying to speed it up and slow it down.

Easy to understand, but what will be the equivalent for our PCs?.

Your PC may experience conflicting demands when you try to run multiple programs, if its operating system is not well designed to handle such conflicts. The result may be that one or more of your programs will behave erratically, or the whole PC may shut down. Because its goal-seeking capabilities are very primitive, however, this is as close as your PC can get to becoming neurotic.

Is there a treatment for a PC's neurosis?

A cognitive approach to machine neuroses would create self-monitoring systems that scan for inconsistent or dangerous orders and would set corrective actions in motion.
If a machine can receive conflicting instructions that will cause confusion about its goals, the machine must have monitoring programs to, first of all, detect conflicts and inconsistencies and then to take corrective action. It could be trained, for example, to deal with certain classes of conflicts by questioning the apparently contradictory instructions or by requesting further information. For conflicts not specifically provided for, the machine could respond unpredictably. Since the causes and remedies of "crazy" machine behavior will eventually lie beyond the understanding of humans, the solution to Douglas Adams's dilemma posed at the beginning of this chapter may well require built-in mechanical psychologists and psychiatrists.

In this article, Georges also looks at the benefits and dangers of nanotechnologies. All in all, a highly recommended article.

Source: Thomas M. Georges, in Darwin Magazine, April 2003

11:53:03 AM  Permalink  Comments []  Trackback []

Click here to visit the Radio UserLand website. © Copyright 2004 Roland Piquepaille.
Last update: 01/11/2004; 11:44:33.

April 2003
Sun Mon Tue Wed Thu Fri Sat
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30      
Mar   May

Search this blog for

Courtesy of PicoSearch

Personal Links

Other Links

Ars Technica
Daily Rotation News
Microdoc News
Smart Mobs


Dave Barry
Paul Boutin
Dan Bricklin
Dan Gillmor
Mitch Kapor
Lawrence Lessig
Jenny Levine
Karlin Lillington
Jean-Luc Raymond
Ray Ozzie
John Robb
Jean-Yves Stervinou
Dolores Tam
Dylan Tweney
Jon Udell
Dave Winer
Amy Wohl

Drop me a note via Radio
Click here to send an email to the editor of this weblog.

E-mail me directly at

Subscribe to this weblog
Subscribe to "Roland Piquepaille's Technology Trends" in Radio UserLand.

XML Version of this page
Click to see the XML version of this web page.

Technorati Profile

Listed on BlogShares