Stupid Human Programming
Talk on software development.

Subscribe to "Stupid Human Programming" in Radio UserLand.

Click to see the XML version of this web page.

Click here to send an email to the editor of this weblog.

Monday, December 25, 2006

The Solution to C++ Threading is Erlang

This amusing and disturbing post on C++ Threading paints C++ going the same way Java went, into a corner. Having been on more than a few large distributed programming projects I have kept careful tally on how many people can get large complex threading projects correct. After a long and careful analysis the results are clear: 11 out of 10 people can't handle threads. It gets too complex too fast. I do not have fMRI studies showing brains literally melting when thinking about multi-threaded programs, just many many anecdotes.

I recall fondly one of our most experienced engineers creating a lock that was off by one instruction and how that 1 in a million error took weeks of debugging by a tiger team to find. The release was eventually canceled. I recall how I took a lock on a data structure that when the system scaled up in size lasted 100 milliseconds too long which caused backups in queues throughout the system and deadly cascade of drops and message retries. Very embarrassing, but even more difficult to anticipate. I can recall how having a common memory library was an endless source of priority inversion problems. These same issues came up over and over again. Deadlock. Corruption. Priority inversion. Performance problems. Impossibility of new people to understand how the system worked.

Yet when it all worked it worked so beautifully the pain was almost worth it. Almost. It's like the image of the City on the Hill. Almost within our earthly grasp, but just out of reach. Yet we keep striving, hubris driving us ever forward in her many threaded chariot.

One reaction to the threading nightmare is to ban the use of threads in favor of a process based architecture. Multiple single threaded processes will be started instead of fewer threaded processes. The advantage is single threaded code is refreshingly simple to write. You don't have to worry about locks or shared state or corruption. There's much less dread of a pager going off at 2AM to handle a customer's 5-9s application that was declared dead of unnatural causes.

Process based architectures are not without faults. Scaling is handled by adding more machines, which isn't possible in embedded applications and is costly in a data center. And you tend to push complexity elsewhere. Where is shared state stored? In a threaded application it would be in the same process protected by locks. In a process architecture you either have to 1) connect to a data store, 2) keep shared memory, or 3) publish the shared state to interested parties. This becomes even more interesting when an application must act on multiple information streams with low latency. Threaded applications are good at this, process applications are not.

So going to one extreme of an all process based architecture works well for some applications, but not all. We can go to the other extreme and use our Posix thread library and let everyone do their own thing. That way lies the madness we touched on earlier.

There's a middle ground that is highly performant, easy for programmers, and difficult to mess up. The middle ground is provided by Erlang. Erlang is a functional language and clearly C++ is not, but that doesn't matter. The distributed message passing model used by Erlang works perfectly for C++, in fact, I've used something like it in many very successful projects.

What does Erlang do that is so special? It has one mechanism for handling concurrency: processes. It creates lightweight processes that are not mapped to OS processes and your application communicates between processes using messages. Within a process there is no locking or any of the other complexities associated with threaded programming. Your data structures don't need locks. You don't have to worry if someone is scribbling over your memory. You write clear simple code. The above link talks more about Erlang and how it works. I don't want dive too deeply now.

BTW, you can simulate much of Erlang in C++ using Actors, and I talk about how in my Architecture page, but without language support it's not safe. A programmer can always go around your library and cause disaster. Safety requires language support.

The important point is that C++ can do something really interesting and powerful in the threading area that will make C++ an exciting language to work in again. They don't have to do the same old thing in the same old horrible way.


9:18:42 AM    

Click here to visit the Radio UserLand website. © Copyright 2007 todd hoff.
Last update: 1/18/2007; 1:10:23 PM.
December 2006
Sun Mon Tue Wed Thu Fri Sat
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
Jul   Jan