Lean development.
Mary Poppendieck's book Lean Development, to be released in June this year, seems very promising after having read the introduction and the first chapter (of the review copy). As I understand it, lean development comes from lean manufacturing, which is an approach invented by Japanese automobile companies, such as Honda and Toyota, in the early 1980's, to compete with American car makers.
Lean manufacturing is concurrent rather than sequential; decisions are made as late as possible; inventory is kept to a minimum (the ultimate goal is to not do anything until there's a customer order); critical decisions aren't pushed up the hierarchy, but made by the engineers themselves; a single leader envisions what the car should be, and constantly communicates this to the engineers.
The car design environment, writes Poppendieck, is as complex as in software, and the lean approach has been tried and proven there. She lists seven principles of lean thinking: Eliminate Waste; Amplify Learning; Decide as Late as Possible; Deliver as Fast as Possible; Empower the Team; Build Integrity In; Avoid Sub-Optimization. These all match to chapters in the book, so I'll write about them in turn.
Waste is anything that doesn't add value to the end product, such as documentation that no-one reads, extra features put in on speculation, defects, etc. Delay is also waste, waiting for people to decide, for other programmers to complete code that you depend on, switching between tasks. The chapter discusses a technique to map the value stream , how much time is wait and how much time is actually spent on value-adding activities. Poppendieck illustrates this with the map for a can of cola to travel from mine to mouth (from the book Lean Thinking by James P. Womack and Daniel T. Jones). The process takes 319 days, of which processing accounts for three hours. The point is to identify where the delays are, and reduce or eliminate them.
I'm currently reading the chapter on amplifying learning; in it, Poppendieck compares development to creating a recipe, while production is like following a recipe. A chef creating a recipe needs to try many variations to discover the best. Software similarly to a large extent about trial and error. She quotes a study made by Raymonde Guindon, in which
she found that cycling between high level design and detailed solution was typical of good designers when dealing with ill-structured problems, that is, problems which do not have a single right answer or a best way to arrive at a solution. Guindon theorized that this unstructured approach is necessary to understand and give structure to such problems.
Poppendieck continues: The bulk of the work of software development is a problem-solving activity similar to that investigated by Guindon. In other words, since creativity is about problem solving, programming is a creative activity, where it's not a question of simply following set rules and practices – rather, it's an exploration of the unknown.
Lean development #2.
In chapter two, Amplify Learning, Mary Poppendieck writes that the preferred method for solving complex problems is to use the scientific method. You observe, create a hypothesis, then devise an experiment to test the hypothesis. The interesting thing, she writes, is that to maximize learning, you have to have a 50% probability of failure when testing your hypotheses. In other words, if your hypotheses are always correct, you don't learn as much. It is by failing that you learn. This speaks loudly in favor of experimentation in software development, either by creating prototypes or (which is even better) by writing real code.
One thing I didn't know: you often see the familiar Fred Brooks quote: Plan to throw one away, you will anyway. What he referred to was that, in order to learn enough to develop a system, you need to create it twice: first you create an experimental prototype, then you create the actual system. However, he retracted this twenty years later, saying that it implied a sequential process, a waterfall process – which he says is wrong: to succeed, you need an incremental process, with progressive refinement. I guess that some parts of a system will be clear the first time around, while some parts will have to be rewritten a few times.
Poppendieck quotes Ed Yourdon as saying that any piece of code needs to be rewritten three or four times to be consistuted as an elegant, professional piece of work. For prose, we take this for granted, he says, why not for code?
Today, iterative, adaptive processes are widely regarded as the most suitable for software development. But Poppendieck writes that upon trouble, we are inclined to turn to a more disciplined process, that in fact is sequential, or waterfall: we first want the requirements to be signed off by the customer, then we consider them frozen and start implementing, resisting any changes the customer wants, and so on. I feel that this is true. This, she continues, contradicts control theory, since you are actually lengthening the feedback loop. What you must do is to shorten it: do shorter iterations, with more frequent feedback from the customer; accept new requirements only at the start of the iterations, and so on.
With too short iterations, you get what in control theory is called thrashing – that is, that the system (the team of developers) never has time to react to feedback until new feedback arrives. At the other extreme, with too long iterations, you get oscillation – when you act on incoming feedback, reality has plenty of time to deviate until feedback arrives again, so the course corrections are too big. The appropriate iteration length depends on local circumstances, though. You'll have to try and see what works. My gut feeling is, however, that it is better to start with shorter iterations. For XP, the recommended length is 1–3 weeks, and I'd say that many are inclined to pick 3 weeks in the beginning, but I feel that they would be in a better position to judge after having tried one-week iterations. I guess that if you are unfamiliar with agile processes, you feel that there will be too much time in meetings with so short iterations.
Lean development #3.
The end of the chapter on amplifying learning in a software project is brilliant. Mary Poppendieck talked about increasing feedback as a response to trouble, rather than increasing discipline, reverting to a waterfall model. She continues to talk about disruptive technologies – a term coined by Clayton Christensen in his The Innovators Dilemma and that is defined by Business 2.0 as any technology that overturns a traditional business model . It has to do with emerging markets, and Poppendieck writes about how established companies fail to enter emerging markets because of the nature of disruptive technology.
I'm not sure I fully understand disruptive technology and how it relates to punctuated equilibrium, but I do understand Poppendieck's point. She talks about how established companies fail because of their having a rational approach to decision making, when circumstances call for an intuitive approach, due to the complexity and fluidity of the situation. Here, she talks about firefighters as intuitive decision makers, something I have been thinking about. Something that was new to me was that a firefighter, at times when his experience is inadequate, resorts to rational decision making. This must be the same mechanisms that lies behind the tendency to seek a more disciplined process when meeting challenges in a project.
Rational decision making removes context, she writes. It suffers from tunnel vision. When I read this I came to think (again) of Alan Watts talk about the floodlight and the spotlight (see here).
Then she writes about U.S. Marines, citing a book called Corps Business by David H. Freedman, and this I find very interesting. Marines are trained to deal with complexity, with paradoxical and contradictory circumstances. Their training aims to expose them to more challenging situations than they will ever experience in reality, using situational training to build intuitive decision making skills.
For marines, mistakes are considered necessary. First they make plans to understand the essence of a situation. Then the organizational structure is collapsed, leaving the decisions to the men at the front line, which are less likely to make mistakes than the remote officers, since they are where the action is. The principle with the training is to expose them to situations where they are given a goal, but also the freedom to decide the details in fulfilling that goal. The focus is on small teams at the lowest levels, rather than way up the hierarchy.
I think the essence of this has to do with three things: communicating a clear statement of intent to the team, give them a common vision or goal; the freedom to find the best way to fulfill the goal; and that this is done using a set of simple rules or guidelines.
Poppendieck introduces the simple rules by talking about social insects, swarm intelligence, and restates the seven lean principles as these simple rules: Eliminate Waste (do only what adds value to the end product); Amplify Learning (when in trouble, increase feedback); Decide as Late as Possible (keep your options open); Deliver as Fast as Possible (in order to get feedback soon); Empower the Team (freedom, not orders from above); Build Integrity In (integrity must be paid attention to from the start); Avoid Sub-Optimization (worry only about the performance of the whole). I'm excited, because these take a different approach than the twelve practices of Extreme Programming. I can't explain why, but I have strange feelings about the latter two rules; we'll see if I discover what that's about.
Lean development #4.
(More on Lean Development by Mary Poppendieck.) She writes about pull systems , systems that are pulled by need rather than pushed by schedule or plans. For example, in manufacturing, there's a technique called kanban , where a kanban card is left as a part is taken from inventory. Workers then take the kanban card and sets out to make a new part, as specified by the card. The part might consist of subparts, which means that new kanban cards are left for these, causing a ripple across the system, creating a whole slew of new parts, all by demand.
Software development is often a push system, where plans and schedules are made to decide what to do. XP, however, would be more of a pull system, since you don't actually work by schedule; rather, you identify a set of features (user stories ) for an iteration, that are too represented by cards. These cards are preferably kept visible, so that everybody can see what remains to do, what is being worked on for the moment, and what is done – for example in the form of a bulletin board with three sections, To Do, Now, and Done .
Poppendieck talks a lot about depth-first and breadth-first . This is generally a preference: some people are inclined to prefer depth-first, others prefer breadth-first. But when faced with a new problem, most people tend to approach it depth-first, trying to get a sense for the details. Earlier in the book she quoted Raymonde Guindon, who had discovered that good designers often cycles between details and the whole (see my first post on this book). Depth-first requires a sense of the whole, or your assumptions of what details to focus on might be wrong. As I experimented with finding XP metaphors, I discovered how having a metaphor made it easier to return to the whole when I had gotten stuck on some detail – I just tried to restate the problem in the vocabulary of the metaphor.
Lean development #5.
(More on Lean Development by Mary Poppendieck; available as a review copy here.) She talks about queue theory, which was quite interesting. At banks you often have one queue for all tellers, and in shopping centers there's often an express lane for people with 10 items or less. At a bank you can't anticipate which customers will take more time, as you can in a shopping center. In software projects, queues form for example when features are handed over to testers, or when a customer decides which features should be implemented.
To prevent congestion, Poppendieck writes, you need to move things around at a steady rate (little variability), in small batches. This translates to fixed iterations (better to defer features to the next iteration than to extend the current iteration with a few days) that aren't too long (a few weeks). Slack is also important, and as I interpret this, the reason is that as a feature moves from one station to another, it should be likely that the person there is ready to take care of it (that is, no queues). The last thing you need to do is to move variation downstream . I'm not sure I understood this, but I think it means that it's better for queues to form late in the project, than early (since it tends to accumulate for each cycle, resulting in longer and longer queues). She uses the analogy of people leaving a football match, queueing to leave the stadium, to enter the parking lot, driving out of the parking lot, and finally as they are about to enter the freeway. In this case, it would be better if queues only formed at the freeway entrance. I guess the total queue time is lower.
With iterations, though, the end of an iteration might be downstream within the context of the current iteration, but it is upstream for all future iterations. In other words, long queue times in acceptance testing is not acceptable, since the results of acceptance testing is feedback to the next iteration.
Lean development #6.
(More on Lean Development by Mary Poppendieck; available as a review copy here.) We'll see if I manage to finish this post before going to the meeting at 9. The fifth chapter, titled Empower the Team , is very interesting, and reading it generated lots of associations. I thought about how I consider XP, as it is described in Extreme Programming Explained, to be an implementation of an architecture . You have a set of core values (simplicity, communication, feedback and courage) which, under particular circumstances, gains form as twelve practices.
Therefore, doing by-the-book XP is just wrong. It's not what XP is about. However, unless you are perfectly aware of what you are doing, starting by following the book is highly recommended. But without constantly evaluating your process, trying to improve it and adapt it to the local circumstances, you won't experience as great benefits. I'll have to go to the meeting now, but I'll definitely return to this as soon as possible. This is interesting stuff.
Lean development #7.
(Continuing my unfinished post from yesterday.) So, doing by-the-book XP is wrong (unless, magically, your circumstances happen to match those of the primordial XP project). It's a starting point; after a few iterations, you will begin adapting it to your organization.
Poppendieck writes about an automobile plant that was a real disaster: alcohol problems both on and off site; strikes; high sick leave rates. It is closed down, only to re-open under mixed management between GM and Toyota. The bulk of the previous workers return. The plant turns into a huge success: productivity is higher than any other GM plant; quality is much higher; the problems are essentially gone and the 90% of the workers say they are satisfied, or very satisfied.
The decisive factor seems to have been the fact that the workers are reorganized into small teams with the power to self-organize, self-manage. Previously, there were trackers measuring their effectivity; after the reopening they were trained to measure themselves and left with the freedom to find out how to improve. (This makes me think of the Engines of Democracy article in Fast Company that I have blogged about earlier.) But the interesting thing was when GM wanted to transfer this success to their other plants, something that they according to Poppendick have been unable to do.
Poppendieck is careful to point out that there are no simple answers to this question, but one interesting clue is that they might just have tried to clone what had evolved at the successful plant, forgetting that it had evolved to that form because of local circumstances, and that it was still evolving. It's the same with XP.
One thing I remember having read on XP mailing lists (and in some XP book, I think) is what you should do with team members that refuse to program in pairs. The basic idea is that you should tell them that pair programming is the most effective way to do things, and if they don't stop resisting, you should have them replaced. But you can't force XP on people. It won't work if it isn't accepted by the team members. Adopting XP is in most cases a rather dramatic change of culture; even if you consider yourselves agile from the start, there can be many subtle, but significant culture clashes – subtle because you thought that your organization was something that it was not, so the reality clashes with XP and you might not be able to easily see why. (See my XP and culture post.)
Popppendieck quotes someone (I can't find it now) as saying that the greatest demotivators are policy, supervision, and administration, and that the greatest motivators are achievement, recognition, and responsibility. I have thought lately about the importance of individual freedom in software projects. These demotivators and motivators are definitely about freedom, and the NUMMI story shows that individual freedom can make a big difference (the same goes for the team portrayed in the Engines of Democracy article). What is needed is a clear overall vision and a team of skilled people that are free to find the best way to make that vision real. I can't remember if I have blogged about this, but Poppendieck writes about the importance that individuals feel that they are allowed to make mistakes – or, in fact, that they should make mistakes. Making mistakes is a necessary element of achieving success, and she writes that companies that are very successful (and hence innovative) have a tolerance for mistakes (I know that she mentions 3M in this context).
Lean development #8.
Last night I finished reading Mary Poppendieck's draft copy of Lean Development. It will be an great book when it is released, I'm sure. Myself, I haven't read much books on the agile process – only a bunch of books on XP. Although many of the XP books are great, too, I'm very familiar with the lingo by now; there are few surprises besides in the details – small new discoveries here and there in the more recent books and articles. But this book looks at these things from new angles, relying to a great extent on comparisons with other fields. And this is something I believe we must do to sell agile: we must show examples of other businesses and industries that we would call agile. There are reasons that people seek ever more sequential and rational processes in software, given that many feel that software largely has failed (blown budgets, gotten lost in the woods), and they must be reassured that there are better ways to do things, and that these ways might seem at the first glance to be chaotic and irrational.
Poppendieck mainly relies on looking at the car industry in this book, something that I find to be very relevant: making cars are very costly and riskful endeavors; there's a long time to market and no-one knows really what cars people will want when a new one is ready for release; there are many people involved; there's a pressure to invent new things with each car. And it might come as a surprise to many that the most effective approaches to making automobiles are far from what is sought in software. They don't strive to have a perfect design, where every tiny little aspect is covered and guaranteed not to be changed, before they start producing the vehicles. Instead, what software people want to do in different phases, with clear milestones in between, is done concurrently in the automobile industry. In fact, change is seen as a given, and mistakes are seen as a natural ingredient in discovering the design for a new car.
I think that many people are fascinated by these stories of odd but highly effective approaches, and that the surprise effect can be used to convey the message of the agile movement. The stories seem odd because the practices we have come to use are so unnatural. As Poppendieck shows in her book, we tend to seek discipline and rationality in chaotic times; we focus on details and ignore the whole; we grasp what we feel to be in control of. Rational, sequential processes seems to comforting, but in fact they are to blame for many of the failures in software. For example, the much cited cost-escalation curve: that a change in analysis and design is cheap, but a change late in development costs a hundred, if not a thousand times more. This is indeed true for some things, but far from all. Most of the changes aren't that more expensive to make at a later time. And many late changes become expensive because of us trying to nail the design at an early stage.
Again, by looking at the automobile industry, the approach invented by Toyota in the 1980's involves delaying decisions to the last responsible moment . By keeping your options open, you are free to change. And since your entire process is centered around change, with each short iteration being about producing something, and finding out if we need to improve the result. In a change-driven process, changes become less expensive. The most powerful example from the auto industry is the one about die cutting – the process of creating the die to shape the individual parts of the car's body. Poppendieck writes (emphasis mine):
When sheet metal is formed into a car body, a massive machine called a stamping machine presses the metal into shape. The stamping machine has a huge metal tool call [sic!] a die which makes contact with the sheet metal and presses it into the shape of a fender or a door or a hood. Designing and cutting these dies to the proper shape accounts for half of the capital investment of a new car development program, and drives the critical path. If a mistake is made and a die is ruined, the entire development program suffers a huge set-back. So if there is one thing that automakers want to do right, it is the die design and cutting.
Before the US car industry had to change to catch up with the Japanese, the cost of changes to the die design was 30–50 percent of the total die cost; in Japan it was 10–20 percent. And the Japanese starts the die cutting process at the same time as the designers start working on a new car design – something that is profoundly contrary to our intuition: if failures are so expensive, how can you dare beginning before you have a solid feel for the design?
(You'll have to read the book yourself to learn how it is done.) This is just one example of how it is possible to start without a detailed architectural plan or design; of how you can't do things effectively unless you use an iterative, incremental approach. We need more examples like these to explain what agile is about. Poppendieck sets a great example of how this can be achieved. Her seven guidelines borrowed from lean thinking, and adapted to software, are very effective:
- Eliminate waste.
- Amplify learning.
- Decide as late as possible.
- Deliver as fast as possible.
- Empower the team.
- Build integrity in.
- Avoid sub-optimization.
As I began reading the book, I understood the essence of all these, except for the last two. I think I felt that they had more to do with the actual programming, rather than with the process. True, they can, and should, all be applied to software as well. In fact, you can't really separate the process from the product, but somehow the last two felt odd in company of the others. I'll run through them quickly backwards from Avoid sub-optimization.
Avoiding sub-optimization means that overall success is what's important – not success of individual tasks. Poppendieck uses Tour de France as an analogy, and lists how many stages Lance Armstrong, winner of the last four races, won at each time: in 1999, he won four; in 2000 he won one; in 2001 he won four; and last year he won one. There are twenty one stages in all, and the point is that you can win even if you don't win all the stages. In fact, aiming to win each stage seems to be considered a lousy strategy by the racers themselves.
So, what does this mean? Well, Poppendieck talks about measurement, and about how you can't measure everything, and that measuring can lead to focusing on the wrong things. She uses an example of a manufacturing company that has invested in a very expensive machine, and where they spread the cost of the machine across all the parts the machine produces. The idea is that the more parts produced, the less part of the unit price will be the machine cost. This measurement resulted in maximizing the utilization of the machine, which led to a large inventory of parts – which looked good in the books, since inventory is filed as an asset, but it didn't lead to profitability; rather, the inventory clogged the arteries of the plant and [made] it very inefficient . The measurements showed success, though, with low unit prices and high machine utilization.
I'm unsure about whether this can be applied to iterations and the success of individual iterations. It seems intuitive that when each iteration succeeds in delivering the features that were planned, the entire project will succeed. But then again, if you measure completed features, you might, I suppose, forget what really matters: producing value for the customer. I can easily imagine a project where all features are in place, but the customer is unhappy about the result. The guideline is, I guess, to know what you are measuring, and to don't worry about measuring at the micro level.
Next, about building integrity in. At Oops, we are often located at the customers' sites, but we try to communicate as much as possible with each other. We have an internal chat server, for example, and every now and then the tone gets quite playful (especially in Friday afternoons, which we call Funday ). The other day, we had a discussion about the definition of gadgets. I said that the size of a gadget matters: if it is too big, it isn't a gadget however cool it is. Urban (I think) asked whether some four-wheel drive Volvo could be seen as a gadget, but I said that it couldn't – unless it had an autopilot. Then I thought that Segway definitely is a gadget, although that it is big compared to something hand-held. I said that the whole is important: manuals, the packaging, etc – a potential gadget can be ruined if it is poorly packaged or documented. This led me to think that the Babybjörn definitely is a gadget: it is a brilliant product; very well designed and packaged.
Perhaps the discussion got quite out of hand. I'm not sure whether the Babybjörn actually qualifies as a gadget. Perhaps gadgets must be electronic devices. But what we were talking about was a quality that Poppendieck calls perceived integrity – based on what Kim Clark calls external integrity (he has written several books about this, but I'm on the train now so I can't link to them at Amazon). Perceived integrity is what we were actually talking about, perhaps constrained a little by what constitutes a gadget. Poppendieck writes, for example, about Google and how it seems to be designed with a profound knowledge of what you want; it's as if they were inside my head, she writes.
She also talks about conceptual integrity , which is something vital to achieve perceived integrity: with poor conceptual integrity, it's hard, if not impossible, to create a user interface that just feels right. In this context, she talks about refactoring as the system evolves. The users' reality evolves, so what was once a system with perceived integrity, might in a few years lose that quality; so when the system evolves with its users, the problem is that conceptual integrity must be maintained, to which the solution is constantly refactoring.
Ever since I read Refactoring by Martin Fowler the first time, I've repeated the refactoring mantra to myself – but I seem to always do too little of it (although I'm definitely getting better). An experience last week reminded me again that each time I think that something needs to be refactored, I should do it immediately instead of putting it off to a later time.
I'll continue this post later.
[Tesugen.com]
11:14:04 PM 
|
|
