Adventures in home working

The last post has got me thinking more about the whole concept of daily builds.  I mentioned in passing that the concept is not just applicable to software development but I did not explain the comment.  I went out for a walk and started to think through how the concept could be applied to a systems integration project.  The one I chose is quite topical for me at the moment, a Windows XP desktop refresh and desktop management project.

So first let’s look at some characteristics of this sort of project:

1. A standard system image that needs to be deployed tens of thousands of times to many different types of hardware
2. The need to deploy thousands of applications on top of this standard system image, and to deploy these applications hundreds or thousands of times
3. The need to access seamlessly thousands of file, print, authentication, management and application servers
4. An environment that tens of thousands of users will use for perhaps 2-3 hours a day on average, this means hundreds of millions of pounds of deliverables depend on its usability and reliability

So let’s look at the daily, (or perhaps regular), build process and ensuring that the project team “eat their own dog food”, and how these techniques help with a project of this type:

1. It dramatically reduces risk, as deliverables evolve incrementally and at each build cycle the progress is the subject of widespread review.  Misunderstandings, omissions and errors rapidly come to light.
2. The collective intellect of the project team can be directed at improving the solution
3. The project team are forced to think, “automation, reliability, configuration control, reproducibility and quality”, on a daily basis
4. Rapid and efficient build and deployment processes are tested and refined from the beginning
5. Rapid application provisioning and deployment processes are tested and refined from the beginning, application packages are deployed many times to many different build standards and different hardware types long before they are delivered to real users
6. Architects get to verify daily that the conceptual and logical architecture are actually being implemented as envisioned, and that changes/optimisations that occur in the physical implementation get reflected back into the architecture
7. The whole provisioning, deployment, usage and management infrastructure gets integrated and exercised in an evolutionary way, greatly reducing the risk that would otherwise occur when different teams wait until integration testing in the lab.  Prior to the lab the only integration occurred through document review.
8. Every time the environment is rebuilt, (not every device is rebuilt every day of course), or a device within it is rebuilt the quality of upgrade, and migration processes gets tested.
9. Every time a device is rebuilt the success of separating system state, from user state gets tested.  Users get very unhappy when they keep loosing user state information.
10. User experience, i.e. how the whole environment fits together from the users perspective gets tested,  otherwise user experience only gets tested in pilots, which is normally too late to really make significant improvements.  This is a key issue, most projects are structured around creating bits of a solution, architects are responsible for making sure the bits of the solution work together but its rare to find sufficient focus on making sure that the bits not only work but integrate into an environment that’s effective, usable and discoverable

If you wanted to run a project this way what would you need:

1. You would need to create a “dog food” environment that the project team use.  You might not start with the whole of the project team just the core developers and architects first, then the project managers, then the programme managers etc.
2. You would also need system and integration test labs, and probably a customer reference lab, because not all test scenarios can be exercised in a dog food lab with real users with work to do.
3. You would need to implement the whole solution in this “dog food” lab, as close as possible to the user’s environment.  At first the implementation might be un-configured, i.e. products as they are installed out of the box.  Some elements may not exist at all place holders would be implemented instead just to show that a bit is missing.  For example if “end user application provisioning” is not there yet, make sure the link is on the desktop, and that a web page opens saying this bit is not done yet.
4. Make sure you tell people every day what has changed.  A set of RSS feeds is ideal for this.
5. Make it really easy for people to report problems, comments, suggestions etc, and make sure these get logged and that the project team classify and respond.  A discussion database is ideal for this.  Appoint someone in the lab team who is motivated to encourage and help the testers to identify and describe bugs, issues and comments.  Don’t worry too much about making the testers log bugs etc in a particular way, getting input should be the priority regardless of whether it’s via the phone, one to one, logged on a web site etc.  That’s why you need someone dedicated to helping the users to log the problems.
6. Make this person also take responsibility for chasing the developers to make sure they respond.  Testers are much more motivated when they get responses.
7. Make sure that the management organisation get involved in operation and management of the environment.  It’s a great opportunity for them to learn and to test out the “management experience”, in the same way as I described the user experience above.
8. Once things start settling down consider switching to testers logging issues through the normal help desk channels.  In the early stages this would stifle the process, but in the late stages when the number of events being logged is much reduced its worth it to help test that process.
9. Make sure the lab team responsible for the “dog food lab”, think of themselves as customers of the development team.  Make sure they drag capabilities out of development, make sure they are demanding and set down standards of quality and reliability and automation, make sure they gather input from testers and chase up developers for responses.  Make sure that lab team don’t get dragged too far into development otherwise they will stop being effective customers and become too understanding of the developers challenges.

- Posted by Steve Richards - 10:50:46 PM - comment []

Another link to Joel this time on daily builds another of my favorite techniques.  Many people rebel against the idea, especially project managers who like to see release schedules and milestones.  There is nothing in the daily build concept that contradicts good project management process however, its just that the progress towards milestones is tested daily, so that the chance of suprises are reduced and the dependency on key individuals is reduced.  Here is a snipit but please read the whole article if you develop and IT system, and take note that the concept can be applied to all types of development not just software.  I used the daily build concept on one of my projects and I think it was a great success.

Here are some of the many benefits of daily builds:

1. When a bug is fixed, testers get the new version quickly and can retest to see if the bug was really fixed.
2. Developers can feel more secure that a change they made isn't going to break any of the 1024 versions of the system that get produced, without actually having an OS/2 box on their desk to test on.
3. Developers who check in their changes right before the scheduled daily build know that they aren't going to hose everybody else by checking in something which "breaks the build" -- that is, something that causes nobody to be able to compile. This is the equivalent of the Blue Screen of Death for an entire programming team, and happens a lot when a programmer forgets to add a new file they created to the repository. The build runs fine on their machines, but when anyone else checks out, they get linker errors and are stopped cold from doing any work.
4. Outside groups like marketing, beta customer sites, and so forth who need to use the immature product can pick a build that is known to be fairly stable and keep using it for a while.
5. By maintaining an archive of all daily builds, when you discover a really strange, new bug and you have no idea what's causing it, you can use binary search on the historical archive to pinpoint when the bug first appeared in the code. Combined with good source control, you can probably track down which check-in caused the problem.
6. When a tester reports a problem that the programmer thinks is fixed, the tester can say which build they saw the problem in. Then the programmer looks at when he checked in the fix and figure out whether it's really fixed.

- Posted by Steve Richards - 7:47:33 PM - comment []

Joel, writes up an interesting example of NOT eating your own dog food, (ie using the IT solutions you are developing yourself), until it was almost too late:

Eating your own dog food is the quaint name that we in the computer industry give to the process of actually using your own product. I had forgotten how well it worked, until a month ago, I took home a build of CityDesk (thinking it was about 3 weeks from shipping) and tried to build a site with it.

Phew! There were a few bugs that literally made it impossible for me to proceed, so I had to fix those before I could even continue. All the testing we did, meticulously pulling down every menu and seeing if it worked right, didn't uncover the showstoppers that made it impossible to do what the product was intended to allow. Trying to use the product, as a customer would, found these showstoppers in a minute.

And not just those. As I worked, not even exercising the features, just quietly trying to build a simple site, I found 45 bugs on one Sunday afternoon. And I am a lazy man, I couldn't have spent more than 2 hours on this. I didn't even try anything but the most basic functionality of the product.

Monday morning, when I got in to work, I gathered the team in the kitchen. I told them about the 45 bugs. (To be fair, many of these bugs weren't actual defects but simply things that were not as convenient as they should have been). Then I suggested that everybody build at least one serious site using CityDesk to smoke out more bugs. That's what is meant by eating your own dog food.

This is just SO IMPORTANT, as an architect of enterprise infrastructures I get really frustrated when I try and use them, if I did not get that frustrated I would not be motivated to fix the issues, in fact I would not even know they existed.  In my ideal project the project must use the environment its creating, even if it has to distort its processes a bit to fit the product, its key it uses it if at all possible.  Its likely to be a real pain to te project to do this because it means the project is running on/depending on something that a bit flakey but its worth it. 

As an architect its especially important because it allows you to validate that your concept is indeed being implemented and that your concept is correct, which it never is!  If you don't use it I don't see much chance of either of these being achieved.

I really dread to think what a solution would look like if it didn't go through this process, if the only way to define what something looked like was a specification, and the prayer that some how the author of the spec and the 50-100 people who developed the solution from it actually were able to maintain not just conceptual integrity but also usability!

- Posted by Steve Richards - 7:23:23 PM - comment []

This is another of my regular updates on how I am progressing/coping with Adult Onset Stills Disease.  The following graph gives you an overview of symptoms:

Here are the highlights:

1. For most of this period I have been on 20MG of Prednisolone
2. I have continued to have good and bad days, more bad than good until recently
3. Then I had a period of 18 good days, the longest period of good health for 7 months, I put this down to the Steroids finally kicking in.  Of course during this time the sleeplessness got worse so I was still pretty tired, but when there is less pain life seems so much better!
4. I went to see my specialist last week he said given the fact that I am feeling much better I need to cut the Prednisolone to 10MG and then drop it by 1MG per week.  As I have now been on Prednisolone for well over a year I needed a bone scan as well
5. I had the bone scan and it reveals that there is some cause for concern, but it not too bad at the moment.  I was pretty disappointed by this as I have put a lot of effort into my diet and exercise to try and maintain bone density.
6. My specialist has also said that he wants me to consider going onto Azathioprine or Methotrexate and that I need to decide which set of side effects I fancy having the most L.  Neither looks very inviting. 
7. Feedback from the Stills Message Board seems to favour Methotrexate but I am keen to avoid either, I just don’t know what to do to achieve it, but I am trying as many lifestyle, and diet options as possible.
8. I have just got back from 3 nights away in Scotland.  Something strange happens on holidays I always seem to feel much better, in fact twice on holiday I have been completely symptom free for 2 days!.  This has had me thinking that there is some miracle cure waiting to be found and I have thought through the options.
   1. Less stress; I don’t think this is the reason, holidays are actually quite stressful, lots of time cramped in the car and caravan with 4 squabbling kids etc.
   2. Different diet; I thought this was a factor as on one holiday by chance I had a dairy free diet, but I have tested that since and it didn’t have a huge effect, certainly not significant enough to account for the variation I see in symptoms
   3. More exercise; when we are on holiday we walk a lot; of course I only go on holiday when I am feeling pretty good in the first place.  I think there is something in this, the continued low intensity exercise does seem to help, and I have noticed this at home as well.  The problem seems to be when I stop even for a day symptoms spring back and then it takes a while to build up the exercise levels again.  I am talking about 3 hours of walking a day here, that’s a lot of exercise! When my symptoms spring back 10 minutes is too much!  When I am, flaring getting down the stairs is way to much.
   4. I eat more, because I exercise more I allow myself to eat more snacks, mainly in the form of fruit or dried fruit bars.  It's possible that there is something to investigate here.
   5. Weather; we have been pretty lucky this year in that we have had really good weather, very sunny and not too humid.  I think there is something in this as well, partly mental and partly physical.  Also good weather tends to encourage exercise, so it reinforces the point above
   6. No work on the computer; I hope this isn’t a factor, (because its my job), but its possible that the sitting around for a lot of the day is a factor.  I use RSI guard on my PC that forces me to keep taking breaks and this does seem to help.
   7. I sleep better, I almost always take sleeping tablets on holiday as the caravans are so hot and the beds uncomfy etc, so I get a very good nights sleep.  I think there is something in this as well.  When I feel really bad I do find that taking sleeping tablets for a couple of days helps.
9. Anyway I am back from holiday and my symptoms are back again, (I have also been under more pressure than usual at work for a few days).  Wrists, Ankles and Knees are the worst.  I have a sore throat again, headaches every day and difficulty concentrating.  I am hoping that some of this is Steroid withdrawal. 
10. I am going to try the sleeping pill trick again for 2 nights, and try and exercise but its difficult.  However I have learned that the symptom pattern is very variable and it may be at the end of the day that I never find a pattern of external events that cause a trigger.  I am just not at the point where I give in to it being something I am unable to influence.

- Posted by Steve Richards - 4:44:20 PM - comment []