Closer To The Ideal

I recently oversaw a project that had me working with a distributed team (in the US, Europe and India). The same project also worked exclusively with independent contractors, whose work was hired on an assignment basis. The work was for a new web-startup. I have some concerns about such an arrangement being used for a new, innovative project. I’m going to attempt to explain my concerns in this essay.

First, I’ll talk about using freelance, independent contractors for a web start-up. The seeming advantage of having no permanent employees is that the operation has no fixed costs. An advantage? It depends. If you really want to avoid all fixed costs, simply close your business. Being in business means having expenses, the real issue is whether you are making a profit – does the labor you hire help you make more money than the cost of the labor?

The project I oversaw moved slowly, and some of the slowness can be attributed to the freelance nature of the labor we hired. The programmers would do whatever we asked them to, and they all did high quality work. However, because we made no long term commitment to them, they of course had to keep busy with other work. Sometimes we needed them but they were busy with their other clients. Also, they were not part of the conversations we were having in New York City, so they lacked a broad overview of the project and its goals, so they could not easily take the initiative to move ahead on their own.

If a project has no particular deadline, or if it is simple, then working with independent contractors is a great way to keep fixed costs low. However, if a project is suppose to move fast, or if it needs to achieve certain deadlines, and especially if it is very complex, then I would recommend against the use of this kind of arrangement. For fast moving, complex projects, you need full-time, committed programmers.

Now I’ll offer some thoughts about agility, and the importance of physical proximity.

Agile practices have 2 main hopes:

1.) to allow a fast-forming team to quickly become highly-productive.

2.) to control costs by ensuring discipline in regards to goals and deadlines.

I used the word “hopes” instead of “goals” because the ultimate aim of agile practices is more of an ideal to be aspired to rather than a reality that can be achieved, and by that, all I mean is that we are constantly learning more about what these practices truly mean, and in the future we can reasonably hope to do better than we are doing today. I’m speaking for the entire profession of computer programming when I say this.

I am in the United States, in New York City. I’ve been working on websites since 1996. I’ve worked on many small teams, and over the years I’ve used some agile practices, such as frequent iterations, frequent in-person consultations with the client, and unit tests. When I started on this last project, I first thought it would offer the chance to experiment with a fuller range of agile practices. However, I came to realize that most agile practices depend on physical proximity. The programmers need to be able to meet, preferably every day (”every day” in the sense of casual conversation, not “every day” in the sense of “let’s have a formal meeting”). Consider some of these practices:

1.) pair programming (two people looking at the same screen, checking for errors)

2.) the use of index cards to map objects

3.) stand up meetings (meetings kept short and lively because everyone is standing)

None of these make any sense if the programmers are in different countries.

I’d also add code reviews. One can do a code review at a distance, but I think in person is much more effective. And code reviews don’t make much sense when you are working with independent contractors. Code reviews are meant to develop each programmer’s long-term abilities, and also they are useful for helping the whole team agree on certain programming habits. But one would have to have a long-term relationship with the programmers before doing such a thing would make sense.

Programming is an art, and art needs passion. Many agile practices are aimed at soliciting a greater emotional commitment from the programmers. I might write “For an agile project to go well, the programmers need to emotionally commit to it” but that isn’t quite right. What would be more correct would be “For an agile project to go well, the programmers need to emotionally commit to their own professional ideal.” That is, they need to care about the craft of programming. They need to be constantly pushing themselves to always be better programmers.

I would never argue that agile practices are needed on every project. If a project is routine, then long-distance, non-agile practices might be perfect. For instance, suppose you decide to launch a new online magazine. After researching your options, you decide you will use WordPress as the software that powers your site. I’d suggest you find a good designer, local to you, and work with them to come up with the design. Once you have the design, the rest of the process is mundane. You could certainly hire a team in India to install WordPress and implement your design for you. And you would not need agile practices for such a project. In fact, you do not even need great programmers for such a task. Mediocre programmers can do a reasonable job with tasks that are standard, routine and mundane.

However, I would argue that agile practices are needed at projects that need to be fast moving (this would include most start-ups), or at any project where the aim is to create something altogether new and unique. If your project is venturing out into the great unknown, if you are going where no one has ever gone before, you will need a team of excellent, committed adventurers to go along with you.

Let’s look at the history of agile practices, going back centuries.

In the 1990s, DARPA invested some money in agile research, which lead to such books as that by Goranson: Agile Virtual Enterprises.

Goranson looks back at the whaling industry of the mid-1800s and examines how 2 small towns in Massachusetts were able to capture 90% of the whaling industry, when most of the actual hunting was done in the South Pacific. He found that the patterns of hiring and team building had much in common with what start-ups aspire to nowadays – a tight-knit, fast-forming team of highly competent professionals held together by a commonly understood body of professional ethics and expectations.

Goranson emphasizes the role of geographical concentration (and therefore physical proximity), to the evolution of culturally unique traits, leading to competitive advantages:

Geographical and professional isolation allowed for the evolution of a unique and very effective system which for many years put a brand new, fairly high risk/high payoff, virtual enterprise in the water at the rate of one every two weeks. The return of each whaling expedition triggered the formation of another. It was considered an invitation to bad luck to reuse the same combination of partners, so during the six to nine months it took to recondition the ship for another voyage, the owner of the boat assembled a new group of key players who would join him in setting up the basic physical and social conditions needed for a successful venture.

The primary partners required to launch a voyage consisted of a ship owner, an insurer (of the ship cargo), a provisioning financier to supply the expedition with food and other consumables, a captain, and often a manufacturer who agreed to buy the oil at a set price. This component of the partnership was formed in the first couple of months, the partners being determined partially by availability.

A month or less before the ship was ready to sail, a secondary group of partners, the crew, was formed. They shared a distinct cultural background; almost none of them had, or would ever, serve in the navy or the merchant marine. This professional distinctiveness, coupled with an intense geographical concentration, fostered the development of a unique culture based on the virtual enterprise.

Goranson uses the word “agility” to refer to fast forming supply chains, and also (relatedly) the ability to overcome the administrative costs that might keep a new supply chain from forming (in the early 90s, when DARPA funded Goranson’s work, one problem seemed especially prevalent: the fact that the most innovative technologies in America were being developed at small start-ups, and yet large Fortune 500 companies were having difficulty integrating small start-ups into their supply chains, because of the administrative hassles of developing relationships with a multitude of small companies). He uses the word “virtual” to refer to an organization that is project-specific and not meant to last. For instance, in the modern context, the best examples of agile, virtual enterprises are in the movie industry. Each movie is its own independent business, and the production of each movie entails the fast formation of a team of highly competent professionals who understand they are being hired to work on a specific, time-limited project, which will disband as soon as the project is done.

Whaling ships and modern web start-ups share certain traits::

Every voyage included a team of skilled craftsmen – carpenter, blacksmith, cooper, and a sailmaker (often a boatwright, a rigger and a cook were also in this class) – whose combined expertise allowed the enterprise to respond effectively to a broad range of situations. Each of these professionals, along with the tools, supplies, and sometimes apprentices they brought abroad, formed an essentially self-contained business which was integrated into the enterprise as a whole. In these cases, it was not just the person who signed up for the voyage, but their business. From the shipowner to the cook, everyone was paid with a pre-arranged percentage of the take.

In other words, every ship was its own little agile, fast-forming start-up.

Goranson’s use of the word “agility” is somewhat different than the way the word “agility” is used by its proponents in the software industry, but the meaning of the word overlaps for both groups in the sense of “the ability to get supply from a new source quickly”. Consider the words of the Agility Manifesto:

We have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

With only minor tweaks of the wording, you could use the above as a statement of the professional culture that pervades Hollywood. The emphasis is on doing what is necessary to get something working into production fast. By contrast, the slow moving process of low trust, heavily documented, laborious negotiated, carefully vetted contracts are anathema in this context.

(Stephen M. R. Covey has a written a book called The Speed Of Trust, which does an excellent job explaining how much a project speeds up when the participants can trust each other, and therefore how important it is to work toward facilitating the long-term development of trust in any business relationship.)

Goranson emphasized that one of the legal traditions that the whaling industry and movie industry share is a very strong reliance on verbal contracts, as opposed to written contracts and detailed documentation. Goranson points to the bankruptcy of Kim Bassinger as an outstanding example of the importance given to verbal contracts in the movie industry. (Bassinger met some producers for dinner to discuss the possibility of her starring in their movie Boxing Helena. She promised, over dinner, to star in the movie. When she later backed out of her promise, they were able to sue her for $8 million.) The Agility Manifesto de-emphasizes contracts, but still puts a heavy emphasis on the importance of verbal communication, a point made clear by “Customer collaboration over contract negotiation”.

Goranson emphasizes the role that geographical concentration played in allowing this culture of virtual enterprises to flourish. I believe that geographic concentration still plays the predominate role in allowing the continued evolution of unique agile practices.

I’ve had it suggested to me that the cheapness of overseas programmers (in places like India) can speed a project up, simply because, since they are cheap, you can hire a lot more of them. This is flatly contradicted by Fred Brooks thesis in The Mythical Man-Month. When the System 360 project was running late at IBM, Brooks tried to speed up the project by adding more programmers to the team. But he found that the additional programmers actually slowed the project down. Each new programmer became one more person who needed to be kept up to date with the latest documentation – and every few additional programmers required that a new manager be brought into the project to manage them. The growing complexity of communication across a growing mass of workers helped to slow the project down, rather than speed it up.

In this sense, the cheapness of overseas programmers actually encourages bad habits. Because they are cheap, you may get tricked into thinking that you can hire a lot of them. But the more you hire, the more time you will need to invest trying to keep everyone unified in their understanding of what the code is suppose to do. And after a certain number, you’ll need to start hiring additional managers.

If you are a project manager, and you rely on the cheapness of your programmers to keep a project’s costs under control, then you need to fire yourself. There is only one correct way to keep costs under control, and that is through disciplined focus on goals and deadlines. Agile practices have evolved to make it easier to keep complex projects on track. To the extent that geography matters to agile practices, then geography matters for the goals and deadlines of your project.

Nicole Radziwill approaches this subject from a different angle, but she makes a similar point about the importance of geography:

Space does matter. We know this when we are designing facilities and plant layouts, for example, because one of our common considerations is to minimize traffic between areas and departments. More often than not, we do this to minimize the time spent moving people or equipment around a plant, so that time is not wasted. But the same concept could apply to our supply chains. Why aren’t we minimizing the time that components or goods spend traveling through the supply chain, when it could lead to reductions in energy costs? Furthermore, why aren’t we shortening our supply chains to strengthen local and regional businesses, and train the next generation of skilled workers (who can actually do something useful for the regional economy)?

The logic has been something like this: energy is cheap, therefore transportation is cheap, and transportation is easily available and accessible through third-party providers like FedEx and UPS. But I can’t shake the feeling that “supply chain status quo” is not good for quality in the long-term – because it encourages us to source the products and components that are most affordable, rather than the ones that might help us cultivate a quality consciousness in our local areas.

To offer a partial answer to the question that Nicole poses, I think the reason why products get sourced to where they are most affordable (at the current moment, in the short term) is that people tend to fear losses more than they value potential gains. There is an abundant literature on this subject. In regards to sourcing, buyers go with what is cheapest in the short-term, rather than developing long-term innovations that might lead to much greater gains in the future. Consider:

Prospect theory… argues that people center their expectations around a variable and moveable reference point. The reference point is most easily thought of as the status quo, though it is not always so. In most situations, people are more likely to ensure that they preserve their current situation, than to risk losses by seeking potential improvements. Indeed, people tend to sacrifice a great deal in order to maintain their current situation. As Levy notes, “people tend to be risk-averse in choices among gains, but risk acceptant with respect to losses.” This is often the problem in intractable conflicts; the risks required to resolve the conflict seem much too great compared to the potential losses. This sort of framing is what leads ultimately to hurting stalemates. Loss-oriented framing is, ultimately, this tendency to minimize losses while missing opportunities to improve one’s situation.

The status quo of any culture, in the short-term, can be regarded as an under-developed blandness, relative to the unknown brilliance of some unknown future innovation. It takes a leap of faith to believe in that unknown future brilliance. It is easier, and safer, to go with what is cheapest in the short-term. This is cowardice, but it is extremely common. True business leadership is rare.

In terms of developing some unknown future brilliance, I’m thinking especially of those cultural traits that, in part, give rise to comparative advantage. There is, at this point, a mountain of literature about the advantages that agile teams have when it comes to fast-forming, fast-moving projects. Agility is a cultural innovation that deserves further refinement, and each nation, depending on its legal traditions and its labor laws, is going to develop agility in a unique direction. The future development of these potential innovations is undermined by the extended supply chains that send work to distant parts of the world. I’m arguing that, just as Goranson stressed geographical isolation as playing a role in the evolution of a uniquely competitive culture in Massachusetts, then the dispersal of work from a geographic center must lead to the opposite – the dissolution of uniquely competitive cultural traits.

I can imagine someone reading this essay and raising certain objections. What about Linux? What about Apache? Certain open source projects have been successful with teams that are scattered all over the world. But then, these were not fast moving projects. Teams that never meet in-person can become highly productive, but it takes years. And by the time these projects hit their strides, much of the core team had, in fact, traveled to different countries and met each other in-person.

One other fact argues for the importance of physical proximity. For a team to be highly productive, it needs to have both high levels of communication and high levels of trust. Body language is important to trust. In a normal, in-person conversation, the bulk of the communication is non-verbal. Below I’m going to quote from an article that is aimed at teaching salespeople how to make more sales. On a start-up, the team members are not trying to sell products to each other, but they are often in the position of having to try to sell a given strategy to each other. One team member may feel that their site should be pitched to consumers, another team member may feel strongly that the site needs to be pitched to businesses. The success of the start-up depends on all of the team members being to able to listen to all points of view, take in the maximum amount of information and opinion, and reach the best decision.

It really doesn’t matter how knowledgeable you are about your product line or how many closing techniques you have mastered, unless you earn your prospect’s trust and confidence you’re not going to make the sale.

…Body language is a mixture of movement, posture and tone of voice. Research indicates that in a face-to-face conversation, more than 70 percent of our communication is nonverbal.

Our body language reveals our deepest feelings and hidden thoughts to total strangers. In addition, nonverbal communication has a much greater impact and reliability than the spoken word. Therefore, if your prospect’s words are incongruent with his or her body language gestures, you would be wise to rely on the body language as a more accurate reflection of their true feelings.

On a slightly humorous note, I’d like to offer a movie review. The best movie ever made about an agile, virtual enterprise was Akira Kurosawa’s 1954 movie, The Seven Samurai. Of course we need to translate the movie into business-speak before the comparison becomes clear. I’m only partly kidding when I write this. I really do believe this movie offers a good example of some of the above ideas.

In the opening scene, we meet the client (poor village farmers). We then come to see their problem: bandits who plan to raid the village as soon as the harvest is in. The villagers then go to a big city to hire a contractor who can solve this problem for them. They meet Kambei, a samurai. Though the villagers can pay very little, he takes pity on them and agrees to help them. He therefore steps into the role of project manager. He must then assemble a team, and this part is pure agile enterprise in practice. The city is full of many samurai, and out of this sea of potential recruits, a group of 6 is selected. These are highly skilled workers with a strong emotional commitment to their own professional ideal – that is, they all strive to be the best samurai that they can be. The 7 samurai then set out for the village. They almost instantly form a tight-knit, highly cohesive team, because they share a commonly understood body of professional ethics and expectations. The enterprise is “virtual” in the sense that Goranson uses the word – it is project-specific, and will disband as soon as the project is complete. At the village, the samurai work in close consultation with the client to ensure the goals of the project (the defense of the village). Finally, after a decisive fight, the goals of the project are achieved – the bandits are defeated. The client is happy. The surviving samurai then disband.

In this movie, the samurai are recruited in the same fashion as sailors might have been recruited for a whaling ship, or the same way that a computer programmer might be recruited for a web start-up.

To recap, my current experience teaches me that:

1.) Agile teams should be small (as per Brooks).

2.) They should meet in person several times a week.

3.) Everyone on the team must be highly competent.

4.) Everyone on the team must be emotionally committed to their own professional ideal.

5.) For the duration of the project, everyone on the team should be full-time

Furthermore, on projects that aim to be unique or innovative, or on projects that are complex, or on projects that need to be fast-moving, I will be discouraging clients from these practices:

1.) part-time, independent contractors

2.) distributed teams, with workers geographically dispersed.

One last point, unrelated to the rest. A number of people have said to me that they feel the cheapness of overseas workers (in places like India) is driving down wages in America. I do not believe this is true. Wages in America have declined over the last 40 years, but that is because the American people have themselves become unwilling to invest in their own country. Consider the fact that in 1955 the average middle class family saved 17% of their income, but in 2000 the US savings rate hit 0%. Wages will decline in any nation that does not invest in itself. By contrast, consider Germany, which has a high investment rate. Not only does Germany have higher average wages than America, it has also had more economic growth over the last 30 years. And right now China is saving and re-investing 40% of its income. I do not know a single American family that saves 40% of its income. Most of my friends, even those with college educations and good paying jobs, only save 10% or 15% of their income. It is wrong for Americans to first engage in an orgy of over-consumption and under-investment, and then blame poor nations like India and China for the obvious problems that must result from such reckless behavior.

Coding Horror has a nice set of quotes and links related to catastrophic software project failures. Of interest to anyone who does project management on software projects:

If Las Vegas sounds too tame for you, software might just be the right gamble. Software projects include a glut of risks that would give Vegas oddsmakers nightmares. The odds of a large project finishing on time are close to zero. The odds of a large project being canceled are an even-money bet (Jones 1991).

In 1998, Peat Marwick found that about 35 percent of 600 firms surveyed had at least one runaway software project (Rothfeder 1988). The damage done by runaway software projects makes the Las Vegas prize fights look as tame as having high tea with the queen. Allstate set out in 1982 to automate all of its office operations. They set a 5-year timetable and an $8 million budget. Six years and $15 million later, Allstate set a new deadline and readjusted its sights on a new budget of $100 million. In 1988, Westpac Banking Corporation decided to redefine its information systems. It set out on a 5-year, $85 million project. Three years later, after spending $150 million with little to show for it, Westpac cut its losses, canceled the project, and eliminated 500 development jobs (Glass 1992). Even Vegas prize fights don’t get this bloody.

Keith Casey writes a brilliant post about the need for technology experience on a team launching a firm that involves technology. This is a post that I wish I’d written, because it says a lot of things I’ve been thinking for the last year or two:

If you’re selling the best hamburger ever or making signs or running a franchise, most of the time, your technology can be an afterthought.

But if you’re building a social network, creating a new ad network, building a mobile phone platform or any number of other things, the technology is the core to what you’re doing.

That has some major implications. It means:

• First, you have to consider it from day one. You have to know how it fits with what you’re doing. What goes into it, what doesn’t, and get an idea of your budget (time, $’s, etc).
• Next, you need a technologist in the mix to figure out the boundaries. You need to know what is and isn’t possible and how to approach the process… more importantly, you need to know how to approach things, schedule things, and what “It’s 90% done!” really means. ;)
• Next, you need a technologist in the mix to filter the candidates. When you don’t know what you’re doing (fact, not judgement), you need someone who does to find the right people to hire and filter out the posers.
• Finaly, you don’t outsource it. I don’t mean offshoring, but that could be included. I mean the people who know your product – because they’re building it! – should be part of your organization.

Failure to have a serious tech person involved causes some terrible decisions.

For example, I recently tracked the progress of an angel-funded startup:

• First, since they lacked any technical background, the sole developer-turned-investor promptly developed the entire application… in ColdFusion.
• After that caught on fire and sunk into the swamp, they decided to rebuild it in .net… er… Rails… er… PHP. Their “technologist” didn’t have any experience in any of them and was therefore open to anything.
• Next, since they developed their “architecture” (aka PHP), they began recruit for PHP-types. Since none of them had any experience whatsoever in PHP, they hired the first PHP’ers that could pass the single-round phone screen.
• Finally, they called the ColdFusion version of the application a “reference implementation except for the wrong parts” and didn’t bother writing the spec as a result.

Fast forward a few months and inside reports say the team is in shambles, the project is months late, and major portions of it are being rebuilt from the ground up.

This strikes me as obvious, so I’m astonished when I run into a differing attitude. Yet just in the last year, I’ve had 3 different clients or potential clients who wanted to build online social networks and who thought they could leave the details of building the website to an outside team.

Last year I had to oversee a Flash project. The goal was to develop one of the most sophisticated video players ever. This thing could read XML files from other sites, and become an embeddable store. We were developing it for iHanuman.com, though it was never put into use there. The goal was to allow yoga enthusiasts to use their own blogs to promote the videos of their favorite yoga teachers. We were initially thinking the project would cost $10,000, but the owner of the company kept asking for additional features, so we thought maybe the cost might rise to $20,000. But in the end, the final cost was close to $50,000. Flash is not a serious programming environment, and as soon as you try to do something ambitious with it, you learn a painful lesson about its limits. For this reason, I am very excited about JavaFX. CJT sums up a number of my thoughts on the subject:

The maturity of the underling technology (the Java Runtime Environment) cannot be overlooked. One of the biggest gripes we have with, for example, the Flex framework, is it is bug ridden. Start doing any kind of serious project with it and you cannot get away from this. The mx components have been completely re-written no less than three times now! And they are still buggy. This slows development, there is no getting around it. Ask any developer who has done any serious stuff with the Flex framework and they will be able to testify to this. Hopefully with Flex, this will improve now that the framework is open-source.

Java is very mature and so we shouldn’t see soo many issues in this department.

The JRE is also leaps and bounds ahead of the Flash Player in terms of functionality, along-side the fact there are tons more Java developers (and more experienced) out there than actionscript developers, which as an employer, is also a key point. Finding decent actionscript developers is tough compared to experienced Java developers and this isn’t because of demand, its because actionscript 3 (I pinpoint 3 because 2 was half-baked and the Flash Player AVM1 was pants compared to AVM2 – hence incomparable to Java and the JRE) is a relatively new language and most actionscript “developers” have come from a creative/design background as opposed to programming. We like designers to do design and hard-core coders to code and for the two teams to collaboratively work together to create great results.

I don’t know much about Flash. Last year, for the video project, we brought in a Flash programmer as an independent  contractor. I was assigned to manage her, since I was the senior programmer with the firm I was working with. She started the project using ActionScript2, but when she realized how ambitious the goals of the project were, she switched to ActionScript3. We were asking her to do a lot of parsing of XML files, which of course AS3 makes easier than AS2.

As the project fell further and further behind schedule, I began to wonder if she was any good. So I contacted my old friend Nick Piazza, who was the Flash programmer for Category4.com (where I once worked).  I asked him to review her work and tell me if she was any good. So he reviewed her work, and he was very impressed.  She was, after all, doing cutting edge work in AS3. She was using many of its new features. He admitted to me that he himself, along with most other Flash programmers, were still perfectly content to do their work in AS2. In his opinion, she was doing the most advanced Flash work of anyone in central Virginia.

Great, I thought. So she is excellent. So why is the project falling so far behind schedule? She explained she was up against some intractable bugs. I asked if I could sit down with her and go over  the code. Some programming problems are, after all, logic problems, they are not specific to a language.

So we sat together and went over her code, trying to figure out the solutions to certain bugs. And what I learned was this: Flash has terrible tools for debugging. Adobe has only recently started to take ActionScript seriously as a language for advanced programming, and they are only slowly providing the tools needed for advanced programming. And of course, chief among the needed tools are debugging tools.

I walked away from those meetings with the sense that Flash was uniquely weak in debugging tools, and that this was a major source of budget risk when contemplating doing anything ambitious with it.

JavaFX starts off in a better situation than Flash. JavaFX is going to be a hit product. I’ll admit, for small animations Flash is too entrenched to ever be dislogded, but Flash just lost its bright future. For big, ambitious multimedia projects, JavaFX offers less  budget risk  than Flash.

The modern, affluent standard of living depends on society’s infrastructure, both physical and intellectual: airports, highways, bridges, ports, telecommunication networks, databases, power plants, and the software that makes it all run. 150 years ago it was still common for massive bridge projects to meet with catastrophic failure. The engineers of the 1800s slowly mastered the art of working with iron and then steel. The projects grew in scale, and the skills of the project managers needed to keep apace.

Nowadays we rarely hear of catastrophic bridge failures. This is a particular type of infrastructure that has come to be well understood, both in its engineering and in the project management that oversees its construction. However, we, as a society, are still struggling to find the right way to develop large software projects. This is the newest type of infrastructure, and the correct way to engineer it and manage its development is still a source of controversy.

Robert L. Glass has compiled a book of entertaining and instructive stories regarding the massive software project failures of our time: Software Runaways: Monumental Software Disasters. Anyone with an interest in the management of software projects should read it.

The worst of these project failures, the most expensive and the most ambitious, was the attempt made by the Federal Aviation Authority to modernize the computer system it uses to keep track of what planes are in the air. The effort began in 1981 and ended in complete failure in 1994. The government hired IBM to do the actual work, and over the course of 14 years, IBM burned through $3.7 billion dollars. Nothing was accomplished. The project was finally shut down by Congress. Nothing came out of the project, not a single piece of software, nor even a line of code, was ever used for anything.

Robert N. Britcher, who was involved with the project, offers a full write up of the project. I will try to entice you to buy the book with some excerpts:

The Advanced Automation System began, in concept, in 1981 and ended in 1994, “terminated for convenience” by the government. Billions of dollars were spent on it. It is hard to describe. You can’t learn anything from the name. You know it’s about air traffic control because I told you, or because you read about it in the papers. Maybe part of the problem was the name. It sounds like the system to end all systems.

One engineer I know described the AAS this way. You’re living in a modest house and you see the refrgerator going. The ice sometimes melts, and the door isn’t flush, and the repairman comes out, it seems, once a month. And now you notice it’s bulky and doesn’t save energy, and you’ve seen the new ones at Sears. So it’s time. The first thing you do is look into some land a couple of states over and think about a new house. Then you get I.M. Pei and some of the other great architects and hold a design run-off. This takes awhile, so you have to put up with the fridge, which is now making a buzzing noise that keeps you awake at night. You look at several plans and even build a prototype or two. Time goes on and you finally choose a design. There is a big bash before building starts. Then you build. And build. The celebrating continues; each brick thrills. Then you change your mind. You really wanted a Japanese house with redwood floors and a formal garden. So you start to re-engineer what you have. Move a few bricks and some sod. Finally, you have something that looks pretty good. Then, one night, you go to bed and notice the buzzing in the refrigerator is gone. Something’s wrong. The silence keeps you awake. You’ve spent too much money! You don’t really want to move! And now you find out the kids don’t like the new house. In fact, your daughter says “I hate it”. So you cut your losses. Fifteen years and few billion dollars later, the old refridgerator is still running. Somehow.

At $3.7 billion, the Advanced Automation System was one of the largest civilian computer contracts ever; maybe the largest. It was the largest single contract in IBM’s history. From the moment it was awarded, until near the project’s demise, IBM patted itself on the back. There was something for everyone, beginning with a great ball in Union Station, featuring Chubby Checker and “The Twist”.

At its peak the project employed over 2,000 people. About a million dollars a day. If you thought like the IBM project manager, this was a good deal. Many people were working and money was being made. It was going to last forever. No one considered that it wouldn’t. And everyone was getting ahead. (One of the ironies of conceptual work is that it is easy to believe you are farther along than you are.)

The AAS must have been the most supervised project in history: this atop its enormous size and complexity, and the extreme and constantly changing requirements. One programmer described it this way: “Working on the project was like working on a car inside the garage with the motor running. Eventually, even the crickets hopping around the tires suffocate.”

What I saw on the FAA’s Advanced Automation System would have made Sisyphus weep… Whatever commitment and discipline there was… was worn down by a battery of watchfulness that I can only ascribe to fear of failure. In spite of tens of millions of dollars spent on new computers for AAS, the most important piece of equipment on the project was the overhead projector. There were endless meetings attended by dozens of people – as if we were never quite sure about the whole thing. The people in charge simply lacked the confidence and the finesse of the space team: NASA, contractors, and astronaughts.

It has been noted by everyone from the New York Times to the Vice-President of the United States that the main problem on the Advanced Automation System was “changing requirements”. For those involved in large-scale computer systems, that is nothing new. No one can perfectly surmise the shape and feel of a system years in advance. Even replacing some aspect of a system you know by heart is not immune from thinking twice about it. … [But] the requirements churn (it was called) on the Advanced Automation System was not normal. It was the result of our enchantment with the computer-human interface, the CHI. The new controller workstation, fronted by a 20″ by 20″ color display, because it was capable of seemingly endles variety of presentations, mesmerized the population of AAS like the O.J. Simpson trial mesmerized the nation…

The project was handed over to human factor pundits, who then drove the design. Requirements became synonymous with preferences. Thousands of labor-months were spent designing, discussing, and demonstrating the possibilities: colors, fonts, overlays, reversals, serpentine lists, toggling, zooming, opaque windows, the list is huge. It was something to see. (Virtually all of the marketing brochures – produced prematurely and in large numbers – sparkled with some rendition or other of the new controller console.) It just wasn’t usable…

The cost of what turned out to be a 14-year human factors study did not pay off. Shortly before the project was terminated a controller on the CBS evening news said: “It takes me 12 commands to do what I used to do with one.” I believe he spoke for everyone with common sense.

Rummaging through one of the closets at the far end of the hall on the fifth floor one day, looking for some standards document, I found an envelope left by someone who left the company – as many did after so many years advancing against stone, while the wheels of commerce were accelerating on what everyone referred to as “the outside”. It contained “A Brief History Of The Advanced Automation System”. It was printed by hand and left, perhaps inadvertantly, or perhaps with the hope that some anthropologist might some day discover it and make a pronouncement. In every important way, it is the truth:

“A young man, recently hired, devotes years to a specification written to the bit level for programs that will never be coded. Another, to a specification that will be replaced. Programmers marry one another, then divorce and marry someone in another subsystem. Program designs are written to severe formats, then forgotten. The formats endure. A man decides to become a woman and succeeds before system testing starts. As testing approaches, she begins a second career on local television, hosting a show on witchcraft. An architect chases a new technology, then another, then changes his mind and goes into management. A veteran programmer writes the same program a dozen times, then transfers. The price of money increases eight times. Programmers sleep in the halls. Committees convene for years to discuss keystroking. An ambitious training manager builds an encyclopedia of manuals no one will ever use. Decisions are scheduled weeks in advance. Workers sit in the hallways. Notions of computing begin in the epoch of A, edge toward B, then come down hard on A + B. Human factors experts achieve Olympian status. The Berlin Wall collapses. The map of Europe is redrawn. Everything is counted. Quality becomes mixed with quantity. Morale is reduced to a quotient, then counted. Dozens of men and women argue for thousands of hours: What is a requirement? A generation of workers retire. The very mission changes and only a few notice. Programming theories come and go. Managers cling to expectations, like a child to a blanket. Presentations are polished to create an impression, then curbed to cut costs. Then they are studied. The work spikes and spikes again. Offices are changed a dozen times. Management retires and returns. The contractor is sold. Software is blamed. Executives are promoted. The years rip by with no end in sight. A company president gets an idea: make large small. Turn methods over to each programmer. Dress down. Count on the inscrutability of programming. Promote good news. Turn a leaf away from the sun. Maybe start over.”