Closer To The Ideal

Stuff like this is frustrating. The chemistry books of the era when my older brother was a kid are now considered too dangerous, a fact which fits in with the general trend of America retreating from science (Science is dangerous). There are some risks to chemistry but there are some risks to everything. My dad grew up in the 1930s, wanting to be a chemist. When he was 11 he got really into building bombs, and at one point he set off a bomb in Central Park that had the police running in from every direction. This was in 1938. All the same, he learned a lot. This is the way people learn – they experiment.

The book is controversial, as many of the experiments contained in the book are now considered too dangerous for the general public. There are apparently only 126 copies of this book in libraries worldwide. Despite this, its known as one of the best DIY chemistry books every published.

It was also a source of inspiration to teenager David Hahn, who tried to collect a sample of every chemical element and also built a model nuclear reactor in his back shed in the 1990’s.

Kevin Dewalt suggests that programmers with experience in large corportions are bad for startups, because the emphasis in corporations is on doing things the right way, whereas the emphasis in startups has to be on cutting corners so that you can hip quickly:

So how did I design the database for my most recent startup? Having been in situations where poorly managed database relationships resulted in months of rework, I used foreign keys. That was my experience, and I’m guessing the experience of the developers who made the comments above.

Unfortunately I found that foreign keys in my Rails migrations were a constant source of headaches. Moreover, I decided to migrate to Heroku during the project and had to re-write the foreign keys for Postgres instead of MySQL. What did I learn about customers in this process? Nothing.

In retrospect, I believe I followed bad practice for lean startups building working prototypes. Database integrity is an important issue when you have achieved some measure of success.

May I be so blessed to have these type of problems in my next startup.

Cringley predicts the world of higher education is facing a revolution:

There is enough good material available for free online right now that it would be easy to create a virtual university (WikiVersity?) with the only thing missing being the granting of degrees. It’s that whole “degree from MIT” thing that allows that school not to worry about sharing its lecture bounty, because in the education system lectures are viewed as worthless unless they lead to a degree.

Why is that?

My friend Richard Miller (he designed the Atari Jaguar video game console eons ago) is one of the smartest engineers I’ve ever met yet he doesn’t have a degree in engineering. Apple II designer Steve Wozniak got his degree from UC Berkeley only after leaving Apple in the early 1980s. In both cases their employers couldn’t have cared less.
What drives the education industry is producing degrees while what drives the computer industry is producing products and services.

When was the last time any employer asked to see your academic transcript? Have they ever?

What’s missing here is the higher education equivalent of a GED. Someone will come up with one, or they should, because all the other parts of the system are ready to go.
Cushing Academy, a tony prep school in western Massachusetts, is right now replacing its 20,000-volume library with a “learning center” containing 18 eBook readers, three giant TV screens, and a $12,000 espresso machine. I wonder why they need a building or even a room at all; wouldn’t it be cheaper just to give each kid an eBook reader and a Starbuck’s gift card?

Sounds appealing, doesn’t it? I, for one, always hated school. Always. And I have long wondered why people put up with the current structure of education. But I would not be the entrepreneur that Cringley wants. I would not invest in this particular revolution, for 3 reasons:

1.) Never underestimate the shocking conservatism of the American people, and their general unwillingness to change.

2.) People who go to college are trying to reduce their risks – they want a guarantee that they will be allowed into the middle class (their parents also want that guarantee). The market for safety is not going away any time soon.

3.) The people who are willing to take risks, the true entrepreneurs, are few and far between. (And many of these people don’t bother going to college.)

In other words, so long as the universities can claim that they are reducing risk, they are immune to any challenge from the Internet. So far the Internet has proven itself capable of setting up radical challenges to existing institutions, but it hasn’t replaced any institutions and become the new status quo. The revolutionaries are not yet the new conservatives. We are at least 20 years away from the day when those of us who believe in the Internet can say “4 legs good, 2 legs better.”

Coding Horror has a nice set of quotes and links related to catastrophic software project failures. Of interest to anyone who does project management on software projects:

If Las Vegas sounds too tame for you, software might just be the right gamble. Software projects include a glut of risks that would give Vegas oddsmakers nightmares. The odds of a large project finishing on time are close to zero. The odds of a large project being canceled are an even-money bet (Jones 1991).

In 1998, Peat Marwick found that about 35 percent of 600 firms surveyed had at least one runaway software project (Rothfeder 1988). The damage done by runaway software projects makes the Las Vegas prize fights look as tame as having high tea with the queen. Allstate set out in 1982 to automate all of its office operations. They set a 5-year timetable and an $8 million budget. Six years and $15 million later, Allstate set a new deadline and readjusted its sights on a new budget of $100 million. In 1988, Westpac Banking Corporation decided to redefine its information systems. It set out on a 5-year, $85 million project. Three years later, after spending $150 million with little to show for it, Westpac cut its losses, canceled the project, and eliminated 500 development jobs (Glass 1992). Even Vegas prize fights don’t get this bloody.

Whether we are talking about the evolution of finches  in the Galápagos Islands  or the evolution of the software projects that we work on, my sense is that diversity offers its greatest benefit during  a crisis. The worst thing about monoculture is the powerful reward it offers to pathogens. Or, as Wikipedia says:

The dependence on monoculture crops can lead to large scale failures when the single genetic variant or cultivar becomes susceptible to a pathogen or when a change in weather patterns occur.

Extending that as a metaphor for business, groupthink (a monoculture of thought) can lead to catastrophic failure when some foundational assumption of the group is proven wrong. A monoculture of thought offers a powerful reward to pathogenic behavior. Consider the meltdown at Enron, where top executives all agreed on the profitability of reckless energy trades, and they continued to agree with each other almost till the very moment company declared bankruptcy. Likewise, the top executives at AIG were certain that they had distributed risk in a such a way that the downside of that risk would never catch up with them. – people with dissident viewpoints were squeezed out of their jobs. Or consider the 40 year decline of the United States auto industry, an industry that has suffered more than most from groupthink and inaccurate assumptions. The executives of the 1970s and 1980s felt, despite the gathering evidence, that price was more important to Americans than quality, and that quality automatically meant expensive, and so they lost a generation of car buyers.

A corporate culture that values homogeneity is at grave risk of punishing non-conformists. A good manager is always on guard against the kinds of social bullying, however subtle, that can cause people to censor their opinions. This is a basic task of risk management: reduce risk by challenging core assumptions. Make sure divergent view points are heard.

I should add, if you are working at a new start-up, struggling to find its place in the world, you should treat every day as a crisis.

Genetic diversity allows a population multiple avenues to move forward when a radical change in the external environment dooms the existing species, in their current forms. Genetic diversity helps facilitate the transformation of sub-sections of those populations to evolve into new forms. Likewise, when a corporation faces a crisis, having a diverse range of opinions is healthy, and the more those differences of opinions reach down to core assumptions, the healthier. In boom times, such diversity of opinion could potentially be viewed as annoyingly disruptive of the good times, but in a crisis, what’s needed is the maximum of diversity: in viewpoint, in history, in current circumstances, in goals, in future expectations, etc.

In theory, a genius of a manager could possibly assemble a team made up solely of white males, which still had enough diversity of opinion to perform well in a crisis, but as a practical matter, the most reliable way to put together a diverse team is to recruit people from different backgrounds, different genders, different races and, where possible, different countries.

I regard the cultivation of diversity on a project as a fundamental survival technique, so I devote a lot of time to recruiting newcomers to the field of programming. And so, I read with interest Kirrily Robert’s discussion of recruiting women to work on an open source project (what follows is from Robert’s blog post):

———————————————–

I surveyed women on the Dreamwidth and AO3 projects and asked them about their experiences. You can read a fuller report of their responses on my earlier blog post, Dispatches from the revolution.

One of the first things I asked them was whether they had previously been involved in open source projects. They gave answers like:

I’d never contributed to an open source project before, or even considered that I could.

I didn’t feel like I was wanted.

I never got the impression that outsiders were welcome.

I considered getting involved in Debian, but the barriers to entry seemed high.

Those who got a little further along still found it hard to become productive on those projects:

It’s kind of like being handed a box full of random bicycle parts: it doesn’t help when you don’t know how they go together and just want to learn how to ride a bike.

People without a ton of experience get shunted off to side areas like docs and support, and those areas end up as the ladies’ auxiliary.

But on Dreamwidth and AO3…

What I like most is that there isn’t any attitude of “stand aside and leave the code to the grown-ups”. If there’s something that I’m able to contribute, however small, then the contribution is welcome.

And this one, which is my favourite:

Deep down, I had always assumed coding required this kind of special aptitude, something that I just didn’t have and never would. It lost its forbidding mystique when I learned that people I had assumed to be super-coders (surely born with keyboard attached!) had only started training a year ago. People without any prior experience! Women! Like me! Jesus! It’s like a barrier broke down in my mind.

So, what can we learn from this? Well, one thing I’ve learnt is that if anyone says, “Women just aren’t interested in technology” or “Women aren’t interested in open source,” it’s just not true. Women are interested, willing, able, and competent. They’re just not contributing to existing, dare I say “mainstream”, open source projects.

And this is great news! It’s great news for new projects. If you are starting up a new open source project, you have the opportunity to recruit these women.

I’m going to talk about a real project from a year ago, but I’m changing the name of the project to “Tango”. Here is part of a message that I posted, several months ago, to the Tango project area on our Basecamp account:

Traffic to the website is down 12%. All of the effort expended to encourage people to use the chat rooms has shifted traffic around on the site, but it hasn’t lead to an increase in traffic. Also, the money spent on Facebook ads has turned out to be a complete waste of money – for the number of new visitors that Tango got, twice as much money was spent than would have been spent using the same money to hire new writers.

To some extent, I was criticizing strategies that had been proposed by other people, but which I’d been willing to go along with to see how they might work out. Part of the reply that I got from one of my co-workers:

I’d appreciate a more positive attitude.

I am increasingly afraid of “positive attitudes”. Or rather, what is often called a positive attitude. A project is in deep trouble when the people working on it fail to point out the weaknesses that they see. How can improvements be made without criticism?

Never mind what the real meaning of the phrase “positive attitude” is. As a practical matter, in a business context, what it usually means is “Let’s ignore the bad news and focus on the good news.” But such an attitude only makes sense if you believe the bad news is unfixable. In other words, the bad news indicates that the whole project, the whole company, is doomed. So what is often referred to as a positive attitude is, in fact, a profoundly pessimistic attitude.

In truth, I have a positive attitude: I believed, and still believe, that the project will be a great success. But only if the project’s weaknesses are addressed. And if you look at what I wrote in my Basecamp post, you’ll see it implies a number of things that should be fixed, and which are probably fixable.

What is the point of this thing called a “positive attitude”? Much of it is political – a desire to avoid facing painful realities. Often this is done to keep together a coalition of people – that is, to keep a group of people supportive of a project. We might be talking about the Iraq war, where the American government has relentlessly painted a prettier picture than what reality justified – and this lying was meant to keep the American people supportive of the war. Or we might be talking about a money losing business, in which case the workers and managers might by lying to the investors, so as to keep the investors supportive of the business. And for those who want to make their living that way, this is the one real use of a “positive attitude”.

This is the only positive attitude that a project really needs: the belief that a project can succeed. Once you have that belief, everything else should be criticism: what isn’t working, what needs to be fixed? Happy talk is the enemy of truth, and lies are an effective way to take a viable project and kill it.

This is a horrifying failure of risk management and system administration good practice:  JournalSpace loses all data in its database and has no backup:

Blogging platform JournalSpace (which I’d never heard of to date) has ceased to be, following a wipe-out of the main database for which there was no back-up in place. According to the JournalSpace blog, the database was overwritten as a result of a malicious act from a disgruntled ex-employee.

The lack of backups is the fault of management, for they had the authority to make better decisions, and they had the ethical responsibility to protect the data of their users. Nevertheless, they try to shift the blame to one of their employees:

It was the guy handling the IT (and, yes, the same guy who I caught stealing from the company, and who did a slash-and-burn on some servers on his way out) who made the choice to rely on RAID as the only backup mechanism for the SQL server. He had set up automated backups for the HTTP server which contains the PHP code, but, inscrutibly, had no backup system in place for the SQL data. The ironic thing here is that one of his hobbies was telling everybody how smart he was.

The employee might be guilty of criminal actions here, but that doesn’t let management off the hook for having been so unprepared. If it hadn’t been an employee it might have been a tornado or earthquake or some other disaster – and the blame still would have belonged to management. Multiple backups, in different locations, is the precaution that a responsible company must make.

cb sums it up well in the comments:

lol, gotta love an internet company that has ‘a guy handling IT’. As if the IT side of things is an afterthought-which apparently it was in this case.

There is a second part to this story that I find very sad. One of the users of JournalSpace, a woman calling herself tinythoughts, shows up in the comments at TechCrunch and expresses her sadness, whereupon she is immediately attacked for her having ever used JournalSpace. I am puzzled and worried by the attitude that would defend the company and blame the customer.

This is tinythoughts:

i had one of the oldest journals on journalspace. i am really upset about losing about 6 years of writing, and my layouts which i made. it was bad enough when they lost years of comments. this is far worse. i am pretty sure i archived most of everything up til about a year ago on my external hd. i’m actually a lot sadder about this than i thought i would be.

This is the criticism that is then thrown at her:

If you value your work so much, you shouldn’t be using something that’s free and expect not to lose it.

At the end of the day, piss all you want. It’s your damn fault for leeching off a free service and expect it to continue to provide for you.

The day of FREE is over!

And then this was her response:

it wasn’t free. i was a paying customer for most of the time i was on there, until the first big data loss. after that, i did not get a pro account anymore and also began to write less on there.
as for losing my stuff, which i did value, as i said, i did back it up myself after that first data loss. however, i liked it where and how it was, accessible online to me and anyone else. we are talking about almost 6 years of content. that is not some small thing. even if you’re dumb, you should be able to understand that.
btw, i work in this industry myself, and i am pretty sure the days of free are not over. but all the best to you on being rude anonymously to others online.

She also adds:

I don’t believe their story. I think there is more to it. They’ve had problems before and they always lay it out like their users are technically stupid and willing to accept any dumb answer given to them. What happened really was a great loss for many users, who had been there for years, a community of really great people. The greatest loss is all of the time, life, love, and community each of those users put into journalspace, where it was all documented and washed away like sandcastles on the beach. I might be upset for my own loss, but not nearly as sad as I am for many of my friends there. I think journalspace owes them more than a lame excuse and an empty sorry.

The fact that someone is willing to attack the customers in this case, rather than the grossly irresponsible company, actually saddens me more than the already sad fact that a lot of people lost years worth of work. (Though if I lost that much work, I’d cry for days.)

Sarah Lacy points out that risk aversion and short term thinking are overwhelming the tech industry:

I think there’s also a mindset problem when it comes to venture capital. Investors and many entrepreneurs are no longer focused on building companies and taking real risk. Paul and I did another clip yesterday about Facebook, where he argued it doesn’t make sense for Facebook to stay a stand alone company anymore because the ad markets are going to be locked up for 24 months.

I love P-Ked, but what the hell does a 24 month contraction have to do with building a company? Especially a company that’s still private, growing like mad, has loads of money in the bank and is essentially break even? We’ve got to break ourselves from this quick-fix, quarter-to-quarter mentality of Wall Street– and increasingly Silicon Valley– if any next great tech companies are going to be formed. The very reason great companies are typically started during downturns is they’re started by people who aren’t obsessed with timing a market. They’re started by real entrepreneurs.

In her BusinessWeek column, she also points out that it’s been a bad decade for tech focused venture capital:

But check your calendar. We’re closing in on 2009. And even if the financial system on which VCs depend for returns averts collapse, it’s still in for a few years of serious wound-licking and stepped-up government regulation. Ask anyone in Silicon Valley whether Sarbanes-Oxley had a chill on IPOs.

This realization hit me like a ton of bricks during a recent trip to Boston, coincidentally the same day Lehman Brothers (LEHMQ) filed for Chapter 11. I was sitting down with Tom Crotty, of the venerable Battery Ventures, which had a unique approach to the tech meltdown. Battery, full as it is with more financial gurus than Valley-style engineers, responded by diversifying from traditional startup investments into so-called Private Investments in Public Stocks (PIPEs). Battery also capitalized on the consolidation of cash-rich but fragmented industries.

Crotty hopes the atypical investment approach will insulate Battery, but he nonetheless sees a reckoning coming—specifically toward the end of 2010. He points to 2000 as the “last really good year” for venture capital. Looking back, “the one-year, three-year, and five-year indexes are all going to be terrible,” Crotty says. “And once 1999 and 2000 fall off, the 10-year will be, too. It’s going to be painful.”

But she also points out that there are new markets out there waiting for investment:

That won’t be such a bad thing. After a painful period of forced reckoning with bad past decisions, VC will emerge stronger. Entrepreneurs and the larger U.S. economy will still need venture capital. Some venture capitalists will engineer a new way to make venture-style returns, like Battery Ventures did eight years ago. Many will turn to emerging markets such as India and China. There will be even fewer Google-like (GOOG) home runs. Then again, a leaner, smarter industry may not need as many.

She puts needed qualifiers around the idea that downturns are good for innovation:

To those of you who say platitudes like: “Downturns are GREAT times for innovation!” yes, that’s true, but you are missing the point. This isn’t just the cause of a downturn. This is a structural change in the industry that needs to occur and has been building for nearly a decade. There is far too much money, tech is maturing, and clean tech isn’t mature enough.

Real innovation can not be done in 90 days. Real innovation does not conform to quarterly profit reports. Real innovation comes from those people who can see a potential business when it is still way out on the horizon, and then find ways to reduce the risk of getting from here to there:

People think entrepreneurs are risk-loving. Really what you find is successful entrepreneurs hate risk, because the founding of the enterprise is already so risky that what they do is take their early resources, the small amounts of capital that they have, whatever assets they have, and they deploy those resources systematically, eliminating the largest risk first, the second-largest risk, and so on, and so on.

Jeff Bezos

In the title of this post I used the word “coward”. That is a strong accusation. It is deserved by those who feel the normal risk minimization of entrepreneurs is no longer enough to allow a business to be trusted with money. It is deserved by those who now only place sure bets, and only short term ones, too.

The tech industry, especially the web portion of it, has been infected with speculative excess. During the South Sea Bubble of 1720 financial manipulators raised money for “a company for carrying out an undertaking of great advantage, but nobody to know what it is” and today we have similar ill dealings: “The company will ‘let its purpose and presence be known’ as soon as they reach their goal of raising a Series A capital round of $10 million”.

For the true entrepreneur, the goal is always MR4MR: minimum risk for maximum reward. But they have bold goals that demand big risks be taken — they then set out to find the least risky way of achieving that goal. This applies to many fields. Shakespeare, Napoleon, Thomas Edison and Bill Gates had this common: they were all successful at minimizing the risks they had to run to achieve their goals, yet their goals were so ambitious that even the least risky path still entailed big gambles. The NASA team that first sent men to the moon was fanatic about reducing risk, yet still the mission was among the riskiest endeavors ever undertaken by the human race.

The driving force of the digital revolution were two fold: the increasing power of computer chips, and the dramatic fall in the price of communications, due to breakthroughs in multiplexing. The pulse of change for both has grown slower. Wal-Mart, Fed-Ex, Google and Twitter were all instances of a creative soul finding that the digital revolution allowed a new way to organize human activity. Such surprising alterations to the existing scene will continue as long as there is still growth in computing power and communicative capacity, though the appearance of such novel businesses can be expected to slow as innovation in the underlying technology does.

We should not allow the term “innovation” to be killed by its over-association with the industries that grew during the last few booms. There are other fields desperate for the intervention of  the entrepreneur: green energy, green transport, recycling, lower impact production methods, the education of adults, the teaching of children, child care, environmental clean up. We will soon see a new wave of change in the oldest industry, I mentioned community supported agriculture in the post where I made predictions, in some sense no economic activity is more important than how the human race feeds itself. And at some point in the next 50 years we will see considerable breakthroughs in the way the human race relates to living things on earth, as our understanding of the underlying rules of DNA come into focus.

Downturns are a great time for innovation, but the invitation is only there for those who are thinking long term.

The most concrete way to lower risk, in regards to the creation of websites, is to start small and grow organically. During the design phase, this means you need to adopt an iterative process that demands, with each cycle, the creation of something real.

Boxes and Arrows:

An Iterative Process

While prototyping with XHTML isn’t tied to a specific design process, iterative development seems to effectively leverage its strengths. There are many reasons for this, but perhaps the most significant is that in both cases the prototype, and later the application itself, doubles as a specification. We’ll explore what that means in a bit, but first let’s walk through a suggested process for prototyping with XHTML.Let us start with an overview of the larger design process:In this (iterative) methodology, rather than design the entire application before starting to build it, one designs and builds a unit of the application and then uses what has been built to inform and serve as a starting point for other application units. As with other design methods, the design work begins with sketching, which plays a particularly important role relative to prototyping.

Sketching: A Freeform Question

The term ‘sketching’ refers here to any freeform exploration unconstrained by a specific technology. This includes production of wireframes, which in this model are reframed, as it were, from specification artifact to refined sketch. When thought of, and presented to stakeholders, as sketches, its more natural to discard your wireframes once the design has evolved beyond them. This is usually after a prototype equivalent has been produced. With the design team I work with, we’ve found that when prototyping with XHTML, wireframes often became superfluous, and it’s more effective to go directly from sketch to prototype.

Prototyping: A Concrete Response

Prototyping has a counterpoint relationship to sketching. To paraphrase Bill Buxton, while sketches ask a question—”Is this a good design idea?”— prototypes provide a response. By making the idea manifest, prototypes force upon it the concrete realities and user experience idiosyncrasies of the actual production technology and offer a crisp verdict on the quality of what you dreamed up in drawings.

The Prototype/Build Relationship

When prototyping with XHTML, especially in an iterative model, the build and prototype become very intertwined. If you’re prototyping a new application or product, the XHTML prototype is essentially a rough draft of the actual application. However, when updating the design of an existing application, the production version can serve as the starting point for the prototype of the new solution.

Shelley Powers once wrote that she wouldn’t use an online service unless she was sure of the exit route:

I won’t use a hosted web service like Typepad or weblogs.com. It’s too easy for them to decide that you’re ‘violating’ terms of service, and next thing you know, all your weblog entries are gone. I saw this with wordpress.com in the recent events that caused so much discussion: in fact, I would strongly recommend against using wordpress.com because of this–the service is too easily influenced by public opinion.

I don’t use either my Yahoo or Gmail mail accounts. Regardless of whether I can get a copy of my email locally, if I decide to not use either account I have no way of ‘redirecting’ email addresses from either of these to the email address I want to use. (Or if there is a way, I’m not aware of it.) Getting a copy of my data is not an exit strategy–it’s an export strategy. An exit strategy is one where you can blow off the service and not suffer long-term consequences. A ‘bad’ email address is definitely a long-term consequence*.

A few months ago we started using Stikipad to get ourselves organized. We started using the site under conditions that were close to an emergency – we had a web site that was past its deadline, and we were trying to regain control over a situation that had become chaotic. We began to post bug reports and notes to ourselves. Stikipad was useful to us as an easy-to-use online notepad, which we could use informally. It helped that it also had certain wiki features – it kept track of who made each edit, and it allowed us to revert changes when we made mistakes in editing.

Since then, we’ve started listing all the hours that we work on there. This data is vital for when we send out invoices to our clients. Also, all of the long, complicated to-do lists, for each project we’ve been working on, are all on there. We did not realize how important the site had become to us – we’d set up a quick, free account as a simple way to organize one project, but our use of Stikipad has grown so that lately it has been central to the way we schedule our time.

For the last three days, when we go to the site, the only thing we get is this error page:

The whole entire site has been down. You could not start a new account, nor reach any of the pages on our account, nor even reach the “Support” page. We all kept trying, at different times during Friday, Saturday and Sunday. There was no way into the site.

The site just came back to life tonight. But we are planning on giving it up. We feel we can’t trust it anymore. There has been no word on the Stikipad blog about what just happened. Their silence does not inspire confidence in us.

Stikipad does have an export option, which we could use religiously to keep our data safe. I blame myself for not already automating a daily download of this data. I’m fixing this particular oversight tonight. All the same, Stikipad can’t value our most vital data to the same extent we can, so it is perhaps best if we keep that data on our own server, and make the multiple backups of that data which we feel is needed.

Alex Marshall has up a great essay at Spotlight on the Region. He suggests large construction projects are now taking longer than they used to:

It took four years to complete in 1904 the city’s first subway system, much of it hacked away using pick axes and mules.

It took less than a decade to complete in 1915 the central water line from the Catskill Mountains to New York City, a mammoth 92-mile aqueduct and tunnel system worthy of the Roman Empire.

It took just over a year to complete in 1930 the Empire State Building, the tallest building in the world by far at the time, an intricately detailed structure of tiles and brick.

Today things generally take a lot longer, whether it’s the public or the private sector at work.

The first portion of the 2nd Avenue subway, which will be built using state-of-the–art tunnel-boring machines, will take six years to complete – if all goes well and enough money is found. Running from 96th Street on the Upper East Side to 63rd Street, it will have just three new stations, spaced every ten to 14 blocks.

Compare that to the city’s first subway line, the IRT, which ran from City Hall up to Grand Central, over to Times Square, and then up to 145th street. The IRT had 28 stations as well as local and express lines. And it was all built in just four years.

So just to make this contrast clear, the first phase of the 2nd Avenue subway in terms of track is a tenth the size of the original IRT and has three stations as opposed to 28. Yet it will take six years to complete as opposed to four for the much larger IRT. Why is this? …

Almost every engineer or administrator I ask about this seems to agree that things do take longer today, but explanations vary. The usual suspect is environmental and regulatory review, but I’m not including that in my perspective. I’m just talking about actual construction time.

The modern, affluent standard of living depends on society’s infrastructure, both physical and intellectual: airports, highways, bridges, ports, telecommunication networks, databases, power plants, and the software that makes it all run. 150 years ago it was still common for massive bridge projects to meet with catastrophic failure. The engineers of the 1800s slowly mastered the art of working with iron and then steel. The projects grew in scale, and the skills of the project managers needed to keep apace.

Nowadays we rarely hear of catastrophic bridge failures. This is a particular type of infrastructure that has come to be well understood, both in its engineering and in the project management that oversees its construction. However, we, as a society, are still struggling to find the right way to develop large software projects. This is the newest type of infrastructure, and the correct way to engineer it and manage its development is still a source of controversy.

Robert L. Glass has compiled a book of entertaining and instructive stories regarding the massive software project failures of our time: Software Runaways: Monumental Software Disasters. Anyone with an interest in the management of software projects should read it.

The worst of these project failures, the most expensive and the most ambitious, was the attempt made by the Federal Aviation Authority to modernize the computer system it uses to keep track of what planes are in the air. The effort began in 1981 and ended in complete failure in 1994. The government hired IBM to do the actual work, and over the course of 14 years, IBM burned through $3.7 billion dollars. Nothing was accomplished. The project was finally shut down by Congress. Nothing came out of the project, not a single piece of software, nor even a line of code, was ever used for anything.

Robert N. Britcher, who was involved with the project, offers a full write up of the project. I will try to entice you to buy the book with some excerpts:

The Advanced Automation System began, in concept, in 1981 and ended in 1994, “terminated for convenience” by the government. Billions of dollars were spent on it. It is hard to describe. You can’t learn anything from the name. You know it’s about air traffic control because I told you, or because you read about it in the papers. Maybe part of the problem was the name. It sounds like the system to end all systems.

One engineer I know described the AAS this way. You’re living in a modest house and you see the refrgerator going. The ice sometimes melts, and the door isn’t flush, and the repairman comes out, it seems, once a month. And now you notice it’s bulky and doesn’t save energy, and you’ve seen the new ones at Sears. So it’s time. The first thing you do is look into some land a couple of states over and think about a new house. Then you get I.M. Pei and some of the other great architects and hold a design run-off. This takes awhile, so you have to put up with the fridge, which is now making a buzzing noise that keeps you awake at night. You look at several plans and even build a prototype or two. Time goes on and you finally choose a design. There is a big bash before building starts. Then you build. And build. The celebrating continues; each brick thrills. Then you change your mind. You really wanted a Japanese house with redwood floors and a formal garden. So you start to re-engineer what you have. Move a few bricks and some sod. Finally, you have something that looks pretty good. Then, one night, you go to bed and notice the buzzing in the refrigerator is gone. Something’s wrong. The silence keeps you awake. You’ve spent too much money! You don’t really want to move! And now you find out the kids don’t like the new house. In fact, your daughter says “I hate it”. So you cut your losses. Fifteen years and few billion dollars later, the old refridgerator is still running. Somehow.

At $3.7 billion, the Advanced Automation System was one of the largest civilian computer contracts ever; maybe the largest. It was the largest single contract in IBM’s history. From the moment it was awarded, until near the project’s demise, IBM patted itself on the back. There was something for everyone, beginning with a great ball in Union Station, featuring Chubby Checker and “The Twist”.

At its peak the project employed over 2,000 people. About a million dollars a day. If you thought like the IBM project manager, this was a good deal. Many people were working and money was being made. It was going to last forever. No one considered that it wouldn’t. And everyone was getting ahead. (One of the ironies of conceptual work is that it is easy to believe you are farther along than you are.)

The AAS must have been the most supervised project in history: this atop its enormous size and complexity, and the extreme and constantly changing requirements. One programmer described it this way: “Working on the project was like working on a car inside the garage with the motor running. Eventually, even the crickets hopping around the tires suffocate.”

What I saw on the FAA’s Advanced Automation System would have made Sisyphus weep… Whatever commitment and discipline there was… was worn down by a battery of watchfulness that I can only ascribe to fear of failure. In spite of tens of millions of dollars spent on new computers for AAS, the most important piece of equipment on the project was the overhead projector. There were endless meetings attended by dozens of people – as if we were never quite sure about the whole thing. The people in charge simply lacked the confidence and the finesse of the space team: NASA, contractors, and astronaughts.

It has been noted by everyone from the New York Times to the Vice-President of the United States that the main problem on the Advanced Automation System was “changing requirements”. For those involved in large-scale computer systems, that is nothing new. No one can perfectly surmise the shape and feel of a system years in advance. Even replacing some aspect of a system you know by heart is not immune from thinking twice about it. … [But] the requirements churn (it was called) on the Advanced Automation System was not normal. It was the result of our enchantment with the computer-human interface, the CHI. The new controller workstation, fronted by a 20″ by 20″ color display, because it was capable of seemingly endles variety of presentations, mesmerized the population of AAS like the O.J. Simpson trial mesmerized the nation…

The project was handed over to human factor pundits, who then drove the design. Requirements became synonymous with preferences. Thousands of labor-months were spent designing, discussing, and demonstrating the possibilities: colors, fonts, overlays, reversals, serpentine lists, toggling, zooming, opaque windows, the list is huge. It was something to see. (Virtually all of the marketing brochures – produced prematurely and in large numbers – sparkled with some rendition or other of the new controller console.) It just wasn’t usable…

The cost of what turned out to be a 14-year human factors study did not pay off. Shortly before the project was terminated a controller on the CBS evening news said: “It takes me 12 commands to do what I used to do with one.” I believe he spoke for everyone with common sense.

Rummaging through one of the closets at the far end of the hall on the fifth floor one day, looking for some standards document, I found an envelope left by someone who left the company – as many did after so many years advancing against stone, while the wheels of commerce were accelerating on what everyone referred to as “the outside”. It contained “A Brief History Of The Advanced Automation System”. It was printed by hand and left, perhaps inadvertantly, or perhaps with the hope that some anthropologist might some day discover it and make a pronouncement. In every important way, it is the truth:

“A young man, recently hired, devotes years to a specification written to the bit level for programs that will never be coded. Another, to a specification that will be replaced. Programmers marry one another, then divorce and marry someone in another subsystem. Program designs are written to severe formats, then forgotten. The formats endure. A man decides to become a woman and succeeds before system testing starts. As testing approaches, she begins a second career on local television, hosting a show on witchcraft. An architect chases a new technology, then another, then changes his mind and goes into management. A veteran programmer writes the same program a dozen times, then transfers. The price of money increases eight times. Programmers sleep in the halls. Committees convene for years to discuss keystroking. An ambitious training manager builds an encyclopedia of manuals no one will ever use. Decisions are scheduled weeks in advance. Workers sit in the hallways. Notions of computing begin in the epoch of A, edge toward B, then come down hard on A + B. Human factors experts achieve Olympian status. The Berlin Wall collapses. The map of Europe is redrawn. Everything is counted. Quality becomes mixed with quantity. Morale is reduced to a quotient, then counted. Dozens of men and women argue for thousands of hours: What is a requirement? A generation of workers retire. The very mission changes and only a few notice. Programming theories come and go. Managers cling to expectations, like a child to a blanket. Presentations are polished to create an impression, then curbed to cut costs. Then they are studied. The work spikes and spikes again. Offices are changed a dozen times. Management retires and returns. The contractor is sold. Software is blamed. Executives are promoted. The years rip by with no end in sight. A company president gets an idea: make large small. Turn methods over to each programmer. Dress down. Count on the inscrutability of programming. Promote good news. Turn a leaf away from the sun. Maybe start over.”