Closer To The Ideal

On LinkedIn, I had a useful conversation with Kris Herlaar, which I will repeat here.

Personal, individual frameworks made a lot of sense in 2000 and 2001 and 2002, but now? Of my 3 most recent contracts, 2 were rebuilds of sites built by some other PHP programmer. In some cases the code was good, but the framework was their own, all together personal, and undocumented.

I think we betray our clients if we use personal frameworks in 2010. You can build a great site using Cake, Symfony, CodeIgniter, or WordPress, Joomla, or Drupal. You can build a great site using modified versions of existing CMSs, or using an open source framework to build your software. These open source projects offer standardization, and standardization is important. These open source projects offer documentation, and documentation is important.

Kris wrote:

People with different backgrounds have different ideas about what is good, and there are probably frameworks for every possible set of ideas you and I could think of.

Yes, and this fact is expensive for my clients.

I do think what I’m writing about here is, in fact, a trend. When I look for new PHP gigs, I notice more and more of them require the use of some framework or CMS. The most common requests are Drupal, WordPress, Joomla, Symfony and CodeIgniter. My sense is that businesses are realizing that it is expensive to allow a programmer to invent their own framework, so the businesses are insisting on the use of some open source software which is well documented.

Here is the scenario that I have seen a lot during the last 5 years:

A company hires a PHP programmer. The company (called “abc”) says “Build us a website that does xyz.” The programmer invents their own software and builds xyz. Since the programmer thinks they are working alone, they allow themselves to cut corners.They copy and paste code. They do not refactor as much as they should. They do not document much, since they assume they are the only ones working with the code.

After 3 years, the programmer leaves to take a new job somewhere else. The “abc” company now hires a new PHP programmer. The company says “Please fix all the bugs in xyz”. The new programmer struggles to learn the code written by the last programmer. There is no documentation regarding how the software works. The last programmer, now gone, rarely responds to questions via email.The new programmer eventually figures out that the old code had some architecture, but the exceptions to the architecture continue to baffle. And why does the code repeat in some places? Is that a bit of copy and paste, or was that really necessary somehow?

After the bugs are fixed, the company says “Great, now, lets extend xyz and add in features lmnop.” Extending the old code is difficult for the new programmer, since they have no idea how the last programmer intended for the code to be extended. More so, the last programmer didn’t even foresee some of the new needs of lmnop so the old code is badly suited to the new job. The new programmer struggles with the task of adding features while keeping the code base backwards compatible. A job that should take 1 month instead takes 2 months.

Finally, with that done, the company says to the programmer, “Great, now we are going to start a totally new project, the def project. You can use any technology you want for this. You can invent your own framework for this, if you want.” The new programmer rejoices! Finally, they will be allowed to do things their way! They will get to write code that makes sense to them! They will get to build an architecture that makes sense to them! The programmer invents their own software and builds def. Since the programmer thinks they are working alone, they allow themselves to cut corners.They copy and paste code. They do not refactor as much as they should. They do not document much, since they assume they are the only ones working with the code.

After 3 years, the programmer leaves to take a new job somewhere else. The “abc” company now hires a new PHP programmer. The new programmer must now work with the patched, extended, bloated project of xyz. They must also work with def, which they find confusing. Each project was written by a different programmer, uses a different architecture, makes different assumptions about what is good code.

You see the problem? This is not sustainable.

Consider Kris’s words:

“People with different backgrounds have different ideas about what is good, and there are probably frameworks for every possible set of ideas you and I could think of.”

That is the problem that can be fixed by committing to a framework.

Just to be clear, I think the choice of a framework might be arbitrary, and yet will still be useful. That is, the framework does not have to be the “best”. It merely needs to offer consistency. There is a great benefit to consistency, especially over the long term. I mean, it is good if you can get several programmers who will work with some basic set of core assumptions – a decent framework will offer that.

Another problem that I have seen a lot of is the programmer who thinks they are working alone. Either they are the only programmer at the business, or they are assigned some project that belongs to them and which no other programmer is allowed to touch. So they think they are alone. But after a few years, they leave the company, and some other programmer is brought in to handle their work.

Over time, all projects involve multiple programmers. Programmers seem to be slow to recognize this. After some years, they will move on, and some other programmer will have to work on their code. I wish more programmers could be made to see this.

I am especially aware of this right now, because, as I said, 2 of my last 3 gigs I have been brought in to save/rebuild code left by some other programmer.

Tim Bray:

What I’m writing here is the single most important take-away from my Sun years, and it fits in a sentence: The community of developers whose work you see on the Web, who probably don’t know what ADO or UML or JPA even stand for, deploy better systems at less cost in less time at lower risk than we see in the Enterprise. This is true even when you factor in the greater flexibility and velocity of startups. This is unacceptable. The Fortune 1,000 are bleeding money and missing huge opportunities to excel and compete.

The word “agile” is overloaded with multiple meanings, but long before it became a fad in the software world, it was a subject of study for academics who were trying to improve the efficiency of supply-chains at large firms. And that is basically what Tim Bray is talking about: the efficiency of supply-chains at large firms.

Regarding software development, I wrote about agile practices in Geography still matters: physical proximity is essential to agile practices.

Using agile practices in large organizations is a tough issue and it always has been. Large firms must, by their very nature, be about hierarchy, whereas agile practices depend on trust, and in some respect trust and hierarchy are opposites. For someone at the top of a large hierarchy to send an order down to the bottom, they are implicitly overriding whatever the lower level employees might think best – therefore, a lack of trust is implied in every order.

In the 1990s, DARPA invested some money in agile research, which lead to such books as that by Goranson: Agile Virtual Enterprises. Goranson looks back at the whaling industry of the mid-1800s and examines how 2 small towns in Massachusetts were able to capture 90% of the whaling industry, when most of the actual hunting was done in the South Pacific. He found that the patterns of hiring and team building had much in common with what start-ups aspire to nowadays – a tight-knit, fast-forming team of highly competent professionals held together by a commonly understood body of professional ethics and expectations. That environment sounds a lot like the startup scene that now generates new web companies.

The curious thing is how little has changed: small firms arising from a tight-knit, high trust culture continue to lead the way in innovation, whereas large firms continue to lag behind. But large firms can possibly afford to lag behind, they have other strengths, including the capital that they are charging monopoly rents on.

I think Tim Bray is terribly wrong when he suggests that the way forward is, basically, to just do stuff better:

Here’s a thought experiment: Suppose you asked one of the blue-suit solution providers to quote you on building Ravelry or Twitter or Basecamp. What would the costs be like? And how much confidence would you have in a good result? Consider the same questions for a new mobile-network billing system. ¶

The point is that that kind of thing simply cannot be built if you start with large formal specifications and fixed-price contracts and change-control procedures and so on. So if your enterprise wants the sort of outcomes we’re seeing on the Web (and a lot more should), you’re going to have to adopt some of the cultures and technologies that got them built.

It’s not going to be easy; Enterprise IT has spent decades growing a defensive culture based on the premise that you only get noticed when you screw up, so that must be avoided at all costs.

I’m not the only one thinking about how we can get Enterprise Systems unjammed and make them once again part of the solution, not part of the problem. It’s a good thing to be thinking about.

In my opinion, there are certain problems that come up at large scales that will never get fixed, because they are fundamental to that scale. Scale determines the context. Consider mass: I am not pulled toward an ant, a house or a mountain, but I am pulled toward planet Earth, because once you get enough mass in one place, you start to notice its gravity. I do not think that the gravity of planet Earth will ever be “fixed”, I think it is normal that the amount of mass that makes up the Earth should have the Earth’s gravity. Likewise, I do not think hierarchy (and its attendant risk avoidance) will ever be fixed in large organizations, because I think hierarchy is natural to large organizations.

However, things will get smaller. And that is the real solution. And it has been going on for a long time now. Since the recession of 1991, the Fortune 500 have shrunk significantly, both as a percentage of the GDP, and as a percentage of total workforce employed. And there are more small firms.

That is the fix that I see: we will not fix large organizations, but we will have less of them.

When you are in the office 90 hours a week, frequent work breaks are important.

Working 90 hours a work week requires frequent, and highly effective, work breaks. In the center of Macintosh work area in Bandley 3 we had a ping pong table, a nice stereo system, and a Defender video game machine. We found that competitive play gave us a jolt of adrenaline, and a refreshed mind-set when we resumed work. We also learned a lot about our coworkers and how they excel during competition. While playing Defender one day I got some great insight into how Burrell accelerates his own learning process.

…One day Burrell started doing something radical. Andy came by my cube and said “You’ve got to come see what Burrell’s doing with Defender.” “How can you innovate with a video game?” I wondered. I’d seen Burrell and Andy innovate on all kinds of things, but I couldn’t image how he could somehow step outside the box of a video game – the machine controlled the flow and dictated the goals. How could you gain some control in that environment?

Some people argue that 90 hour work weeks are inefficient because eventually people get tired. That would depend on what “efficient” means. It is possible that there is a law of diminishing returns that sets in once people have worked more than 20 or 30 hours in a week. But so what? If you are only 25% as efficient, after 80 hours, as you were during your first hour, then you are still getting work done. In such a scenario you are only achieving 15 minutes of work for every hour that you work, but you are still moving forward. And there are a lot of projects where that is what is needed – every possible minute that can be put in on the project. No matter how far down you go on the declining tail of diminishing returns, you are still getting some additional output for every hour worked.

But I would also argue with those who say there is a law of diminishing returns. I’d say, just as often, there is a law of increasing returns that applies to situations where tough problems must be solved. How often do you think Einstein thought about relativity during 1904/1905? Maybe 100 hours a week? 120? Every waking minute? I think the important phrase is “time on task”. It is crucial for big breakthroughs. There is a huge difference between playing videos at work, knowing you are going back to work, and playing videos at home, knowing you’ll be chilling out for the rest of the evening. Context is everything. In one situation your brain works on the problem, in the background, in the other situation your brain drifts away from the problem. Going home means shifting gears and thinking about other things. Staying at work means staying basically focused on your task, even if you take frequent recreational breaks.

Colin Steele, on the difficulties of communicating what you mean:

Meaning is heavy stuff. And, as it turns out, difficult to transmit using language. Why? Because language (words) sits one level of indirection away from the models themselves. Follow me, here.

We have the thing itself. We’ll stick with “apple”. Out there, somewhere, is the apple we’re talking about. Then, one level removed from that, are the sensory experiences associated with that apple. (Level of indirection: 1). Aggregated in your head is that sensory input, memories, associations, etc — the mental model of that apple. (Indirection now 2.) Then there is the word I slap onto that model – “apple”. Note that the word represents the model, but isn’t the model itself. (Indirection: 3.) Phew. We’re three hops from the damn piece of fruit, and just getting rolling…

Now I try to talk about the apple. I use the word apple, and a bunch of other words that are intended to connect the dots – make the same associations for your version of “apple” as I have attached to mine. Again, add a level of indirection. (4.) You aggregate into your “apple” model. (5.) You slap the word “apple” on that model. (6.)

Yep. That’s lossy. Your copy of the “apple” model that I’ve tried to convey to you sits about 6 hops away from the thing itself. Maybe more, I’m not sure. And there are no checksums.

That nicely sums up the way that meaning gets lost as it moves from one person to another. Colin then suggests that storytelling is the most effective tool to protect against meaning-loss, and the most compressed version of a story is a metaphor:

Then, invent a defining, central story. A system metaphor captures the essence of the system in the most effective communication tool we have – the story.

This suggests to me the importance of having relationships that last a while. Friendship (or at least prolonged exposure to someone) acts as a kind of checksum. What is a checksum after all, but a pattern whose presence we’ve learned to expect? With old friends, we look for that pattern, we check what they are saying right now against the pattern we’ve seen in the past. If you try to tell me about an apple, and if we’ve known each other awhile, then I’ve all our previous conversations to refer back to. I can recall the conversations we’ve had about pears, grapes and oranges. Possibly I might remember that you prefer tart fruit to sweet, so I know your apple is probably more of a macoun and less of a mutsu – our previous conversations help me narrow down what kind of apple you mean.

This has an interesting implication for work, especially web development. It suggests areas where hiring freelance workers is perhaps inappropriate. To the extent that hiring freelance workers suggests loose, impermanent relationships, then freelancers are a bad choice for startups. Since long relationships make it slightly easier to communicate meaning, and since startups must wrestle constantly with defining their aims and goals, then startups are probably the kind of environment that should prefer long term workers instead of freelancers.

One other thing that this brings up, Colin is partly talking about the illusion of agreement.

Whenever this issue comes up, I’m reminded of 37 Signals philosophy of getting real. In terms of web work, getting real means that you should work with images, rather than functional specs:

Functional specifications documents lead to an illusion of agreement. A bunch of people agreeing on paragraphs of text is not real agreement. Everyone is reading the same thing, but they’re often thinking something different. This inevitably comes out in the future when it’s too late. “Wait, that’s not what I had in mind…” “Huh? That’s not how we described it.” “Yes it was and we all agreed on it — you even signed off on it.” You know the drill.

Functional specifications document are “yes documents.” They’re political. They’re all about getting to “yes” and we think the goal up front should be getting to “no.” Functional specs lead to scope creep from the very start. There’s very little cost in saying “yeah, ok, let’s add that” to a Word document.

As much as possible, you want to make your product real, as fast as possible, so that people in your company can argue over the actual thing, rather than their different conceptions of what that thing should be. If you have the time to build a simple prototype, you should do so. It is true, of course, that in a literal sense “there are no checksums” when 2 people are trying to communicate complex ideas to one another about what direction a software project should take. But there are ways to create checks, to ground the conversation as close to reality as possible, and images of how the software should look on screen is a good one. And simple prototypes of the interface is another good one. 2 people can go very far astray when they argue over words in a document, but when they are looking at an image of the interface, the image helps limit the number of places they can misunderstand each other.

Dustin of MarxSoftware lists some of the problems that arise when programmers are afraid to ask questions:

Inferior Solution

If a developer is not comfortable asking questions of more experienced (or even differently experienced) developers, it is more likely that the developer will not come up with the optimal solution. I have seen in my career several times when a developer did something less efficiently or in a non-standard or non-maintainable way simply because he or she did not know there was a better, more efficient, more standard, and/or more maintainable way. Some of these cases have occurred because the offending developer did not want to admit lack of knowledge of what he or she was trying to do. In one example, the developer wrote his own XML parsing code because he could not get the Xerces-C XML parser working. It was a simple matter of making his XML well-formed, but he did not know this and did not ask for help and ended up wasting many hours writing a custom “XML” (mostly) parser.

I read this and I thought about my experience trying to manage a remote team in India. Here is a problem I noticed with the team – they were terrified to ask questions. When an American company out sources work to India, the American company is often looking for the lowest possible price. This leads to a lack of trust, and the lack of trust then undermines the project, partly due to the reasons that Dustin is describing – the team in India does not want to ask questions because they do not want to admit to not knowing something. And it doesn’t matter that I was on Skype every night to answer questions. I could create a relationship with the programmers over there, but it did not override the relationship (or lack thereof) that my manager was creating with their managers.

In my post How Much Do Websites Cost I wrote of those mistakes which I kept seeing clients make, over and over again. My list somewhat overlaps with the list of classic mistakes that Steve McConnell writes about:

People Mistakes

1. Undermined motivation
2. Weak personnel
3. Uncontrolled problem employees
4. Heroics
5. Adding people to a late project
6. Noisy, crowded offices
7. Friction between developers and customers
8. Unrealistic expectations
9. Lack of effective project sponsorship
10. Lack of stakeholder buy-in
11. Lack of user input
12. Politics placed over substance
13. Wishful thinking

Process Mistakes

14. Overly optimistic schedules
15. Insufficient risk management
16. Contractor failure
17. Insufficient planning
18. Abandonment of planning under pressure
19. Wasted time during the fuzzy front end
20. Shortchanged upstream activities
21. Inadequate design
22. Shortchanged quality assurance
23. Insufficient management controls
24. Premature or too frequent convergence
25. Omitting necessary tasks from estimates
26. Planning to catch up later
27. Code-like-hell programming

Product Mistakes

28. Requirements gold-plating
29. Feature creep
30. Developer gold-plating
31. Push me, pull me negotiation
32. Research-oriented development

Technology Mistakes

33. Silver-bullet syndrome
34. Overestimated savings from new tools or methods
35. Switching tools in the middle of a project
36. Lack of automated source control

He’s added some new ones:

* Confusing estimates with targets
*
Excessive multi-tasking
*
Assuming global development has a negligible impact on total effort
*
Unclear project vision
*
Trusting the map more than the terrain
*
Outsourcing to reduce cost
*
Letting a team go dark (replaces the previous “lack of management controls”)

Of these, the ones that I’ve seen my clients make would include:

Unclear project vision

Outsourcing to reduce cost

Feature creep

Lack of effective project sponsorship

Unrealistic expectations

Insufficient management controls

Overly optimistic schedules

These tend to be related. I’ve mostly worked with inexperienced entrepreneurs whose inexperience leads them into a manic-depressive cycle of over-confidence followed by extreme self-doubt, followed again by over-confidence. The lack of confidence leads to feature creep, which grows out of a desire to be all things to all people. This reflects an “unclear vision for the project”, which leads to a “lack of effective project sponsorship”. The lack of experience also leads to “unrealistic expectations” which feed into “overly optimistic schedules”. But all of these are really just parts of the larger problem of wishful thinking, which McConnell describes at length:

I am amazed at how many problems in software development boil down to wishful thinking. How many times have you heard statements like these:

“None of the team members really believed that they could complete the project according to the schedule they were given, but they thought that maybe if everyone worked hard, and nothing went wrong, and they got a few lucky breaks, they just might be able to pull it off.”

“Our team hasn’t done very much work to coordinate the interfaces among the different parts of the product, but we’ve all been in good communication about other things, and the interfaces are relatively simple, so it’ll probably take only a day or two to shake out the bugs.”

“We know that we went with the low-ball contractor on the database subsystem and it was hard to see how they were going to complete the work with the staffing levels they specified in their proposal. They didn’t have as much experience as some of the other contractors, but maybe they can make up in energy what they lack in experience. They’ll probably deliver on time.”

“We don’t need to show the final round of changes to the prototype to the customer. I’m sure we know what they want by now.”

“The team is saying that it will take an extraordinary effort to meet the deadline, and they missed their first milestone by a few days, but I think they can bring this one in on time.”

Wishful thinking isn’t just optimism. It’s closing your eyes and hoping something works when you have no reasonable basis for thinking it will. Wishful thinking at the beginning of a project leads to big blowups at the end of a project. It undermines meaningful planning and may be at the root of more software problems than all other causes combined.

Kevin Dewalt suggests that programmers with experience in large corportions are bad for startups, because the emphasis in corporations is on doing things the right way, whereas the emphasis in startups has to be on cutting corners so that you can hip quickly:

So how did I design the database for my most recent startup? Having been in situations where poorly managed database relationships resulted in months of rework, I used foreign keys. That was my experience, and I’m guessing the experience of the developers who made the comments above.

Unfortunately I found that foreign keys in my Rails migrations were a constant source of headaches. Moreover, I decided to migrate to Heroku during the project and had to re-write the foreign keys for Postgres instead of MySQL. What did I learn about customers in this process? Nothing.

In retrospect, I believe I followed bad practice for lean startups building working prototypes. Database integrity is an important issue when you have achieved some measure of success.

May I be so blessed to have these type of problems in my next startup.

Great essay by Paul Graham (from 2003):

The example of painting can teach us not only how to manage our own work, but how to work together. A lot of the great art of the past is the work of multiple hands, though there may only be one name on the wall next to it in the museum. Leonardo was an apprentice in the workshop of Verrocchio and painted one of the angels in his Baptism of Christ. This sort of thing was the rule, not the exception. Michelangelo was considered especially dedicated for insisting on painting all the figures on the ceiling of the Sistine Chapel himself.

As far as I know, when painters worked together on a painting, they never worked on the same parts. It was common for the master to paint the principal figures and for assistants to paint the others and the background. But you never had one guy painting over the work of another.

I think this is the right model for collaboration in software too. Don’t push it too far. When a piece of code is being hacked by three or four different people, no one of whom really owns it, it will end up being like a common-room. It will tend to feel bleak and abandoned, and accumulate cruft. The right way to collaborate, I think, is to divide projects into sharply defined modules, each with a definite owner, and with interfaces between them that are as carefully designed and, if possible, as articulated as programming languages.

I recently worked on a project where all the code was shared by all of the programmers. What a disaster. Each programmer had their own way of doing things. And each programmer wanted to use whatever library they felt most comfortable with. Thus, for the Javascript, the project included jQuery, Prototype, and a whole bunch of custom libraries that various programmers had written. Even worse, we had a dispersed team, so none of us ever got to meet one another. We had 2 programmers in New York City, one in North Carolina, one in Europe, and about 8 at a firm in India. And, even worse still, everyone was part time, so no one saw any need to make an emotional investment in the project, or even to study what the other programmers were doing. In short, nearly every mistake that could be made, was made. And I am certain that if the project had been handled by 2 programmers who worked on it full time, and who were in the same city and who could meet each day, then the project would have gone faster, cost less, and had less bugs. (I was nominally the “lead programmer” but all of my decisions were over-ruled by the project manager – he did not get that hacking is an art.)

This is also good:

One way to tell how good people are at empathy is to watch them explain a technical question to someone without a technical background. We probably all know people who, though otherwise smart, are just comically bad at this. If someone asks them at a dinner party what a programming language is, they’ll say something like “Oh, a high-level language is what the compiler uses as input to generate object code.” High-level language? Compiler? Object code? Someone who doesn’t know what a programming language is obviously doesn’t know what these things are, either.

Martin Fowler has a great post about technical debt:

There’s been a few posts over the last couple of months about TechnicalDebt that’s raised the question of what kinds of design flaws should or shouldn’t be classified as Technical Debt.

A good example of this is Uncle Bob’s post saying a mess is not a debt. His argument is that messy code, produced by people who are ignorant of good design practices, shouldn’t be a debt. Technical Debt should be reserved for cases when people have made a considered decision to adopt a design strategy that isn’t sustainable in the longer term, but yields a short term benefit, such as making a release. The point is that the debt yields value sooner, but needs to be paid off as soon as possible.

To my mind, the question of whether a design flaw is or isn’t debt is the wrong question. Technical Debt is a metaphor, so the real question is whether or not the debt metaphor is helpful about thinking about how to deal with design problems, and how to communicate that thinking. A particular benefit of the debt metaphor is that it’s very handy for communicating to non-technical people.

I think that the debt metaphor works well in both cases – the difference is in nature of the debt. A mess is a reckless debt which results in crippling interest payments or a long period of paying down the principal. We have a few projects where we’ve taken over a code base with a high debt and found the metaphor very useful in discussing with client management how to deal with it.

The debt metaphor reminds us about the choices we can make with design flaws. The prudent debt to reach a release may not be worth paying down if the interest payments are sufficiently small – such as if it were in a rarely touched part of the code-base.

So the useful distinction isn’t between debt or non-debt, but between prudent and reckless debt.

I’ve written about this before. Over the last 30 years women have made dramatic gains in field such as law, medicine, management and science. Almost 50% of new doctors are women, so that is a field that has achieved near perfect equality. But there is something wrong with the tech industry, and with computer programming in particular. The percent of computer scientists who are female and who are earning advanced degrees peaked in 1989 and has since retreated. What explains this retreat, when women are still advancing in almost all other professions?

A month ago, Bruce Byfield wrote about sexism in the free software movement, and his post ignited a firestorm. Now Byfield reflects on the angry reactions that his discussion of sexism invoked:

That brings up another point I’ve learned: people who are not consciously sexist themselves tend to be unable to see institutionalized sexism around them. They are not aware of any prejudice against women in themselves, so how could there be any sexism involved? They seem unaware that institutions and customs can be sexist simply by what they value or how they operate, that even something like a discourse developed by men talking to men can institutionalize sexism. Nor do they understand that, by simply accepting such institutions or ways of acting, they become supporters of sexism.

For instance, I am currently part of an email conversation with a prominent FOSS community member who has been pilloried who is hurt and baffled that I (or anyone else) could apply the word “sexism” to them. Their reasoning? They did not intend to be sexist, so therefore they can’t possibly be. Therefore, labelling their behavior as unacceptable is unfair, they argue. The fact that, in context, their actions and remarks could not possibly be described in any other way honestly does not seem to have occurred to them. No matter what I say, they remain hurt and baffled — and, like so many, deeply in denial.
Cutting across existing lines

…I’m not sure, but I think that the logic here is that if you are already part of an idealistic movement, your actions must be above criticism in every other sense. From that assumption, perhaps it follows that anyone questioning any part of your actions must have the most Machiavellian of motives. The fact that some people who raise issues do seem to enjoy fault-finding because of past grievances only makes this assumption all the easier to hold.

With such painful lessons coming my way, this last month has been shocking, disappointing, and — above all else — exhausting. I’ve lost respect for some people I thought I knew, and gained respect for others. At times, I’ve been happy to escape into writing a purely technical article to take a brief holiday from the endless angst.

Startups need to keep their software simple. I want to write “at least during the first iteration” but it is probably more accurate to write “at least during the first hundred iterations”. Complexity in software seems to be like entropy – it takes energy to keep it at bay. So I really appreciate this post from Scout, about how they were ambushed by complexity:

We’ve been deleting a lot of code from Scout. We’re ripping out major infrastructure, and in doing so, pulling the plug on functionality which, just six months ago, we believed would be crucial to our business. Most importantly, we’re simplifying the most complex, error-prone, and poorly-performing parts of the application. At the same time, our revenue and sales pipeline is growing at a faster rate.

How did this happen? How did we get to a place where we can remove code and functionality and see our business will grow because of it?

As they say, “mistakes were made.” You don’t get the satisfaction of throwing out a bunch of cruft and performance-degrading features without having gone through the pain of:

Building those features in the first place.
Fighting the performance problems for a few months before you realize its all untenable and come up with alternatives.

…The ongoing cost of complexity is higher than you think. And it’s easy for complexity to sneak in through the back door in the guise of its nemesis, simplicity. Complexity is sneaky that way. Don’t be fooled.

I’ve worked on projects with 8 or 9 programmers and the managers running the project had no idea which programmers were contributing what. I have often been surprised that some managers are willing to fly in the dark, regarding who is doing what. These tend to be managers with weak technical backgrounds, so perhaps they are unable to analyze the work at the appropriate level.

This analysis of who contributes to Linux seems full of information that managers would want to track about any project:

Beyond that, how one generates statistics from a patch stream is an
interesting question. How does one measure the productivity of
programmers? One possibility is to look at the number of changesets
merged. By that metric, this is the list of the most prolific contributors
to 2.6.20:

Developers with the most changesets
Al Viro	241	4.8%
Andrew Morton	92	1.8%
Jiri Slaby	92	1.8%
Adrian Bunk	87	1.7%
Gerrit Renker	79	1.6%
Josef Sipek	79	1.6%
Avi Kivity	68	1.4%
Tejun Heo	67	1.3%
Patrick McHardy	63	1.3%
Ralf Baechle	61	1.2%
Randy Dunlap	59	1.2%
Alan Cox	58	1.2%
Mariusz Kozlowski	57	1.1%
Andrew Victor	53	1.1%
Paul Mundt	52	1.0%
Stefan Richter	49	1.0%
David S. Miller	48	1.0%
Russell King	44	0.9%
Benjamin Herrenschmidt	44	0.9%
Akinobu Mita	43	0.9%

Looking at patch counts rewards developers who put in large numbers of
small patches. Al Viro’s patches include a vast number of code annotations
(to enable better checking with sparse), include file fixups,
etc. Many of the changes are small – many do not affect the resulting
kernel executable at all – but there are a lot of them. Even so, as the
biggest contributor, Al generated less than 5% of the
total changesets added to the kernel. The top 20 contributors, all
together, generated 28% of the total changesets in 2.6.20.

One could make the argument that a better way to look at the problem is by
the number of lines affected by a patch. In this way, a contributor’s
portion of the whole will not depend on whether it has been split into a
long series of small patches or not. On the other hand, simply renaming a
file can make it look like a developer has touched a large amount of code.
Be that as it may, by looking at lines changed (defined
as the greater of the number of lines added or removed by each individual
changeset), one gets a table like this:

Developers with the most changed lines
Jeff Garzik	20712	6.0%
Patrick McHardy	15024	4.3%
Jiri Slaby	13917	4.0%
Avi Kivity	11726	3.4%
Andrew Victor	9710	2.8%
Amit S. Kale	9537	2.7%
Stephen Hemminger	9120	2.6%
Geoff Levand	8396	2.4%
Michael Chan	8307	2.4%
Chris Zankel	8099	2.3%
Mauro Carvalho Chehab	7390	2.1%
Adrian Bunk	6138	1.8%
Yoshinori Sato	5232	1.5%
Al Viro	4981	1.4%
Benjamin Herrenschmidt	4588	1.3%
Thierry MERLE	4549	1.3%
Dan Williams	4516	1.3%
Jonathan Corbet	3924	1.1%
Gerrit Renker	3857	1.1%
Jiri Kosina	3805	1.1%

I recently oversaw a project that had me working with a distributed team (in the US, Europe and India). The same project also worked exclusively with independent contractors, whose work was hired on an assignment basis. The work was for a new web-startup. I have some concerns about such an arrangement being used for a new, innovative project. I’m going to attempt to explain my concerns in this essay.

First, I’ll talk about using freelance, independent contractors for a web start-up. The seeming advantage of having no permanent employees is that the operation has no fixed costs. An advantage? It depends. If you really want to avoid all fixed costs, simply close your business. Being in business means having expenses, the real issue is whether you are making a profit – does the labor you hire help you make more money than the cost of the labor?

The project I oversaw moved slowly, and some of the slowness can be attributed to the freelance nature of the labor we hired. The programmers would do whatever we asked them to, and they all did high quality work. However, because we made no long term commitment to them, they of course had to keep busy with other work. Sometimes we needed them but they were busy with their other clients. Also, they were not part of the conversations we were having in New York City, so they lacked a broad overview of the project and its goals, so they could not easily take the initiative to move ahead on their own.

If a project has no particular deadline, or if it is simple, then working with independent contractors is a great way to keep fixed costs low. However, if a project is suppose to move fast, or if it needs to achieve certain deadlines, and especially if it is very complex, then I would recommend against the use of this kind of arrangement. For fast moving, complex projects, you need full-time, committed programmers.

Now I’ll offer some thoughts about agility, and the importance of physical proximity.

Agile practices have 2 main hopes:

1.) to allow a fast-forming team to quickly become highly-productive.

2.) to control costs by ensuring discipline in regards to goals and deadlines.

I used the word “hopes” instead of “goals” because the ultimate aim of agile practices is more of an ideal to be aspired to rather than a reality that can be achieved, and by that, all I mean is that we are constantly learning more about what these practices truly mean, and in the future we can reasonably hope to do better than we are doing today. I’m speaking for the entire profession of computer programming when I say this.

I am in the United States, in New York City. I’ve been working on websites since 1996. I’ve worked on many small teams, and over the years I’ve used some agile practices, such as frequent iterations, frequent in-person consultations with the client, and unit tests. When I started on this last project, I first thought it would offer the chance to experiment with a fuller range of agile practices. However, I came to realize that most agile practices depend on physical proximity. The programmers need to be able to meet, preferably every day (”every day” in the sense of casual conversation, not “every day” in the sense of “let’s have a formal meeting”). Consider some of these practices:

1.) pair programming (two people looking at the same screen, checking for errors)

2.) the use of index cards to map objects

3.) stand up meetings (meetings kept short and lively because everyone is standing)

None of these make any sense if the programmers are in different countries.

I’d also add code reviews. One can do a code review at a distance, but I think in person is much more effective. And code reviews don’t make much sense when you are working with independent contractors. Code reviews are meant to develop each programmer’s long-term abilities, and also they are useful for helping the whole team agree on certain programming habits. But one would have to have a long-term relationship with the programmers before doing such a thing would make sense.

Programming is an art, and art needs passion. Many agile practices are aimed at soliciting a greater emotional commitment from the programmers. I might write “For an agile project to go well, the programmers need to emotionally commit to it” but that isn’t quite right. What would be more correct would be “For an agile project to go well, the programmers need to emotionally commit to their own professional ideal.” That is, they need to care about the craft of programming. They need to be constantly pushing themselves to always be better programmers.

I would never argue that agile practices are needed on every project. If a project is routine, then long-distance, non-agile practices might be perfect. For instance, suppose you decide to launch a new online magazine. After researching your options, you decide you will use WordPress as the software that powers your site. I’d suggest you find a good designer, local to you, and work with them to come up with the design. Once you have the design, the rest of the process is mundane. You could certainly hire a team in India to install WordPress and implement your design for you. And you would not need agile practices for such a project. In fact, you do not even need great programmers for such a task. Mediocre programmers can do a reasonable job with tasks that are standard, routine and mundane.

However, I would argue that agile practices are needed at projects that need to be fast moving (this would include most start-ups), or at any project where the aim is to create something altogether new and unique. If your project is venturing out into the great unknown, if you are going where no one has ever gone before, you will need a team of excellent, committed adventurers to go along with you.

Let’s look at the history of agile practices, going back centuries.

In the 1990s, DARPA invested some money in agile research, which lead to such books as that by Goranson: Agile Virtual Enterprises.

Goranson looks back at the whaling industry of the mid-1800s and examines how 2 small towns in Massachusetts were able to capture 90% of the whaling industry, when most of the actual hunting was done in the South Pacific. He found that the patterns of hiring and team building had much in common with what start-ups aspire to nowadays – a tight-knit, fast-forming team of highly competent professionals held together by a commonly understood body of professional ethics and expectations.

Goranson emphasizes the role of geographical concentration (and therefore physical proximity), to the evolution of culturally unique traits, leading to competitive advantages:

Geographical and professional isolation allowed for the evolution of a unique and very effective system which for many years put a brand new, fairly high risk/high payoff, virtual enterprise in the water at the rate of one every two weeks. The return of each whaling expedition triggered the formation of another. It was considered an invitation to bad luck to reuse the same combination of partners, so during the six to nine months it took to recondition the ship for another voyage, the owner of the boat assembled a new group of key players who would join him in setting up the basic physical and social conditions needed for a successful venture.

The primary partners required to launch a voyage consisted of a ship owner, an insurer (of the ship cargo), a provisioning financier to supply the expedition with food and other consumables, a captain, and often a manufacturer who agreed to buy the oil at a set price. This component of the partnership was formed in the first couple of months, the partners being determined partially by availability.

A month or less before the ship was ready to sail, a secondary group of partners, the crew, was formed. They shared a distinct cultural background; almost none of them had, or would ever, serve in the navy or the merchant marine. This professional distinctiveness, coupled with an intense geographical concentration, fostered the development of a unique culture based on the virtual enterprise.

Goranson uses the word “agility” to refer to fast forming supply chains, and also (relatedly) the ability to overcome the administrative costs that might keep a new supply chain from forming (in the early 90s, when DARPA funded Goranson’s work, one problem seemed especially prevalent: the fact that the most innovative technologies in America were being developed at small start-ups, and yet large Fortune 500 companies were having difficulty integrating small start-ups into their supply chains, because of the administrative hassles of developing relationships with a multitude of small companies). He uses the word “virtual” to refer to an organization that is project-specific and not meant to last. For instance, in the modern context, the best examples of agile, virtual enterprises are in the movie industry. Each movie is its own independent business, and the production of each movie entails the fast formation of a team of highly competent professionals who understand they are being hired to work on a specific, time-limited project, which will disband as soon as the project is done.

Whaling ships and modern web start-ups share certain traits::

Every voyage included a team of skilled craftsmen – carpenter, blacksmith, cooper, and a sailmaker (often a boatwright, a rigger and a cook were also in this class) – whose combined expertise allowed the enterprise to respond effectively to a broad range of situations. Each of these professionals, along with the tools, supplies, and sometimes apprentices they brought abroad, formed an essentially self-contained business which was integrated into the enterprise as a whole. In these cases, it was not just the person who signed up for the voyage, but their business. From the shipowner to the cook, everyone was paid with a pre-arranged percentage of the take.

In other words, every ship was its own little agile, fast-forming start-up.

Goranson’s use of the word “agility” is somewhat different than the way the word “agility” is used by its proponents in the software industry, but the meaning of the word overlaps for both groups in the sense of “the ability to get supply from a new source quickly”. Consider the words of the Agility Manifesto:

We have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

With only minor tweaks of the wording, you could use the above as a statement of the professional culture that pervades Hollywood. The emphasis is on doing what is necessary to get something working into production fast. By contrast, the slow moving process of low trust, heavily documented, laborious negotiated, carefully vetted contracts are anathema in this context.

(Stephen M. R. Covey has a written a book called The Speed Of Trust, which does an excellent job explaining how much a project speeds up when the participants can trust each other, and therefore how important it is to work toward facilitating the long-term development of trust in any business relationship.)

Goranson emphasized that one of the legal traditions that the whaling industry and movie industry share is a very strong reliance on verbal contracts, as opposed to written contracts and detailed documentation. Goranson points to the bankruptcy of Kim Bassinger as an outstanding example of the importance given to verbal contracts in the movie industry. (Bassinger met some producers for dinner to discuss the possibility of her starring in their movie Boxing Helena. She promised, over dinner, to star in the movie. When she later backed out of her promise, they were able to sue her for $8 million.) The Agility Manifesto de-emphasizes contracts, but still puts a heavy emphasis on the importance of verbal communication, a point made clear by “Customer collaboration over contract negotiation”.

Goranson emphasizes the role that geographical concentration played in allowing this culture of virtual enterprises to flourish. I believe that geographic concentration still plays the predominate role in allowing the continued evolution of unique agile practices.

I’ve had it suggested to me that the cheapness of overseas programmers (in places like India) can speed a project up, simply because, since they are cheap, you can hire a lot more of them. This is flatly contradicted by Fred Brooks thesis in The Mythical Man-Month. When the System 360 project was running late at IBM, Brooks tried to speed up the project by adding more programmers to the team. But he found that the additional programmers actually slowed the project down. Each new programmer became one more person who needed to be kept up to date with the latest documentation – and every few additional programmers required that a new manager be brought into the project to manage them. The growing complexity of communication across a growing mass of workers helped to slow the project down, rather than speed it up.

In this sense, the cheapness of overseas programmers actually encourages bad habits. Because they are cheap, you may get tricked into thinking that you can hire a lot of them. But the more you hire, the more time you will need to invest trying to keep everyone unified in their understanding of what the code is suppose to do. And after a certain number, you’ll need to start hiring additional managers.

If you are a project manager, and you rely on the cheapness of your programmers to keep a project’s costs under control, then you need to fire yourself. There is only one correct way to keep costs under control, and that is through disciplined focus on goals and deadlines. Agile practices have evolved to make it easier to keep complex projects on track. To the extent that geography matters to agile practices, then geography matters for the goals and deadlines of your project.

Nicole Radziwill approaches this subject from a different angle, but she makes a similar point about the importance of geography:

Space does matter. We know this when we are designing facilities and plant layouts, for example, because one of our common considerations is to minimize traffic between areas and departments. More often than not, we do this to minimize the time spent moving people or equipment around a plant, so that time is not wasted. But the same concept could apply to our supply chains. Why aren’t we minimizing the time that components or goods spend traveling through the supply chain, when it could lead to reductions in energy costs? Furthermore, why aren’t we shortening our supply chains to strengthen local and regional businesses, and train the next generation of skilled workers (who can actually do something useful for the regional economy)?

The logic has been something like this: energy is cheap, therefore transportation is cheap, and transportation is easily available and accessible through third-party providers like FedEx and UPS. But I can’t shake the feeling that “supply chain status quo” is not good for quality in the long-term – because it encourages us to source the products and components that are most affordable, rather than the ones that might help us cultivate a quality consciousness in our local areas.

To offer a partial answer to the question that Nicole poses, I think the reason why products get sourced to where they are most affordable (at the current moment, in the short term) is that people tend to fear losses more than they value potential gains. There is an abundant literature on this subject. In regards to sourcing, buyers go with what is cheapest in the short-term, rather than developing long-term innovations that might lead to much greater gains in the future. Consider:

Prospect theory… argues that people center their expectations around a variable and moveable reference point. The reference point is most easily thought of as the status quo, though it is not always so. In most situations, people are more likely to ensure that they preserve their current situation, than to risk losses by seeking potential improvements. Indeed, people tend to sacrifice a great deal in order to maintain their current situation. As Levy notes, “people tend to be risk-averse in choices among gains, but risk acceptant with respect to losses.” This is often the problem in intractable conflicts; the risks required to resolve the conflict seem much too great compared to the potential losses. This sort of framing is what leads ultimately to hurting stalemates. Loss-oriented framing is, ultimately, this tendency to minimize losses while missing opportunities to improve one’s situation.

The status quo of any culture, in the short-term, can be regarded as an under-developed blandness, relative to the unknown brilliance of some unknown future innovation. It takes a leap of faith to believe in that unknown future brilliance. It is easier, and safer, to go with what is cheapest in the short-term. This is cowardice, but it is extremely common. True business leadership is rare.

In terms of developing some unknown future brilliance, I’m thinking especially of those cultural traits that, in part, give rise to comparative advantage. There is, at this point, a mountain of literature about the advantages that agile teams have when it comes to fast-forming, fast-moving projects. Agility is a cultural innovation that deserves further refinement, and each nation, depending on its legal traditions and its labor laws, is going to develop agility in a unique direction. The future development of these potential innovations is undermined by the extended supply chains that send work to distant parts of the world. I’m arguing that, just as Goranson stressed geographical isolation as playing a role in the evolution of a uniquely competitive culture in Massachusetts, then the dispersal of work from a geographic center must lead to the opposite – the dissolution of uniquely competitive cultural traits.

I can imagine someone reading this essay and raising certain objections. What about Linux? What about Apache? Certain open source projects have been successful with teams that are scattered all over the world. But then, these were not fast moving projects. Teams that never meet in-person can become highly productive, but it takes years. And by the time these projects hit their strides, much of the core team had, in fact, traveled to different countries and met each other in-person.

One other fact argues for the importance of physical proximity. For a team to be highly productive, it needs to have both high levels of communication and high levels of trust. Body language is important to trust. In a normal, in-person conversation, the bulk of the communication is non-verbal. Below I’m going to quote from an article that is aimed at teaching salespeople how to make more sales. On a start-up, the team members are not trying to sell products to each other, but they are often in the position of having to try to sell a given strategy to each other. One team member may feel that their site should be pitched to consumers, another team member may feel strongly that the site needs to be pitched to businesses. The success of the start-up depends on all of the team members being to able to listen to all points of view, take in the maximum amount of information and opinion, and reach the best decision.

It really doesn’t matter how knowledgeable you are about your product line or how many closing techniques you have mastered, unless you earn your prospect’s trust and confidence you’re not going to make the sale.

…Body language is a mixture of movement, posture and tone of voice. Research indicates that in a face-to-face conversation, more than 70 percent of our communication is nonverbal.

Our body language reveals our deepest feelings and hidden thoughts to total strangers. In addition, nonverbal communication has a much greater impact and reliability than the spoken word. Therefore, if your prospect’s words are incongruent with his or her body language gestures, you would be wise to rely on the body language as a more accurate reflection of their true feelings.

On a slightly humorous note, I’d like to offer a movie review. The best movie ever made about an agile, virtual enterprise was Akira Kurosawa’s 1954 movie, The Seven Samurai. Of course we need to translate the movie into business-speak before the comparison becomes clear. I’m only partly kidding when I write this. I really do believe this movie offers a good example of some of the above ideas.

In the opening scene, we meet the client (poor village farmers). We then come to see their problem: bandits who plan to raid the village as soon as the harvest is in. The villagers then go to a big city to hire a contractor who can solve this problem for them. They meet Kambei, a samurai. Though the villagers can pay very little, he takes pity on them and agrees to help them. He therefore steps into the role of project manager. He must then assemble a team, and this part is pure agile enterprise in practice. The city is full of many samurai, and out of this sea of potential recruits, a group of 6 is selected. These are highly skilled workers with a strong emotional commitment to their own professional ideal – that is, they all strive to be the best samurai that they can be. The 7 samurai then set out for the village. They almost instantly form a tight-knit, highly cohesive team, because they share a commonly understood body of professional ethics and expectations. The enterprise is “virtual” in the sense that Goranson uses the word – it is project-specific, and will disband as soon as the project is complete. At the village, the samurai work in close consultation with the client to ensure the goals of the project (the defense of the village). Finally, after a decisive fight, the goals of the project are achieved – the bandits are defeated. The client is happy. The surviving samurai then disband.

In this movie, the samurai are recruited in the same fashion as sailors might have been recruited for a whaling ship, or the same way that a computer programmer might be recruited for a web start-up.

To recap, my current experience teaches me that:

1.) Agile teams should be small (as per Brooks).

2.) They should meet in person several times a week.

3.) Everyone on the team must be highly competent.

4.) Everyone on the team must be emotionally committed to their own professional ideal.

5.) For the duration of the project, everyone on the team should be full-time

Furthermore, on projects that aim to be unique or innovative, or on projects that are complex, or on projects that need to be fast-moving, I will be discouraging clients from these practices:

1.) part-time, independent contractors

2.) distributed teams, with workers geographically dispersed.

One last point, unrelated to the rest. A number of people have said to me that they feel the cheapness of overseas workers (in places like India) is driving down wages in America. I do not believe this is true. Wages in America have declined over the last 40 years, but that is because the American people have themselves become unwilling to invest in their own country. Consider the fact that in 1955 the average middle class family saved 17% of their income, but in 2000 the US savings rate hit 0%. Wages will decline in any nation that does not invest in itself. By contrast, consider Germany, which has a high investment rate. Not only does Germany have higher average wages than America, it has also had more economic growth over the last 30 years. And right now China is saving and re-investing 40% of its income. I do not know a single American family that saves 40% of its income. Most of my friends, even those with college educations and good paying jobs, only save 10% or 15% of their income. It is wrong for Americans to first engage in an orgy of over-consumption and under-investment, and then blame poor nations like India and China for the obvious problems that must result from such reckless behavior.

Nicole Radziwill riffs on one of my earlier posts. I like her elaboration of the problem of disorganization:

Disorganization is expensive because it blocks action. When your house is disorganized, you waste time and energy trying to find stuff. When the processes you use in the workplace are disorganized, time and physical energy can be wasted engaging in non-value-adding activities, and mental and emotional time and energy wasted in unproductive communications. Wasting time and energy can generate short-term real costs (for example, moving parts or products around a factory or supply chain can delay time-to-market while costing more in fuel for transport), long-term real costs (e.g. reinforcing negative behaviors that lead to breakdowns in interpersonal relationships, teamwork, or morale) or opportunity costs.

Coding Horror has a nice set of quotes and links related to catastrophic software project failures. Of interest to anyone who does project management on software projects:

If Las Vegas sounds too tame for you, software might just be the right gamble. Software projects include a glut of risks that would give Vegas oddsmakers nightmares. The odds of a large project finishing on time are close to zero. The odds of a large project being canceled are an even-money bet (Jones 1991).

In 1998, Peat Marwick found that about 35 percent of 600 firms surveyed had at least one runaway software project (Rothfeder 1988). The damage done by runaway software projects makes the Las Vegas prize fights look as tame as having high tea with the queen. Allstate set out in 1982 to automate all of its office operations. They set a 5-year timetable and an $8 million budget. Six years and $15 million later, Allstate set a new deadline and readjusted its sights on a new budget of $100 million. In 1988, Westpac Banking Corporation decided to redefine its information systems. It set out on a 5-year, $85 million project. Three years later, after spending $150 million with little to show for it, Westpac cut its losses, canceled the project, and eliminated 500 development jobs (Glass 1992). Even Vegas prize fights don’t get this bloody.

Whether we are talking about the evolution of finches  in the Galápagos Islands  or the evolution of the software projects that we work on, my sense is that diversity offers its greatest benefit during  a crisis. The worst thing about monoculture is the powerful reward it offers to pathogens. Or, as Wikipedia says:

The dependence on monoculture crops can lead to large scale failures when the single genetic variant or cultivar becomes susceptible to a pathogen or when a change in weather patterns occur.

Extending that as a metaphor for business, groupthink (a monoculture of thought) can lead to catastrophic failure when some foundational assumption of the group is proven wrong. A monoculture of thought offers a powerful reward to pathogenic behavior. Consider the meltdown at Enron, where top executives all agreed on the profitability of reckless energy trades, and they continued to agree with each other almost till the very moment company declared bankruptcy. Likewise, the top executives at AIG were certain that they had distributed risk in a such a way that the downside of that risk would never catch up with them. – people with dissident viewpoints were squeezed out of their jobs. Or consider the 40 year decline of the United States auto industry, an industry that has suffered more than most from groupthink and inaccurate assumptions. The executives of the 1970s and 1980s felt, despite the gathering evidence, that price was more important to Americans than quality, and that quality automatically meant expensive, and so they lost a generation of car buyers.

A corporate culture that values homogeneity is at grave risk of punishing non-conformists. A good manager is always on guard against the kinds of social bullying, however subtle, that can cause people to censor their opinions. This is a basic task of risk management: reduce risk by challenging core assumptions. Make sure divergent view points are heard.

I should add, if you are working at a new start-up, struggling to find its place in the world, you should treat every day as a crisis.

Genetic diversity allows a population multiple avenues to move forward when a radical change in the external environment dooms the existing species, in their current forms. Genetic diversity helps facilitate the transformation of sub-sections of those populations to evolve into new forms. Likewise, when a corporation faces a crisis, having a diverse range of opinions is healthy, and the more those differences of opinions reach down to core assumptions, the healthier. In boom times, such diversity of opinion could potentially be viewed as annoyingly disruptive of the good times, but in a crisis, what’s needed is the maximum of diversity: in viewpoint, in history, in current circumstances, in goals, in future expectations, etc.

In theory, a genius of a manager could possibly assemble a team made up solely of white males, which still had enough diversity of opinion to perform well in a crisis, but as a practical matter, the most reliable way to put together a diverse team is to recruit people from different backgrounds, different genders, different races and, where possible, different countries.

I regard the cultivation of diversity on a project as a fundamental survival technique, so I devote a lot of time to recruiting newcomers to the field of programming. And so, I read with interest Kirrily Robert’s discussion of recruiting women to work on an open source project (what follows is from Robert’s blog post):

———————————————–

I surveyed women on the Dreamwidth and AO3 projects and asked them about their experiences. You can read a fuller report of their responses on my earlier blog post, Dispatches from the revolution.

One of the first things I asked them was whether they had previously been involved in open source projects. They gave answers like:

I’d never contributed to an open source project before, or even considered that I could.

I didn’t feel like I was wanted.

I never got the impression that outsiders were welcome.

I considered getting involved in Debian, but the barriers to entry seemed high.

Those who got a little further along still found it hard to become productive on those projects:

It’s kind of like being handed a box full of random bicycle parts: it doesn’t help when you don’t know how they go together and just want to learn how to ride a bike.

People without a ton of experience get shunted off to side areas like docs and support, and those areas end up as the ladies’ auxiliary.

But on Dreamwidth and AO3…

What I like most is that there isn’t any attitude of “stand aside and leave the code to the grown-ups”. If there’s something that I’m able to contribute, however small, then the contribution is welcome.

And this one, which is my favourite:

Deep down, I had always assumed coding required this kind of special aptitude, something that I just didn’t have and never would. It lost its forbidding mystique when I learned that people I had assumed to be super-coders (surely born with keyboard attached!) had only started training a year ago. People without any prior experience! Women! Like me! Jesus! It’s like a barrier broke down in my mind.

So, what can we learn from this? Well, one thing I’ve learnt is that if anyone says, “Women just aren’t interested in technology” or “Women aren’t interested in open source,” it’s just not true. Women are interested, willing, able, and competent. They’re just not contributing to existing, dare I say “mainstream”, open source projects.

And this is great news! It’s great news for new projects. If you are starting up a new open source project, you have the opportunity to recruit these women.

I’ve had the good fortune to work with some truly great designers. These are people who are understand the client’s needs, who understand the users, and who naturally develop web sites that feel intuitive to the people who use the site. Long ago I learned that, when working with such designers, it was important to empower them with the code they needed, but to otherwise stay out of the way. In particular, such designers need access to the HTML of the site. When I hide the HTML away inside of PHP functions, I’m limiting the ability of these maestros to perform.

This has implications when working with a framework like Symfony.

A lot of computer programmers would add an image to a Symfony template by using one of the image helpers:

<?php echo image_tag(’banner.jpg’) ?>

But what if the designer needs to add a CSS class to this? What if they are working late at night, and they are unable to reach me? Or they are working in the middle of the day, but it is a day I’m spending away from my computer?

The designers I work with can read PHP code fairly well. And they can look up the image_tag helper and figure out its syntax. But this is a big waste of their time. It takes them further into the world of programming than they should need to go.

When creating URLs in a Symfony project, it is important to use helpers. The helpers take care of figuring out what the right URL should be for things like images. The helpers allow us to develop on one machine and deploy to another server when we go live, and even if that other server has a completely different directory structure, we don’t need to change any links, because the helpers take care of all that for us. However, managing the URLs is the only thing that the helpers should do.

So we do not do this:

<?php echo image_tag(’banner.jpg’) ?>

Instead we either do this:

<img src=”<?php echo $sf_request->getRelativeUrlRoot() ?>/images/banner.jpg” />

or we do this:

<?php sfLoader::loadHelpers(array(’Url’)); ?>
<img src=”<?php echo public_path(’banner.jpg’) ?>” />

This way the URL is managed by Symfony’s helpers, but the rest of the IMG tag is still free for the designers to work with. If a designer needs to add a CSS class, they don’t need to call me on the phone and get me to do it, and they don’t have to start researching Symfony helper tags, instead they just add the CSS class like they always have:

<img class=”header” src=”<?php echo public_path(’banner.jpg’) ?>” />

This is an important bit of project management. It allows for a correct separation of concerns – the designers get to focus on design, and the programmers can focus on the programming.

I’m going to talk about a real project from a year ago, but I’m changing the name of the project to “Tango”. Here is part of a message that I posted, several months ago, to the Tango project area on our Basecamp account:

Traffic to the website is down 12%. All of the effort expended to encourage people to use the chat rooms has shifted traffic around on the site, but it hasn’t lead to an increase in traffic. Also, the money spent on Facebook ads has turned out to be a complete waste of money – for the number of new visitors that Tango got, twice as much money was spent than would have been spent using the same money to hire new writers.

To some extent, I was criticizing strategies that had been proposed by other people, but which I’d been willing to go along with to see how they might work out. Part of the reply that I got from one of my co-workers:

I’d appreciate a more positive attitude.

I am increasingly afraid of “positive attitudes”. Or rather, what is often called a positive attitude. A project is in deep trouble when the people working on it fail to point out the weaknesses that they see. How can improvements be made without criticism?

Never mind what the real meaning of the phrase “positive attitude” is. As a practical matter, in a business context, what it usually means is “Let’s ignore the bad news and focus on the good news.” But such an attitude only makes sense if you believe the bad news is unfixable. In other words, the bad news indicates that the whole project, the whole company, is doomed. So what is often referred to as a positive attitude is, in fact, a profoundly pessimistic attitude.

In truth, I have a positive attitude: I believed, and still believe, that the project will be a great success. But only if the project’s weaknesses are addressed. And if you look at what I wrote in my Basecamp post, you’ll see it implies a number of things that should be fixed, and which are probably fixable.

What is the point of this thing called a “positive attitude”? Much of it is political – a desire to avoid facing painful realities. Often this is done to keep together a coalition of people – that is, to keep a group of people supportive of a project. We might be talking about the Iraq war, where the American government has relentlessly painted a prettier picture than what reality justified – and this lying was meant to keep the American people supportive of the war. Or we might be talking about a money losing business, in which case the workers and managers might by lying to the investors, so as to keep the investors supportive of the business. And for those who want to make their living that way, this is the one real use of a “positive attitude”.

This is the only positive attitude that a project really needs: the belief that a project can succeed. Once you have that belief, everything else should be criticism: what isn’t working, what needs to be fixed? Happy talk is the enemy of truth, and lies are an effective way to take a viable project and kill it.

Last year I had to oversee a Flash project. The goal was to develop one of the most sophisticated video players ever. This thing could read XML files from other sites, and become an embeddable store. We were developing it for iHanuman.com, though it was never put into use there. The goal was to allow yoga enthusiasts to use their own blogs to promote the videos of their favorite yoga teachers. We were initially thinking the project would cost $10,000, but the owner of the company kept asking for additional features, so we thought maybe the cost might rise to $20,000. But in the end, the final cost was close to $50,000. Flash is not a serious programming environment, and as soon as you try to do something ambitious with it, you learn a painful lesson about its limits. For this reason, I am very excited about JavaFX. CJT sums up a number of my thoughts on the subject:

The maturity of the underling technology (the Java Runtime Environment) cannot be overlooked. One of the biggest gripes we have with, for example, the Flex framework, is it is bug ridden. Start doing any kind of serious project with it and you cannot get away from this. The mx components have been completely re-written no less than three times now! And they are still buggy. This slows development, there is no getting around it. Ask any developer who has done any serious stuff with the Flex framework and they will be able to testify to this. Hopefully with Flex, this will improve now that the framework is open-source.

Java is very mature and so we shouldn’t see soo many issues in this department.

The JRE is also leaps and bounds ahead of the Flash Player in terms of functionality, along-side the fact there are tons more Java developers (and more experienced) out there than actionscript developers, which as an employer, is also a key point. Finding decent actionscript developers is tough compared to experienced Java developers and this isn’t because of demand, its because actionscript 3 (I pinpoint 3 because 2 was half-baked and the Flash Player AVM1 was pants compared to AVM2 – hence incomparable to Java and the JRE) is a relatively new language and most actionscript “developers” have come from a creative/design background as opposed to programming. We like designers to do design and hard-core coders to code and for the two teams to collaboratively work together to create great results.

I don’t know much about Flash. Last year, for the video project, we brought in a Flash programmer as an independent  contractor. I was assigned to manage her, since I was the senior programmer with the firm I was working with. She started the project using ActionScript2, but when she realized how ambitious the goals of the project were, she switched to ActionScript3. We were asking her to do a lot of parsing of XML files, which of course AS3 makes easier than AS2.

As the project fell further and further behind schedule, I began to wonder if she was any good. So I contacted my old friend Nick Piazza, who was the Flash programmer for Category4.com (where I once worked).  I asked him to review her work and tell me if she was any good. So he reviewed her work, and he was very impressed.  She was, after all, doing cutting edge work in AS3. She was using many of its new features. He admitted to me that he himself, along with most other Flash programmers, were still perfectly content to do their work in AS2. In his opinion, she was doing the most advanced Flash work of anyone in central Virginia.

Great, I thought. So she is excellent. So why is the project falling so far behind schedule? She explained she was up against some intractable bugs. I asked if I could sit down with her and go over  the code. Some programming problems are, after all, logic problems, they are not specific to a language.

So we sat together and went over her code, trying to figure out the solutions to certain bugs. And what I learned was this: Flash has terrible tools for debugging. Adobe has only recently started to take ActionScript seriously as a language for advanced programming, and they are only slowly providing the tools needed for advanced programming. And of course, chief among the needed tools are debugging tools.

I walked away from those meetings with the sense that Flash was uniquely weak in debugging tools, and that this was a major source of budget risk when contemplating doing anything ambitious with it.

JavaFX starts off in a better situation than Flash. JavaFX is going to be a hit product. I’ll admit, for small animations Flash is too entrenched to ever be dislogded, but Flash just lost its bright future. For big, ambitious multimedia projects, JavaFX offers less  budget risk  than Flash.

Well, maybe “broken” is too strong a word. The CSS failed to load. Which happens sometimes. Which is part of why semantic markup is important. How much does the user experience survive the lack of style sheets? This from the front page of Old Navy:

Just curious, but does anyone know some common reasons why the CSS files might fail to load?

Also curious about the marketing. The front page of the site is aimed entirely at women. Do women buy clothes online more than men?

Alex Marshall has up a great essay at Spotlight on the Region. He suggests large construction projects are now taking longer than they used to:

It took four years to complete in 1904 the city’s first subway system, much of it hacked away using pick axes and mules.

It took less than a decade to complete in 1915 the central water line from the Catskill Mountains to New York City, a mammoth 92-mile aqueduct and tunnel system worthy of the Roman Empire.

It took just over a year to complete in 1930 the Empire State Building, the tallest building in the world by far at the time, an intricately detailed structure of tiles and brick.

Today things generally take a lot longer, whether it’s the public or the private sector at work.

The first portion of the 2nd Avenue subway, which will be built using state-of-the–art tunnel-boring machines, will take six years to complete – if all goes well and enough money is found. Running from 96th Street on the Upper East Side to 63rd Street, it will have just three new stations, spaced every ten to 14 blocks.

Compare that to the city’s first subway line, the IRT, which ran from City Hall up to Grand Central, over to Times Square, and then up to 145th street. The IRT had 28 stations as well as local and express lines. And it was all built in just four years.

So just to make this contrast clear, the first phase of the 2nd Avenue subway in terms of track is a tenth the size of the original IRT and has three stations as opposed to 28. Yet it will take six years to complete as opposed to four for the much larger IRT. Why is this? …

Almost every engineer or administrator I ask about this seems to agree that things do take longer today, but explanations vary. The usual suspect is environmental and regulatory review, but I’m not including that in my perspective. I’m just talking about actual construction time.

Earlier tonight I found myself writing the following email (one of several) to a designer I’m working with. I was trying to explain why I favored “design lead development”.

Dear wwwwww,

I think another way to think of your job, for the next 3 weeks, is to figure out what assumptions people are making, and draw those assumptions out with Photoshop mock-ups. What are JJJ, MMM, CCC and I assuming? We’re relying on you to attack those assumptions, as much as possible, and make things visually explicit.

I guess another way to approach this is for you to ask, what do you feel ignorant about? What questions would you like to ask about this project? In an ideal world you would never use words to ask any questions, but instead you’d use a Photoshop mock-up to ask every question that you’re curious about. I’m not sure if that is an attainable ideal, but it’s worth keeping in our heads. Ideally you’d speak to us in Photoshop, not English.

Of course, this is merely my opinion. If you disagree with me, I trust your judgement about how to proceed.

– lawrence krubner

The modern, affluent standard of living depends on society’s infrastructure, both physical and intellectual: airports, highways, bridges, ports, telecommunication networks, databases, power plants, and the software that makes it all run. 150 years ago it was still common for massive bridge projects to meet with catastrophic failure. The engineers of the 1800s slowly mastered the art of working with iron and then steel. The projects grew in scale, and the skills of the project managers needed to keep apace.

Nowadays we rarely hear of catastrophic bridge failures. This is a particular type of infrastructure that has come to be well understood, both in its engineering and in the project management that oversees its construction. However, we, as a society, are still struggling to find the right way to develop large software projects. This is the newest type of infrastructure, and the correct way to engineer it and manage its development is still a source of controversy.

Robert L. Glass has compiled a book of entertaining and instructive stories regarding the massive software project failures of our time: Software Runaways: Monumental Software Disasters. Anyone with an interest in the management of software projects should read it.

The worst of these project failures, the most expensive and the most ambitious, was the attempt made by the Federal Aviation Authority to modernize the computer system it uses to keep track of what planes are in the air. The effort began in 1981 and ended in complete failure in 1994. The government hired IBM to do the actual work, and over the course of 14 years, IBM burned through $3.7 billion dollars. Nothing was accomplished. The project was finally shut down by Congress. Nothing came out of the project, not a single piece of software, nor even a line of code, was ever used for anything.

Robert N. Britcher, who was involved with the project, offers a full write up of the project. I will try to entice you to buy the book with some excerpts:

The Advanced Automation System began, in concept, in 1981 and ended in 1994, “terminated for convenience” by the government. Billions of dollars were spent on it. It is hard to describe. You can’t learn anything from the name. You know it’s about air traffic control because I told you, or because you read about it in the papers. Maybe part of the problem was the name. It sounds like the system to end all systems.

One engineer I know described the AAS this way. You’re living in a modest house and you see the refrgerator going. The ice sometimes melts, and the door isn’t flush, and the repairman comes out, it seems, once a month. And now you notice it’s bulky and doesn’t save energy, and you’ve seen the new ones at Sears. So it’s time. The first thing you do is look into some land a couple of states over and think about a new house. Then you get I.M. Pei and some of the other great architects and hold a design run-off. This takes awhile, so you have to put up with the fridge, which is now making a buzzing noise that keeps you awake at night. You look at several plans and even build a prototype or two. Time goes on and you finally choose a design. There is a big bash before building starts. Then you build. And build. The celebrating continues; each brick thrills. Then you change your mind. You really wanted a Japanese house with redwood floors and a formal garden. So you start to re-engineer what you have. Move a few bricks and some sod. Finally, you have something that looks pretty good. Then, one night, you go to bed and notice the buzzing in the refrigerator is gone. Something’s wrong. The silence keeps you awake. You’ve spent too much money! You don’t really want to move! And now you find out the kids don’t like the new house. In fact, your daughter says “I hate it”. So you cut your losses. Fifteen years and few billion dollars later, the old refridgerator is still running. Somehow.

At $3.7 billion, the Advanced Automation System was one of the largest civilian computer contracts ever; maybe the largest. It was the largest single contract in IBM’s history. From the moment it was awarded, until near the project’s demise, IBM patted itself on the back. There was something for everyone, beginning with a great ball in Union Station, featuring Chubby Checker and “The Twist”.

At its peak the project employed over 2,000 people. About a million dollars a day. If you thought like the IBM project manager, this was a good deal. Many people were working and money was being made. It was going to last forever. No one considered that it wouldn’t. And everyone was getting ahead. (One of the ironies of conceptual work is that it is easy to believe you are farther along than you are.)

The AAS must have been the most supervised project in history: this atop its enormous size and complexity, and the extreme and constantly changing requirements. One programmer described it this way: “Working on the project was like working on a car inside the garage with the motor running. Eventually, even the crickets hopping around the tires suffocate.”

What I saw on the FAA’s Advanced Automation System would have made Sisyphus weep… Whatever commitment and discipline there was… was worn down by a battery of watchfulness that I can only ascribe to fear of failure. In spite of tens of millions of dollars spent on new computers for AAS, the most important piece of equipment on the project was the overhead projector. There were endless meetings attended by dozens of people – as if we were never quite sure about the whole thing. The people in charge simply lacked the confidence and the finesse of the space team: NASA, contractors, and astronaughts.

It has been noted by everyone from the New York Times to the Vice-President of the United States that the main problem on the Advanced Automation System was “changing requirements”. For those involved in large-scale computer systems, that is nothing new. No one can perfectly surmise the shape and feel of a system years in advance. Even replacing some aspect of a system you know by heart is not immune from thinking twice about it. … [But] the requirements churn (it was called) on the Advanced Automation System was not normal. It was the result of our enchantment with the computer-human interface, the CHI. The new controller workstation, fronted by a 20″ by 20″ color display, because it was capable of seemingly endles variety of presentations, mesmerized the population of AAS like the O.J. Simpson trial mesmerized the nation…

The project was handed over to human factor pundits, who then drove the design. Requirements became synonymous with preferences. Thousands of labor-months were spent designing, discussing, and demonstrating the possibilities: colors, fonts, overlays, reversals, serpentine lists, toggling, zooming, opaque windows, the list is huge. It was something to see. (Virtually all of the marketing brochures – produced prematurely and in large numbers – sparkled with some rendition or other of the new controller console.) It just wasn’t usable…

The cost of what turned out to be a 14-year human factors study did not pay off. Shortly before the project was terminated a controller on the CBS evening news said: “It takes me 12 commands to do what I used to do with one.” I believe he spoke for everyone with common sense.

Rummaging through one of the closets at the far end of the hall on the fifth floor one day, looking for some standards document, I found an envelope left by someone who left the company – as many did after so many years advancing against stone, while the wheels of commerce were accelerating on what everyone referred to as “the outside”. It contained “A Brief History Of The Advanced Automation System”. It was printed by hand and left, perhaps inadvertantly, or perhaps with the hope that some anthropologist might some day discover it and make a pronouncement. In every important way, it is the truth:

“A young man, recently hired, devotes years to a specification written to the bit level for programs that will never be coded. Another, to a specification that will be replaced. Programmers marry one another, then divorce and marry someone in another subsystem. Program designs are written to severe formats, then forgotten. The formats endure. A man decides to become a woman and succeeds before system testing starts. As testing approaches, she begins a second career on local television, hosting a show on witchcraft. An architect chases a new technology, then another, then changes his mind and goes into management. A veteran programmer writes the same program a dozen times, then transfers. The price of money increases eight times. Programmers sleep in the halls. Committees convene for years to discuss keystroking. An ambitious training manager builds an encyclopedia of manuals no one will ever use. Decisions are scheduled weeks in advance. Workers sit in the hallways. Notions of computing begin in the epoch of A, edge toward B, then come down hard on A + B. Human factors experts achieve Olympian status. The Berlin Wall collapses. The map of Europe is redrawn. Everything is counted. Quality becomes mixed with quantity. Morale is reduced to a quotient, then counted. Dozens of men and women argue for thousands of hours: What is a requirement? A generation of workers retire. The very mission changes and only a few notice. Programming theories come and go. Managers cling to expectations, like a child to a blanket. Presentations are polished to create an impression, then curbed to cut costs. Then they are studied. The work spikes and spikes again. Offices are changed a dozen times. Management retires and returns. The contractor is sold. Software is blamed. Executives are promoted. The years rip by with no end in sight. A company president gets an idea: make large small. Turn methods over to each programmer. Dress down. Count on the inscrutability of programming. Promote good news. Turn a leaf away from the sun. Maybe start over.”