Closer To The Ideal

I just looked at this list of top 50 web sites from 1998. My friends think I have a good memory. My memory is especially good when I’m traveling – places I’ve been, conversations with friends, people I’ve met. I arrange things by dates. However, I find I have no memory for websites. I have only vague memories of the websites on this top 50 list, despite the fact that I spent a lot of time on them. I spend a huge portion of my life online, but I have few vivid memories of particular websites. I find this worrisome.

Wow. It’s been awhile since I’ve seen a “Best viewed in Netscape 3.0” warning.

Best viewed in Netscape 3.0

I was just looking at the video on DIY Themes that explains their Thesis framework. I was surprised by the part of the video where he describes how to move the nav bar from above the title to below it. Moving the nav bar involves adding 2 lines of code to their custom_functions.php file, as you can see in this screen shot:

Doesn’t it seem bizarre that you need to type these 2 lines of code?

remove_action(’thesis_hook_before_header’, ‘thesis_nav_menu’);
add_action(’thesis_hook_after_header’, ‘thesis_nav_menu’);

I’m curious what theory of user interaction justifies these 2 lines? When you want a particular element to be exposed to control by some end-user, there are 2 ways to go that make sense:

1.) Expose the element in the simplest way, allowing for the most straightforward manipulation.

2.) Control the element from your code, so it can be manipulated from a GUI.

Since we are talking about a block of HTML, the simplest way to expose it would be to leave it as plain HTML in the template. A user could then open the file and cut-n-paste the block of HTML to another part of their template. The user could use a text editor, or perhaps GUI design software such as Dreamweaver. Or you (the programmer creating the code/theme/template) can control the element from your code, and create a GUI that allows the user to drag the element around to wherever they want it. But why would you ever do what DIY Themes does here – control the element from code, and ask your users to edit the code?

In the 1970s, personal computers were usually driven from commands typed into the command line. In 1984, Apple made a big splash when it came out with the Macintosh computer, the first personal computer that had a modern GUI. Since that time its been recognized that GUIs can do a lot to make software easier for end-users to use.

I know a lot of designers who are comfortable with HTML and CSS but not necessarily with PHP code – the risk of some maddening, hard to track down parse error is too high.

I’ve the impression that over the last 2 years frameworks such as Thesis have become popular in the WordPress community. And the kind of code tricks that you see above have become common. This is counter-intuitive. The WordPress community is one that I would have thought would have disliked this kind of design style. WordPress originally became popular with designers because it offered a fairly simple way of editing templates.

I’ve been working with PHP for 10 years. I’m comfortable with the code. But I hate template systems where design elements are controlled from the PHP code. I prefer simple, literal templates full of plain HTML, with only a few PHP commands embedded in the templates. I hate controlling the placement of anything from PHP code. I want designers to be able to open my templates and re-design everything, without ever having to edit any PHP code.

I’m surprised that DIY Themes is taking the approach that it is taking.

Much of the effort in the Thesis framework seems aimed at at wrapping a GUI around some of the technical details of running WordPress. I wonder why they didn’t wrap a GUI around these 2 lines?

Tragic. There is a suite of tests that measures how much each web browser complies with web standards. IE 9, to be released next year, still measures lower than even FireFox 2.0, which was released back in 2006.

Mark Pilgrim has written a fantastic article which looks back at the conversation in 1993 that lead to the introduction of the IMG tag, and draws conclusions about the evolution of HTML. He follows the conversation over the course of several months, as several ideas for new technologies are considered and rejected. Marc Andreessen, Tim Berners-Lee, Tony Johnson and many others participated in the conversation.

After reviewing the full history, Pilgrim concludes:

HTML has always been a conversation between browser makers, authors, standards wonks, and other people who just showed up and liked to talk about angle brackets. Most of the successful versions of HTML have been “retro-specs,” catching up to the world while simultaneously trying to nudge it in the right direction. Anyone who tells you that HTML should be kept “pure” (presumably by ignoring browser makers, or ignoring authors, or both) is simply misinformed. HTML has never been pure, and all attempts to purify it have been spectacular failures, matched only by the attempts to replace it.

And then:

But none of this answers the original question: why do we have an <img> element? Why not an <icon> element? Or an <include> element? Why not a hyperlink with an include attribute, or some combination of rel values? Why an <img> element? Quite simply, because Marc Andreessen shipped one, and shipping code wins.

That’s not to say that all shipping code wins; after all, Andrew and Intermedia and HyTime shipped code too. Code is necessary but not sufficient for success. And I certainly don’t mean to say that shipping code before a standard will produce the best solution. Marc’s <img> element didn’t mandate a common graphics format; it didn’t define how text flowed around it; it didn’t support text alternatives or fallback content for older browsers. And 16, almost 17 years later, we’re still struggling with content sniffing, and it’s still a source of crazy security vulnerabilities. And you can trace that all the way back, 17 years, through the Great Browser Wars, all the way back to February 25, 1993, when Marc Andreessen offhandedly remarked, “MIME, someday, maybe,” and then shipped his code anyway.

I notice Scott Meves linking to this great post which is critical of SEO:

Search Engine Optimization is not a legitimate form of marketing. It should not be undertaken by people with brains or souls. If someone charges you for SEO, you have been conned.

First came the web, and it was a mess. Servers went up everywhere, the net connected them all, pages bloomed like flowers, and no one could find a damn thing.

Then came the search engines. First primitive indexes of dumb keywords, then Google with its rankings of most-linked pages, we were finally able to find the pages we needed, mostly.

The ascendency of Google has meant that, if your goal is to get the most eyeballs possible (as any ad-supported media business’ goal is), then prominent placement in the search engine results became a top priority.

And so, like the goat sacrificers and snake oil salesmen before them, a new breed of con man was born, the Search Engine Optimizer. These scammers claim that they can dance the magic dance that will please the Google Gods and make eyeballs rain down upon you.

Do. Not. Trust. Them.

I went to WordCamp yesterday. Here are some thoughts about some of the speakers that I heard.

Dan Milward offered the best presentation we heard all day, about the WP-e-Commerce plugin. This is an impressive package that integrates smoothly with WordPress. From now on, when I do WordPress e-commerce sites, I’ll always consider this plugin first.

Scott Kingsley Clark was awesome. He spoke about his Pods plugin. This allows a dramatic expansion of the CMS abilities in WordPress. This could be the death knell of Drupal and Mambo and Joomla and Expression Engine. It seemed as if you could build a CMS of unlimited complexity, using the Pods plugin. It offers a GUI interface for creating new database tables, which allows you to add an unlimited number of custom types to a WordPress site.

I’m sorry to say that Mark Jaquith’s talk was weak. We caught his talk about making a living doing WordPress development. He had a handful of good insights, for instance, you should pick a specialty. He said he started off making $20 an hour but once he became a specialist in security, he was able to charge $100 to $150 an hour. But his good advice amounted to maybe 5 minutes of his 45 minutes. The rest was a dull list of slogans that I think most of us already know: keep your day job until you have enough client work coming in, once you go for it then really go for it, and if all of your clients agree with your prices then you may be charging too little. Also, he twice introduced an anecdote by saying “Okay, here is one anecdote…”. Public speakers should work anecdotes into their talk, but I think it is awkward to explicitly announce that you are doing so.

The session about the GPL license was extremely lopsided. There were 4 people on stage. Nominally, there were 2 people who represented the Automattic point of view, and then there were 2 developers of premium themes, and the issue to be discussed was how much those themes needed to conform to the GPL license (or, as Grant Griffins would say, to Automattic’s interpretation of the GPL). But only Griffins (of Headway Themes) disagreed with Automattic, so the session was 3 to 1.

I’m normally sympathetic to advocates of the GPL, however, I think some of the advocates do their cause harm with the stridency with which they advance their cause. Yesterday’s presentation about the GPL was an extreme example of this tendency.

The session had overtones of how the police might try to break a suspect. Of the 2 people from Automattic, one played the role of Good Cop and one played the role of Bad Cop. There was also the Convert – he used to be Evil, but now he is one of the Good, because he has seen the light, and now his interpretation of the GPL license is in sync with Automattic’s. Griffiths was setup as the Bad Guy – he does not need see the need to align his company with Automattic’s view of the GPL.

The Good Cop spoke in reasonable tones about how much happier businesses are, once they comply with Automattic’s view of the GPL – there were plenty of profits to be made even after complying with the community’s norms. The Bad Cop spoke in threatening terms about the selfishness of not complying with the community’s norms – how dare any company try to make money off the WordPress eco-system, without complying with the norms of that eco-system (and in the background was the threat of a lawsuit for companies that did not comply). The Convert said they had once feared the loss of sales that might result from complying with the GPL, but now that they had switched over, they found that their business was still thriving.

Mind you, the way the Good Cop/Bad Cop routine plays out is that the captive comes to think of the Good Cop as a friend, so when the Good Cop says something threatening, it carries more weight. Sure enough, the Good Cop eventually mentioned that his law firm had filed several lawsuits against companies for violations of the GPL. However, he added cheerfully, no suit had ever needed to go to court, because once his firm had filed suit, the companies they targeted immediately saw the wisdom of complying.

If they could have only held Grant for 24 hours without sleep or food, he too would surely have come to see the wisdom of aligning his business with Automattic’s view of the GPL. As it was, he continued to assert that his themes were not necessarily in violation of the GPL. He kept saying, “No one has shown me where we are in violation.” He also said that the GPL should not be treated as a religion. He also made the good point that the fear and ambiguity surrounding the GPL was probably bad for business (surely some businesses avoided GPL technologies out of concern for what the courts might find, regarding enforceability).

Mind you, I think Automattic is correct to strongly defend their interpretation of the GPL. The world has benefited a great deal from the GPL, and so it deserves to have strong debate regarding its proper interpretation. I want to be clear about my criticism here: it is good that Automattic strongly defended its point of view, but the workshop would have been much stronger if they’d had on stage more people who had disagreements with Automattic. In other words, the session would have been better if it had been more balanced. That the session was 3 to 1 had some of us feeling sympathetic to Grant, simply because he was so outnumbered. (And also, I thought he did a good job of defending his point of view.)

Also, there were some people in the audience (including a good friend of mine) who had questions that would have added much to the conversation, but the organizer of the session felt it was more important, at the end, to offer their own summary of the issue, rather than take additional comments from the audience. As I said before, sometimes the advocates of the GPL do their own cause harm through their stridency.

Grant got in the last word. He referenced a recent speech where Matt Mullenweg had apparently called certain premium theme sellers “evil”. Grant had reason to believe that the remark was aimed at him. It sounded like Matt owed Grant an apology. We should all work to avoid a situation where disagreements about the GPL are elevated to the level of Good and Evil.

After the conference, some of us went down the street to get some drinks at Tonic. I met Ramil Teodosio, who has been doing excellent work introducing WordPress as a project management tool. He has worked in some large, conservative corporations that need to innovate the ways they organize teams and resources, and he seems to be doing a good job of bringing agile methods and agile tools into such environments.

All in all, I am glad I went. I learned about 2 really great plugins and met some interesting people. I do think Automattic should consider investing in some public-speaking training for its employees. I think someone like Jaquith would benefit a lot if they had some coaching about how to deliver a talk. Other than that, a conference like this is always an interesting chance to get some sense of a community.

Some articles I want to remember for later:

How to handle the CSS differences between IE6, IE7 and IE8.

Mockflow is a new online mockup/wireframing tool. Worth a look.

Massively collaborative mathematics.

I am curious about Mobicules weblog. Some of their posts are self-serving, but they seem to be trying to get some real information out, which is more than what I’ve seen so far from most of the sites of Indian firms that I’ve looked at. How should I read this post on which platform to use for a massive social site that can scale to tens of millions of users?

ELGG: Due to a very flat and normalized database structure, the only way to scale horizontally with ELGG is to duplicate the database using MySQL replication on multiple servers. The negative with this approach is that it ends up duplicating the complete database, with the result that each machine running a copy of the replicated database will have to be very powerful server, rendering the solution expensive. (MySQL clusters is a standard way of doing it)

Drupal: Similar problems as ELGG with Drupal, but database is not as flat and normalized as ELGG. Scaling up in a similar manner as described for ELGG wity MySQL clusters would probably be cheaper with Drupal.

Symfony: Allows us to use a custom database structure. We can design the database and replicate it as we like. Symfony also uses its own query/object caching mechanism, which is efficient. As an example, the Yahoo bookmarks site supports 20 million users on Symfony.

…Our recommendation in a case where you are looking to scale up to 10s of millions of users would be write everything from scratch (and not use a ‘platform’ like ELGG/Drupal), that allows you to customize and tweak anything and everything. Symfony seems to have a good reputation for enabling creation of very large websites, and so, we would recommend using Symfony as a framework.

I actually agree with their conclusions.

PPK has an unusually harsh post about the way drag-n-drop is being implemented in HTML5.

dragthat and dragsomethingelse

In theory dragenter and dragleave could be great events, since they fire when you enter or leave an HTML element in the middle of a drag operation. If that element is a valid drop target you could change its styles ondragenter and ondragleave to indicate this fact to the user.

With the spec being based on the Microsoft API, I expected dragenter and dragleave to emulate mouseenter and mouseleave. But they didn’t. Not even in IE. They’re based on mouseover and mouseout and suck every bit as badly. And their names are wrong.

Mouseover and mouseout are terrible because they bubble up all the time and make it very hard to distinguish important events from unimportant ones. If I mouse over (or drag enter) a child element of the one that the event is set on, it also fires. And that’s exactly what we don’t want. The events will fire incessantly the whole time the mouse is above the element, and we have to work hard in order to distinguish useful events from useless ones.

Mouseenter and mouseleave, on the other hand, fire only when you enter and leave the element they’re defined on, and they don’t bubble. That makes them much easier to use.

They are Microsoft extensions, by the way, and excellent ones at that. They work only in IE.

Note that originally mouseenter and mouseleave had exactly the same IE only compatibility pattern as drag and drop. Mouseenter and mouseleave are a good idea, drag and drop … isn’t. Guess what the other browsers chose to implement?

No wonder web development is such a fucking pain with fucking morons in charge of the browsers.

Wait, that could be construed as an insult to morons.

Ah, what the fuck.

dragbullshit

Dragover, now, has nothing to do with mouseover. It’s exactly the same as the drag event, except that you can set it on any element instead of just the document. Or something. Whatever.

Why do we need the dragover event if we already have the drag event?

[ ... fume ... ]

To cancel its default action!

If we didn’t have to cancel its default action the dragbullshit event would have no fucking point!

And we can’t have pointless events in our nice specification, now, can we?

So we give it a default action. A very complicated default action.

That has to be canceled. Absolutely, positively has to be canceled.

[ ... crickets ... ]

Anybody LISTENING to what I say?

[ ... drag ... ]

I deserve a fucking MEDAL for this. Above and beyond the call of fucking DUTY.

This company developed their platform in the cloud, but eventually gave it up and moved to a dedicated server at RackSpace.

There are also technical complexities in that you have to provision EBS storage before you use it, and if you need to resize then you have to take a snapshot and rebuild from that.

We would have had to build our own infrastructure management system. This would have to handle the starting and failover of EC2 instances, backups and provisioning additional storage. In order to have no downtime when storage was being re-provisioned, we would need a second instance to replicate the database on. Aside from the extra server costs, all of this would have taken development time away from our efforts of improving the product itself.

We had the choice of working on infrastructure that makes no difference to the customer experience (but which would have been technically interesting and fun to develop) verses tangible progress with our product:

With six years of experience running my own software company I can tell you that nothing we have ever done at Fog Creek has increased our revenue more than releasing a new version with more features. Nothing. The flow to our bottom line from new versions with new features is absolutely undeniable.

- Joel Spolsky

And that’s the hidden cost – development time – something that is particularly important for a startup. As Jeff Atwood says, “Hardware is Cheap, Programmers are Expensive“.

When I wrote “How much do websites cost” I made this comparison:

Imagine you hire a carpenter to build a deck on the back of your house. You tell the carpenter that you want the deck to be 10 by 20 feet, and to be built of oak. The carpenter goes out and buys the needed wood. Now you change your mind, and you tell the carpenter that you want the deck to be 8 feet by 18 feed, and to be built of mahogany. The carpenter has to charge you for all the oak that they bought. Changing your mind is expensive!

When it comes to home renovations, people have an easy time understanding why changes drive up costs. However, for some reason, when it comes to the web, people lack this understanding. It is true that the web is a fluid medium, but still, once work commences, the time invested needs to be paid for. If you hire a web design firm and tell them “I want a simple site that lists the services my company offers” then they will start to build one kind of website. If you change your mind and say “I really need a site where everyone in my company can document their work, so I can track what they are getting done” that is a completely different site and needs completely different code. The work on the first site would be thrown out (but you would need to pay for it) and work would begin on your second idea.

It is hard to get this across to clients, so I love this: If architects had to work like software developers.

Dear Mr. Architect:

Please design and build me a house. I am not quite sure of what I need, so you should use your discretion. My house should have somewhere between two and forty-five bedrooms. Just make sure the plans are such that the bedrooms can be easily added or deleted. When you bring the blueprints to me, I will make the final decision of what I want. Also, bring me the cost breakdown for each configuration so that I can arbitrarily pick one.

Keep in mind that the house I ultimately choose must cost less than the one I am currently living in. Make sure, however, that you correct all the deficiencies that exist in my current house (the floor of my kitchen vibrates when I walk across it, and the walls don’t have nearly enough insulation in them).

As you design, also keep in mind that I want to keep yearly maintenance costs as low as possible. This should mean the incorporation of extra-cost features like aluminum, vinyl, or composite siding. (If you choose not to specify aluminum, be prepared to explain your decision in detail.)

[Update:]

One reason I prefer design lead development is that it reveals assumptions and makes things visually explicit. No one gets everything right the first time, therefore there needs to be a way to iterate through ideas, without great expense. The client and designer and developer need to avoid the illusion of agreement that comes from working with words. Thus, it is better to iterate with visuals. The mockups should be the source of agreement. You want to iterate through a lot of ideas fast, so you can fail fast, so that you can get to success fast. Iterative design processes help lower risk.

Changing your mind is expensive. Also inevitable. The goal has to be to keep costs as low as possible while experimenting with different ideas.

Brad Delong asks “Why does Berkeley think it should be in the business of building its own content management systems anyway?“.

His complaint:

May I say that a content management system that–if you have been off dealing with another crisis in the middle of a task–decides when you come back and try to save your work that you are no longer logged in and dumps you to a login page after which it dumps you not on the page you were working on but on the root page, LOSING YOUR WORK!!!1!!

Such a content management system is HELLSPAWN!! Is WROSE THAN HILTER!1!!!1!…

He is complaining about bspace, which is based on the open-source
Sakai, a content management system written in Java, and focused on the needs of universities.

I think Delong’s post is a good reminder of how infuriating it can be for users when software fails to behave in the ways users expect. I also suspect this is a good example of an issue that users will regard as a bug, but the programmers will see it simply as a potential future feature that they may or may not add (”Should we catch POST info if a user is not logged in?”).

I should add, WordPress has exactly the same problem. Last week I started writing what I thought would be a short post for this blog, but I got carried away by my theme and wrote a long post. Then I went to get some dinner. I left the browser open, with the post unpublished. I came back after dinner and made some more edits, then hit the publish button – and just like that, my work vanished, because while I was out getting dinner, WordPress logged me out (for some reason I’d assumed that the auto-save feature was also refreshing my session info).

One of the nice things about building my own CMS was that I was free to fix the bugs that bothered me most, and this was a big one for me. I added a feature to my CMS that caught any POST info and showed it on screen, even if the person was logged out. This allowed recovery of the post. I worked on my CMS from 2002-2007 and then abandoned it because I could not keep up with projects like WordPress. Nowadays I force myself to use other people’s open source software, because it is economically rational to do so, but I hate some of the choices they make, and some of the features that they fail to implement.

In the comments, Jacob Davies posts this comment, which I thought was very funny and very on point:

Conversation that has happened more times in my career than I care to mention:

Someone else: “How long of a title shall we allow? 32 characters? 64?”

Me: “FOR THE LOVE OF GOD WHY DO WE NEED TO SET A MAXIMUM LENGTH? IS THIS 1952???”

Someone else: “But what if they put in a really long title and fill up the database?”

Me: “THE VERY NEXT FIELD – THE ‘CONTENTS’ FIELD – IS A FREE-TEXT FIELD WITHOUT A LENGTH CONSTRAINT SO IF THEY WANTED TO FILL THE DATABASE THEY COULD DO IT THERE ANYWAY.”

Someone else: “Won’t it waste space if we allow a variable-length string in the title?”

Me: “OH MY GOD YES A TERRIFYING LOSS OF ABOUT 3 BYTES ON A RECORD THAT IS A MINIMUM OF 1024 BYTES LONG AND OFTEN OVER A MEGABYTE, YOU ARE SO RIGHT.”

Someone else: “Yes but every other system has a length constraint for titles.”

Me: “YES AND I SUPPOSE IF EVERYONE ELSE WAS JUMPING OFF A BRIDGE YOU’D DO IT TOO.”

etc

Computer programmers are subject to some kind of strange mental degeneration in which they rate the potential waste of 0.00001% of the capacity of a modern hard disk as more important than the ability to enter titles longer than 32 characters in length.

I recently oversaw a project that had me working with a distributed team (in the US, Europe and India). The same project also worked exclusively with independent contractors, whose work was hired on an assignment basis. The work was for a new web-startup. I have some concerns about such an arrangement being used for a new, innovative project. I’m going to attempt to explain my concerns in this essay.

First, I’ll talk about using freelance, independent contractors for a web start-up. The seeming advantage of having no permanent employees is that the operation has no fixed costs. An advantage? It depends. If you really want to avoid all fixed costs, simply close your business. Being in business means having expenses, the real issue is whether you are making a profit – does the labor you hire help you make more money than the cost of the labor?

The project I oversaw moved slowly, and some of the slowness can be attributed to the freelance nature of the labor we hired. The programmers would do whatever we asked them to, and they all did high quality work. However, because we made no long term commitment to them, they of course had to keep busy with other work. Sometimes we needed them but they were busy with their other clients. Also, they were not part of the conversations we were having in New York City, so they lacked a broad overview of the project and its goals, so they could not easily take the initiative to move ahead on their own.

If a project has no particular deadline, or if it is simple, then working with independent contractors is a great way to keep fixed costs low. However, if a project is suppose to move fast, or if it needs to achieve certain deadlines, and especially if it is very complex, then I would recommend against the use of this kind of arrangement. For fast moving, complex projects, you need full-time, committed programmers.

Now I’ll offer some thoughts about agility, and the importance of physical proximity.

Agile practices have 2 main hopes:

1.) to allow a fast-forming team to quickly become highly-productive.

2.) to control costs by ensuring discipline in regards to goals and deadlines.

I used the word “hopes” instead of “goals” because the ultimate aim of agile practices is more of an ideal to be aspired to rather than a reality that can be achieved, and by that, all I mean is that we are constantly learning more about what these practices truly mean, and in the future we can reasonably hope to do better than we are doing today. I’m speaking for the entire profession of computer programming when I say this.

I am in the United States, in New York City. I’ve been working on websites since 1996. I’ve worked on many small teams, and over the years I’ve used some agile practices, such as frequent iterations, frequent in-person consultations with the client, and unit tests. When I started on this last project, I first thought it would offer the chance to experiment with a fuller range of agile practices. However, I came to realize that most agile practices depend on physical proximity. The programmers need to be able to meet, preferably every day (”every day” in the sense of casual conversation, not “every day” in the sense of “let’s have a formal meeting”). Consider some of these practices:

1.) pair programming (two people looking at the same screen, checking for errors)

2.) the use of index cards to map objects

3.) stand up meetings (meetings kept short and lively because everyone is standing)

None of these make any sense if the programmers are in different countries.

I’d also add code reviews. One can do a code review at a distance, but I think in person is much more effective. And code reviews don’t make much sense when you are working with independent contractors. Code reviews are meant to develop each programmer’s long-term abilities, and also they are useful for helping the whole team agree on certain programming habits. But one would have to have a long-term relationship with the programmers before doing such a thing would make sense.

Programming is an art, and art needs passion. Many agile practices are aimed at soliciting a greater emotional commitment from the programmers. I might write “For an agile project to go well, the programmers need to emotionally commit to it” but that isn’t quite right. What would be more correct would be “For an agile project to go well, the programmers need to emotionally commit to their own professional ideal.” That is, they need to care about the craft of programming. They need to be constantly pushing themselves to always be better programmers.

I would never argue that agile practices are needed on every project. If a project is routine, then long-distance, non-agile practices might be perfect. For instance, suppose you decide to launch a new online magazine. After researching your options, you decide you will use WordPress as the software that powers your site. I’d suggest you find a good designer, local to you, and work with them to come up with the design. Once you have the design, the rest of the process is mundane. You could certainly hire a team in India to install WordPress and implement your design for you. And you would not need agile practices for such a project. In fact, you do not even need great programmers for such a task. Mediocre programmers can do a reasonable job with tasks that are standard, routine and mundane.

However, I would argue that agile practices are needed at projects that need to be fast moving (this would include most start-ups), or at any project where the aim is to create something altogether new and unique. If your project is venturing out into the great unknown, if you are going where no one has ever gone before, you will need a team of excellent, committed adventurers to go along with you.

Let’s look at the history of agile practices, going back centuries.

In the 1990s, DARPA invested some money in agile research, which lead to such books as that by Goranson: Agile Virtual Enterprises.

Goranson looks back at the whaling industry of the mid-1800s and examines how 2 small towns in Massachusetts were able to capture 90% of the whaling industry, when most of the actual hunting was done in the South Pacific. He found that the patterns of hiring and team building had much in common with what start-ups aspire to nowadays – a tight-knit, fast-forming team of highly competent professionals held together by a commonly understood body of professional ethics and expectations.

Goranson emphasizes the role of geographical concentration (and therefore physical proximity), to the evolution of culturally unique traits, leading to competitive advantages:

Geographical and professional isolation allowed for the evolution of a unique and very effective system which for many years put a brand new, fairly high risk/high payoff, virtual enterprise in the water at the rate of one every two weeks. The return of each whaling expedition triggered the formation of another. It was considered an invitation to bad luck to reuse the same combination of partners, so during the six to nine months it took to recondition the ship for another voyage, the owner of the boat assembled a new group of key players who would join him in setting up the basic physical and social conditions needed for a successful venture.

The primary partners required to launch a voyage consisted of a ship owner, an insurer (of the ship cargo), a provisioning financier to supply the expedition with food and other consumables, a captain, and often a manufacturer who agreed to buy the oil at a set price. This component of the partnership was formed in the first couple of months, the partners being determined partially by availability.

A month or less before the ship was ready to sail, a secondary group of partners, the crew, was formed. They shared a distinct cultural background; almost none of them had, or would ever, serve in the navy or the merchant marine. This professional distinctiveness, coupled with an intense geographical concentration, fostered the development of a unique culture based on the virtual enterprise.

Goranson uses the word “agility” to refer to fast forming supply chains, and also (relatedly) the ability to overcome the administrative costs that might keep a new supply chain from forming (in the early 90s, when DARPA funded Goranson’s work, one problem seemed especially prevalent: the fact that the most innovative technologies in America were being developed at small start-ups, and yet large Fortune 500 companies were having difficulty integrating small start-ups into their supply chains, because of the administrative hassles of developing relationships with a multitude of small companies). He uses the word “virtual” to refer to an organization that is project-specific and not meant to last. For instance, in the modern context, the best examples of agile, virtual enterprises are in the movie industry. Each movie is its own independent business, and the production of each movie entails the fast formation of a team of highly competent professionals who understand they are being hired to work on a specific, time-limited project, which will disband as soon as the project is done.

Whaling ships and modern web start-ups share certain traits::

Every voyage included a team of skilled craftsmen – carpenter, blacksmith, cooper, and a sailmaker (often a boatwright, a rigger and a cook were also in this class) – whose combined expertise allowed the enterprise to respond effectively to a broad range of situations. Each of these professionals, along with the tools, supplies, and sometimes apprentices they brought abroad, formed an essentially self-contained business which was integrated into the enterprise as a whole. In these cases, it was not just the person who signed up for the voyage, but their business. From the shipowner to the cook, everyone was paid with a pre-arranged percentage of the take.

In other words, every ship was its own little agile, fast-forming start-up.

Goranson’s use of the word “agility” is somewhat different than the way the word “agility” is used by its proponents in the software industry, but the meaning of the word overlaps for both groups in the sense of “the ability to get supply from a new source quickly”. Consider the words of the Agility Manifesto:

We have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

With only minor tweaks of the wording, you could use the above as a statement of the professional culture that pervades Hollywood. The emphasis is on doing what is necessary to get something working into production fast. By contrast, the slow moving process of low trust, heavily documented, laborious negotiated, carefully vetted contracts are anathema in this context.

(Stephen M. R. Covey has a written a book called The Speed Of Trust, which does an excellent job explaining how much a project speeds up when the participants can trust each other, and therefore how important it is to work toward facilitating the long-term development of trust in any business relationship.)

Goranson emphasized that one of the legal traditions that the whaling industry and movie industry share is a very strong reliance on verbal contracts, as opposed to written contracts and detailed documentation. Goranson points to the bankruptcy of Kim Bassinger as an outstanding example of the importance given to verbal contracts in the movie industry. (Bassinger met some producers for dinner to discuss the possibility of her starring in their movie Boxing Helena. She promised, over dinner, to star in the movie. When she later backed out of her promise, they were able to sue her for $8 million.) The Agility Manifesto de-emphasizes contracts, but still puts a heavy emphasis on the importance of verbal communication, a point made clear by “Customer collaboration over contract negotiation”.

Goranson emphasizes the role that geographical concentration played in allowing this culture of virtual enterprises to flourish. I believe that geographic concentration still plays the predominate role in allowing the continued evolution of unique agile practices.

I’ve had it suggested to me that the cheapness of overseas programmers (in places like India) can speed a project up, simply because, since they are cheap, you can hire a lot more of them. This is flatly contradicted by Fred Brooks thesis in The Mythical Man-Month. When the System 360 project was running late at IBM, Brooks tried to speed up the project by adding more programmers to the team. But he found that the additional programmers actually slowed the project down. Each new programmer became one more person who needed to be kept up to date with the latest documentation – and every few additional programmers required that a new manager be brought into the project to manage them. The growing complexity of communication across a growing mass of workers helped to slow the project down, rather than speed it up.

In this sense, the cheapness of overseas programmers actually encourages bad habits. Because they are cheap, you may get tricked into thinking that you can hire a lot of them. But the more you hire, the more time you will need to invest trying to keep everyone unified in their understanding of what the code is suppose to do. And after a certain number, you’ll need to start hiring additional managers.

If you are a project manager, and you rely on the cheapness of your programmers to keep a project’s costs under control, then you need to fire yourself. There is only one correct way to keep costs under control, and that is through disciplined focus on goals and deadlines. Agile practices have evolved to make it easier to keep complex projects on track. To the extent that geography matters to agile practices, then geography matters for the goals and deadlines of your project.

Nicole Radziwill approaches this subject from a different angle, but she makes a similar point about the importance of geography:

Space does matter. We know this when we are designing facilities and plant layouts, for example, because one of our common considerations is to minimize traffic between areas and departments. More often than not, we do this to minimize the time spent moving people or equipment around a plant, so that time is not wasted. But the same concept could apply to our supply chains. Why aren’t we minimizing the time that components or goods spend traveling through the supply chain, when it could lead to reductions in energy costs? Furthermore, why aren’t we shortening our supply chains to strengthen local and regional businesses, and train the next generation of skilled workers (who can actually do something useful for the regional economy)?

The logic has been something like this: energy is cheap, therefore transportation is cheap, and transportation is easily available and accessible through third-party providers like FedEx and UPS. But I can’t shake the feeling that “supply chain status quo” is not good for quality in the long-term – because it encourages us to source the products and components that are most affordable, rather than the ones that might help us cultivate a quality consciousness in our local areas.

To offer a partial answer to the question that Nicole poses, I think the reason why products get sourced to where they are most affordable (at the current moment, in the short term) is that people tend to fear losses more than they value potential gains. There is an abundant literature on this subject. In regards to sourcing, buyers go with what is cheapest in the short-term, rather than developing long-term innovations that might lead to much greater gains in the future. Consider:

Prospect theory… argues that people center their expectations around a variable and moveable reference point. The reference point is most easily thought of as the status quo, though it is not always so. In most situations, people are more likely to ensure that they preserve their current situation, than to risk losses by seeking potential improvements. Indeed, people tend to sacrifice a great deal in order to maintain their current situation. As Levy notes, “people tend to be risk-averse in choices among gains, but risk acceptant with respect to losses.” This is often the problem in intractable conflicts; the risks required to resolve the conflict seem much too great compared to the potential losses. This sort of framing is what leads ultimately to hurting stalemates. Loss-oriented framing is, ultimately, this tendency to minimize losses while missing opportunities to improve one’s situation.

The status quo of any culture, in the short-term, can be regarded as an under-developed blandness, relative to the unknown brilliance of some unknown future innovation. It takes a leap of faith to believe in that unknown future brilliance. It is easier, and safer, to go with what is cheapest in the short-term. This is cowardice, but it is extremely common. True business leadership is rare.

In terms of developing some unknown future brilliance, I’m thinking especially of those cultural traits that, in part, give rise to comparative advantage. There is, at this point, a mountain of literature about the advantages that agile teams have when it comes to fast-forming, fast-moving projects. Agility is a cultural innovation that deserves further refinement, and each nation, depending on its legal traditions and its labor laws, is going to develop agility in a unique direction. The future development of these potential innovations is undermined by the extended supply chains that send work to distant parts of the world. I’m arguing that, just as Goranson stressed geographical isolation as playing a role in the evolution of a uniquely competitive culture in Massachusetts, then the dispersal of work from a geographic center must lead to the opposite – the dissolution of uniquely competitive cultural traits.

I can imagine someone reading this essay and raising certain objections. What about Linux? What about Apache? Certain open source projects have been successful with teams that are scattered all over the world. But then, these were not fast moving projects. Teams that never meet in-person can become highly productive, but it takes years. And by the time these projects hit their strides, much of the core team had, in fact, traveled to different countries and met each other in-person.

One other fact argues for the importance of physical proximity. For a team to be highly productive, it needs to have both high levels of communication and high levels of trust. Body language is important to trust. In a normal, in-person conversation, the bulk of the communication is non-verbal. Below I’m going to quote from an article that is aimed at teaching salespeople how to make more sales. On a start-up, the team members are not trying to sell products to each other, but they are often in the position of having to try to sell a given strategy to each other. One team member may feel that their site should be pitched to consumers, another team member may feel strongly that the site needs to be pitched to businesses. The success of the start-up depends on all of the team members being to able to listen to all points of view, take in the maximum amount of information and opinion, and reach the best decision.

It really doesn’t matter how knowledgeable you are about your product line or how many closing techniques you have mastered, unless you earn your prospect’s trust and confidence you’re not going to make the sale.

…Body language is a mixture of movement, posture and tone of voice. Research indicates that in a face-to-face conversation, more than 70 percent of our communication is nonverbal.

Our body language reveals our deepest feelings and hidden thoughts to total strangers. In addition, nonverbal communication has a much greater impact and reliability than the spoken word. Therefore, if your prospect’s words are incongruent with his or her body language gestures, you would be wise to rely on the body language as a more accurate reflection of their true feelings.

On a slightly humorous note, I’d like to offer a movie review. The best movie ever made about an agile, virtual enterprise was Akira Kurosawa’s 1954 movie, The Seven Samurai. Of course we need to translate the movie into business-speak before the comparison becomes clear. I’m only partly kidding when I write this. I really do believe this movie offers a good example of some of the above ideas.

In the opening scene, we meet the client (poor village farmers). We then come to see their problem: bandits who plan to raid the village as soon as the harvest is in. The villagers then go to a big city to hire a contractor who can solve this problem for them. They meet Kambei, a samurai. Though the villagers can pay very little, he takes pity on them and agrees to help them. He therefore steps into the role of project manager. He must then assemble a team, and this part is pure agile enterprise in practice. The city is full of many samurai, and out of this sea of potential recruits, a group of 6 is selected. These are highly skilled workers with a strong emotional commitment to their own professional ideal – that is, they all strive to be the best samurai that they can be. The 7 samurai then set out for the village. They almost instantly form a tight-knit, highly cohesive team, because they share a commonly understood body of professional ethics and expectations. The enterprise is “virtual” in the sense that Goranson uses the word – it is project-specific, and will disband as soon as the project is complete. At the village, the samurai work in close consultation with the client to ensure the goals of the project (the defense of the village). Finally, after a decisive fight, the goals of the project are achieved – the bandits are defeated. The client is happy. The surviving samurai then disband.

In this movie, the samurai are recruited in the same fashion as sailors might have been recruited for a whaling ship, or the same way that a computer programmer might be recruited for a web start-up.

To recap, my current experience teaches me that:

1.) Agile teams should be small (as per Brooks).

2.) They should meet in person several times a week.

3.) Everyone on the team must be highly competent.

4.) Everyone on the team must be emotionally committed to their own professional ideal.

5.) For the duration of the project, everyone on the team should be full-time

Furthermore, on projects that aim to be unique or innovative, or on projects that are complex, or on projects that need to be fast-moving, I will be discouraging clients from these practices:

1.) part-time, independent contractors

2.) distributed teams, with workers geographically dispersed.

One last point, unrelated to the rest. A number of people have said to me that they feel the cheapness of overseas workers (in places like India) is driving down wages in America. I do not believe this is true. Wages in America have declined over the last 40 years, but that is because the American people have themselves become unwilling to invest in their own country. Consider the fact that in 1955 the average middle class family saved 17% of their income, but in 2000 the US savings rate hit 0%. Wages will decline in any nation that does not invest in itself. By contrast, consider Germany, which has a high investment rate. Not only does Germany have higher average wages than America, it has also had more economic growth over the last 30 years. And right now China is saving and re-investing 40% of its income. I do not know a single American family that saves 40% of its income. Most of my friends, even those with college educations and good paying jobs, only save 10% or 15% of their income. It is wrong for Americans to first engage in an orgy of over-consumption and under-investment, and then blame poor nations like India and China for the obvious problems that must result from such reckless behavior.

Dan Bezdek posted on LinkedIn, suggesting that it was impossible to launch an online business nowadays due to the hyper-competition on the web. He wrote:

I still think there is a huge difference between an online business and a traditional business. We have this grocery store in the neighborhood which is right next to Safeway. Now, you’d imagine there is no hope for this store; however, not only it is surviving, but in fact has so many customers.

He also wrote:

The main problem is that the culture of internet is free service, and this problem will remain with Internet forever. You might spend a couple of years in development, and provide a great service; but you can’t charge users even a $1 except in very few niche sections.

I wrote a response, but LinkedIn limited me to 4,000 characters, so I was not able to post my whole response. I post it here, instead.

- – - – - – - – - – - – - – -

Dan Bezdek, I’m afraid I’m unable to understand your reasoning. I’m going to try to paraphrase what I think you are saying.

You write “It is so easy for anyone in the world to compete with you even in places that $100 is a week pay.”

I think you are trying to say that because computer programmers can be hired cheaply, it is more difficult for you to build a business? But the opposite is true: cheap computer programmers make it easier to launch a new business. I can not think of a single case in economic history where the falling price of a supply factor made it more difficult to launch a new business. A few examples:

1.) During the late 1600s, in Europe, the dramatic fall in the price of paper allowed for the creation of the first modern newspapers. The Spectator launched on March 4, 1712.

2.) During the mid 1700s, in Europe, the dramatic fall in the price of coffee beans allowed for the first coffee houses to open. Paris had hundreds of small coffee stands at the outbreak of the Revolution.

3.) During the late the 1800s, in America, the dramatic fall in the price of communications (the telegraph and the telephone) allowed for the organization of businesses over a distance, and at a scale, never before seen.

4.) During the late 1900s, all over the world, the dramatic fall in the price of computing power brought several new technologies into the mass market, including cell phones, personal computers, and personal (non-business) software.

As a general rule, the cheaper supplies get, the easier it is to start a business.

You also wrote this:

“In any case, the point of all this is to say that developing even a barely successful online business is probably 10 times more difficult than setting up a grocery shop.”

You didn’t mention which country you are in. If you are in the United States, then your statement is incorrect. The retail sector in the US is overbuilt and is expected to consolidate over the next 10 years. Many grocery chains are expected to go bankrupt. Meanwhile, the Commerce Department projects that the software industry will continue to grow, and much of that growth will be happening on the web.

If you do not live in the US, then I’d have to know what country you are in, to know if there is any truth to what you are saying. Some countries, such as France and Italy, offer strong legal protections to small firms. In those countries, small groceries have some hope of surviving, but that is because of government protection, not economic fundamentals.

I think the concept that you are trying to get at is what economists would call “barriers to entry”. That is, what barriers keep competitors from entering your market and competing with you? And I think what you are thinking is that a local grocery store, because it is grounded in a specific geographical space, has some immunity from competition, whereas a web site has to compete with every other web site on the web.

Again, if you are speaking about the US, you are plainly wrong. The history of post-war economic development in the US is the history of retail consolidation. Mom-n-Pop stores have been relentlessly replaced by big chains such as Wal-Mart. Massive numbers of bankruptcies have happened in every state. The small-scale grocery was once common, and now is nearly extinct. They’ve been hunted to extinction through the relentless competitive pressures of market consolidation.

Consider the graph that the US government has posted here:

In the late 1990s, a number of leading grocery retailers went on a buying spree. Between 1997 and 2000, more than 4,100 stores were acquired, amounting to almost a fifth of all U.S. supermarkets. Mergers and acquisitions by large grocery retailers, including Kroger, Albertson’s, Ahold USA, and Safeway, produced a significant increase in the share of grocery store sales by the largest firms. By 2005, the 20 largest retailers accounted for 61.6 percent of total U.S. grocery store sales, up from 40.6 percent in 1995.

If you look at the facts, you’ll admit that small-scale groceries have mostly been wiped out. And this happened during some of the same decades that saw the explosive growth of the US software market.

Clearly, it is possible to make a lot of money on the web. 37 Signals makes millions of dollars in sales, and Amazon makes billions of dollars in sales. AOL is making a shockingly large gamble on weblogs.

You include this surprising comparison:

Now, compare this store to many 2-3 men operations online trying to make it online offerring some service; there are thousands of such operations which wish they were making as much as that grocery store.

I assume you meant “people” where you wrote “men”. A 3 person web start-up will cost less than running a grocery store. Even a small grocery store is going to have more than $50,000 worth of inventory in it, then it will have labor costs, rent or real estate taxes plus land cost, licenses for health, fire, safety, etc. If you are going to compare businesses that have different capital requirements, then you might as well write “I tried to run a lemonade stand, but it never made as much money as Toyota.” The comparison is absurd.

Of course, as segments of the web mature, the capital requirements for starting a web site go up for that particular segment. Once upon a time, a long time ago, you could start a successful weblog for free. Nowadays, if you are starting a weblog which you hope will develop a mass audience, then you should probably have $100,000 for marketing and writers. Likewise, if for some crazy reason you decide to launch a competitor to YouTube, you should start with $100 million in the bank. In the future, you may need millions of dollars to get into any established segment. But some segments will remain open to the sudden hit that comes form nowhere. Weblogs, for instance, began to consolidate some time ago, yet new weblogs still occasionally burst forth and become large-scale hits. And one of the great things about the web (so far) has been the speed with which new segments emerge.

I’ve already written extensively about the costs of building a web site. And I’ve helped launched a number of successful sites that cost less than $100,000 to get going. And I continue to work with start-ups that have budgets under $100,000, and I’ve great faith that a number of these will become successful.

If you are thinking of starting a new business, you’d be wise to start it online.

I’ve been gathering data. I’m about to start doing a lot of work on EC2. I am curious about RightScale’s suggested upgrade technique, and also the criticism that appeared in the comments:

Once we’re confident in the new test version, we simply reassign the EIP 172.168.5.6 to the www_rel3 instance and shortly thereafter all users accessing the site are now receiving data packets from the new release. Remember, as long as the www_rel2 is available, you can easily swap back and forth between www_rel2 and www_rel3 until you are completely satisfied with the new site. And when you’re ready, you can terminate the old www_rel2 instance. See diagram below.

In the comments, Ajay says:

I’ve tried this technique in practice for upgrading a server on a site with live traffic and I’ve found that it does not work as you describe. Using your terminology, this is what I’ve experienced:

1) I assign the live EIP to www_rel3

2) For 75 seconds, live traffic is still directed to www_rel2

3) Then for 75 seconds after that, requests to http://www.rightscale.com go NOWHERE. Users get an error saying the browser cannot be found

4) Finally, traffic started getting routed to www_rel3, as expected.

The problem is part 3. I had always assumed that the switch was clean–either traffic would go to the old server or the new server, but it wouldn’t get lost.

Have you experienced that? How do you handle upgrades?

I am curious how long such switching really takes, on average.

Someone I’m working with asked me to look at SlideRocket, to see if we should be using it. SlideRocket describes themselves like this:

SlideRocket is online presentation software that provides for every part of the presentation lifecycle. SlideRocket integrates flexible authoring, intelligent asset management, secure delivery and analytics tools in a single on demand application. SlideRocket allows you to quickly create stunning presentations, store, organize, tag and search your assets, collaborate with your colleagues, securely share your presentations in person or online and measure the results, all in one integrated environment.

If you listen to their video, they talk about how great it is to have all of your assets on the web (or the “cloud”, as they prefer to say), so you’ll never run the risk of forgetting it at home.

One thing that occurs to me is how similar this is to what Phillip Greenspun was describing in 1999:

Interesting things can happen when you do something as simple as move an application from the desktop to a server. For example, at a small company once we had the idea of building a server-based replacement for PowerPoint. We wanted something simpler that we could edit from any Web browser. Once the slides were in the database, however, we realized that this could be used for collaboration. User A could authorize User B to edit a presentation and the two of them could authorize User C to give that presentation at a remote site.

Only after talking to people from PowerPoint-oriented organizations did we realize how revolutionary the system, which we called “WimpyPoint”, could be. Suppose Jane Executive went on vacation for three weeks. Upon her return she wouldn’t have to start calling colleagues to find out what had been going on; she could look at the WimpyPoint server and ask to see all presentations created within the last three weeks. When all the work of an organization is presented within PowerPoint it is a terrible waste to have that work locked up in monolithic files.

Once again, I am impressed with how clearly Phillip Greenspun understood the Internet, earlier than most other people did. He foresaw the declining importance of desktop apps:

What people need and, with the ubiquitous Internet, can finally get, are collaborative Web-based applications. Web-based apps let people use computers without becoming mired in system administration. Web-based apps help people collaborate. Web-based apps can weave an individual’s contribution into a larger work produced by many people over the decades.

The future is WimpyPoint, not PowerPoint.

If Web-based apps are so great, why aren’t we all using them now? Desktop apps serve one user at a time and tend to be copies of systems from the ’60s and ’70s. Web-based apps serve thousands of users simultaneously and oftentimes are based on completely new service ideas. Thus Web-based apps require programmers with great skill, imagination, and taste.

If Greenspun’s thinking has a fatal flaw, it is the relentless way he discounts the importance of visual information. He is one of those people who has a strong preference for text. There is nothing wrong with that, of course, but he consistently mistakes his personal preference for some kind of universal truth about communicating information:

You’re building a database. You’re modelling data from the real world. You’re going to have to write computer programs in a formal language. You have to design a user interface for that computer program. If you had an MBA then your natural first step would be . . . hire a graphic designer. After all, this computer stuff is confusing. Databases frighten you. What you really need is something that will look good at your next meeting. Graphic designers make pages that look good. You can always hire a programmer later to actually make the forms work.

…If you despair of learning how to do anything productive, what you might have learned from this chapter is that you should work with the programmer and user interface designer to build the site that fits your publishing model before bringing in a graphic designer to make it pretty.

This particular joke (that all graphic designers are stupid) pervades Greenspun’s work. I find it about as funny as I find “All women are dumb” or “All blacks are lazy” type of jokes. The history of innovation reveals this as a reliable pattern: those who understand the importance of a new technology often have some blindspot that keeps them from being the one who fulfills the potential of the new invention.

What Greenspun offered is an example of extreme minimalist design. It is functional, but confusing. He seems to have thought that if a page was minimalist enough then its purpose would automatically be clear. Any talented designer can explain the problems with that kind of reasoning.

SlideRocket offers some things that were less practical back in the 90s. The spread of broadband Internet connections allows people nowadays to expect video and other rich media. This could be described as the Web 2.0 version of WimpyPoint, but I assume Greenspun would disagree with the comparison. SlideRocket is built using Flash/Flex/Air. Greenspun was extremely critical of Flash. I assume he has been surprised (as I was) to see Flash evolve into Flex, which is, after all, a reasonably solid programming environment.

Still, SlideRocket strikes me as the modern version of what Greenspun had begun to see 10 years ago: the movement of presentation software to the web. I think it is curious how certain ideas (forums, email, chat) keep getting re-invented, over and over again, every few years, and yet, these re-inventions keep offering some real (if incremental) innovation, something that updates the basic idea and brings it into conformance with all that is now known, all the best practices that our growing experience with this new medium allows us to understand.

Having read a little about Apache Wicket, I wanted to learn more, so I went to the website. I was surprised by their list of Java frameworks:

Echo
Cocoon
Millstone
OXF
Struts
SOFIA
Tapestry
WebWork
RIFE
Spring MVC
Canyamo
Maverick
JPublish
JATO
Folium
Jucas
Verge
Niggle
Bishop
Barracuda
Action Framework
Shocks
TeaServlet
wingS
Expresso
Bento
jStatemachine
jZonic
OpenEmcee
Turbine
Scope
Warfare
JWAA
Jaffa
Jacquard
Macaw
Smile
MyFaces
Chiba
JBanana
Jeenius
JWarp
Genie
Melati
Dovetail
Cameleon
JFormular
Xoplon
Japple
Helma
Dinamica
WebOnSwing
Nacho
Cassandra
Baritus
Stripes
Click
GWT

Damn. How can anyone keep up? And that’s just plain Java frameworks. They are, apparently, not counting web frameworks like Grails, written in Groovy, which runs on the JVM.

There are so many languages, so many frameworks, so much new stuff. It has become impossible to make any pretense of choosing the “best” technology for a job. Rather, we are all subject to circumstance now, the accidents of our lives that bring us into contact with a particular technology at a particular time. This has always been true, but it beomces more obvious to me when I see a list like this.

This part of what Wicket says about itself has some appeal to me:

Most existing frameworks require special HTML code.

JSP is by far the worst offender, allowing the embedding of Java code directly in web pages, but to some degree almost all of the frameworks from the list (except Tapestry) above introduce some kind of special syntax to your HTML code.

Special syntax is highly undesirable because it changes the nature of HTML from the kind of pure-and-simple HTML markup that web designers are familiar with, to some kind of special HTML. This special HTML can be more difficult to preview, edit and understand.

Wicket does not introduce any special syntax to HTML. Instead, it extends HTML in a standards-compliant way via a Wicket namespace that is fully compliant with the XHTML standard. This means that you can use Macromedia Dreamweaver, Microsoft Front Page, Word, Adobe Go Live, or any other existing HTML editor to work on your web pages and Wicket components. To accomplish this, Wicket consistently uses a single id attribute in the Wicket namespace (”wicket:id”) to mark HTML tags that should receive special treatment by the toolkit. If you prefer not to render Wicket namespaced tags and attributes to your end-users, Wicket has a simple setting to strip them all out, resulting in ordinary, standards-compliant HTML.

No “special sauce” in your HTML means designers can mock up pages that you can use directly in development. Adding Java components to the HTML is as simple as setting the component name attribute. And you can then give the HTML back to your web designers knowing that they can change it with confidence.

Wicket, more than any other framework gives you a separation of concerns. Web designers can work on the HTML with very little knowledge of the application code (they cannot remove the component name tags and they cannot arbitrarily change the nesting of components, but anything else goes). Likewise, coders can work on the Java components that attach to the HTML without concerning themselves with what a given page looks like. By not stepping on each other’s toes, everyone can get more work done.
Existing frameworks are not easy.

Most of the existing toolkits have poorly defined or non-existent object models. In some cases, the model is defined using special XML syntaxes. The syntaxes may be so cumbersome that special tools are required to manipulate all the configuration information. Since these toolkits are not simple Java libraries you may or may not be able to use your favorite IDE tools such as editors, debuggers and compilers.

Wicket is all about simplicity. There are no configuration files to learn in Wicket. Wicket is a simple class library with a consistent approach to component structure. In Wicket, your web applications will more closely resemble a Swing application than a JSP application. If you know Java (and especially if you know Swing), you already know a lot about Wicket.

[Update: I've re-written this post to take into account the information that Javier Eguiluz posted in the comments.]

[Update: Jacob Coby points me to Google Trends, which offers the visuals for the numbers that I quote below.]

TIOBE only tracks computer languages, not frameworks. I’m interested in how Symfony compares with the other PHP MVC frameworks. Borrowing the search idea from TIOBE, I just ran these searches:

symfony

codeigniter

cakephp

drupal

The results (how many hits on Google):

Symfony – 6,270,000 (3,900,000 for ‘php’ and ’symfony’ together)

CodeIgniter – 748,000

Cake – 4,540,000

Drupal – 28,500,000

A comparison: Ruby On Rails: 12,600,000

Meanwhile, WordPress blows away everything else: 295,000,000

My guess is that WordPress has retarded the growth of PHP MVC frameworks. The 80/20 rule applies here with some force. WordPress meets the needs of most people who need a website. PHP allows for self-contained software, such as WordPress, which is something the world of Ruby has not seen. Ruby on the web has largely meant Ruby On Rails, which one has to be a programmer to setup and use. Designers, intelligent people who are not programmers, all such people can default to WordPress.

From the point of view of a computer programmer, the code in WordPress is fairly awful. But designers love WordPress, and it needs to be given credit for successfully creating a package that designers feel comfortable with. No designer would know how to set up a Ruby On Rails site, but most web designers know how to set up a WordPress site. And for all its limitations and flaws, it must be acknowledged as the dominant platform written in PHP. Compared to WordPress, all the MVC PHP frameworks are just a footnote.

I’m going to repeat these searches every 3 months, and see how these ranks change over time.

Back in 2005 I read The European Revenge. It was published in 1973. It was written by two journalists from the Economist magazine. They made the daring, controversial argument that at some point in the near future, European car makers such as BMW and Mercedes would be able to make cars that had just as much quality as American cars.

Their daring idea is something for my generation to laugh over. I came of age in an era when it was simply assumed that European cars were naturally better than American cars. Over the course of 30 years, the reputations of the various car makers had reversed positions.

Tonight, I find myself thinking daring, controversial thoughts about Ruby On Rails and PHP.

I recall in late 2005 when I first started hearing about Ruby On Rails. It was an amazing system for building big web sites much faster than anything before had allowed. I watched the video that showed how you could build a weblog system in 15 minutes. The company I worked at then hired a consultant who was also a very great programmer, a fellow who, by coincidence, had been writing about Ruby since 1999, well before most American programmers had heard of the language.

At that time, it seemed like Ruby On Rails was going to sweep all before it. PHP seemed like a dinosaur, headed for extinction. Software written in PHP was notoriously sloppy and insecure, and the programmers who worked with PHP tended to be less experienced than the programmers in almost any other programming community.

I started to study Ruby On Rails on the assumption that all PHP work would eventually disappear, and if I wanted to have a job in 2008, I’d better get with the new, new thing.

But, of course, Ruby On Rails failed to sweep all before it. It had a big impact on programming in general and on web development in particular. But of all the programming communities, Java was probably more profoundly effected than PHP. For Java programmers, Ruby On Rails lead to a re-thinking of fundamental assumptions.

When Ruby On Rails first came out, PHP was not competitive. There were no high productivity frameworks for PHP. But that has changed. PHP now has Cake, CodeIgniter, Symfony, and many other frameworks. These are all full-featured MVC frameworks, with command line scaffolding tools that automate the construction of basic CRUD forms and basic layout templates, and which therefore offer some of the same productivity gains that Ruby On Rails offers.

So much contempt has been poured in the direction of PHP that it seems normal to think it must be inferior to every other programming language. But it is worth remembering that some very good programmers have never liked Ruby:

I classify myself as one of those developers who don’t get Ruby, just like I don’t get Perl. Both are too hard for me. And there seems to be something in my brain that prevents me from learning both. It’s strange I never had the problem with C or C++ or Python or Lisp or awk.

…I just don’t like Ruby. This has more to do with how I think and work than with the true quality of the Ruby language. I’m sure Ruby is a fine programming language offering good OO features. But there is something in it that rubs me the wrong way. I tried to learn it when the first edition of the PickAx book came out. But somehow I can’t get past the “we are so OO, we write 3.abs instead of abs(3)” noise. I’m sorry. It’s not for me. There. I said it.

Most programmers first heard of the Ruby language when they heard about Ruby On Rails, so the language and the framework have always come in a bundle. If we unpack that bundle, do we find that some portion of it is stronger than the other portion? Since some good programmers dislike Ruby, yet the whole programming world has fallen in love with high-productivity frameworks like Rails, isn’t it safe to assume that Rails, the framework, is what made Ruby On Rails so awesome? That the awesomeness is clearly not tied to the language is demonstrated by the vast number of frameworks that have now been implemented in other languages. Most of these framework imitate aspects of Rails. If the awesomeness of Ruby On Rails was due to the Ruby language, then surely more programmers would be learning Ruby, rather than trying to imitate Rails in whatever language they know best. Many of these new frameworks have become successful in their own right, and have kept programmers in one language who might have otherwise converted to Ruby. Certainly, back in 2005, it seemed like there was likely to be a mass conversion of Java programmers to Ruby On Rails. That mass conversion was forestalled by the emergence of frameworks like Grails, written in a language derived from Java.

If the language Ruby doesn’t explain the awesomeness of Ruby On Rails, then that awesomeness can be captured by other languages, including PHP. I’ve been working a lot lately with Symfony, and I am impressed with the tools this framework makes available to me. Although all of the PHP frameworks had a late start, relative to Ruby On Rails, the PHP community now has a much greater diversity of frameworks to choose from, and the competition between the various PHP frameworks is likely to cause all of them to improve at a rapid rate, and also to specialize toward particular niches.

Where does that trend get us in another 3 years? Just as Mercedes went from being an inferior car to being a superior one, it seems to me that we could end up in a situation where the PHP frameworks are actually better than Ruby On Rails for any particular purpose that you can imagine.

I originally posted this over two years ago, on the Category4 blog. I want to keep a copy of this here, so I am reposting it. Nowadays I’m using Symfony for most of my projects, and Symfony does a lot of what I here criticize. I’ve come to the conclusion that:

1.) We must have a framework.

2.) It is too expensive to maintain our in-house framework, so we must adopt a well documented, open-source framework.

3.) All frameworks will have their stylistic quirks with which we will disagree.

4.) The benefits of using such frameworks will, in the long run, out weigh the negatives.

This is what I wrote two years ago (with one minor edit):

In my last post I tried to articulate some of the things I’d like to see in a PHP framework. Youngj posted a comment and suggested that I try Qcodo. I have a problem with Qcodo’s template system, and it is a problem that I have with a large number of other PHP frameworks: HTML that gets embedded in the PHP code (also known as PHP code that writes HTML). I’ll let this criticism of Qcodo stand-in for my criticism of all PHP frameworks that do this (which is most).

Please consider the example Qcodo gives on the front page of their site:

<? $this->RenderBegin(); ?>
Id: <? $this->txtId->Render(); ?>
Title: <? $this->txtTitle->Render(); ?>
<? $this->RenderEnd(); ?>

Bad. Not from a programming point of view, but from a work flow point of view, in a professional web design firm. In this example, the form is no longer being controlled by the designers. What happens if the designer decides that the Title should have a red background, but they don’t want to change the CSS values for all inputs on the page, so they need to give that particular input an id? They then need to come to me, the programmer, and say “Could you add an id to the Title input?” Thus, work is shifted from the designers to the programmers. This is bad economics, since in most web design shops the programmers are paid more than the designers. And why would I want extra work? More so, too many projects get delayed because the programming runs behind schedule. It is better to enable designers to do as much as possible on their own.

When the PHP code writes HTML, the programmers end up doing work which the designers can do better.

Much better would be something like this:

<form method=”post” action=”index.php”>
Id: <input type=”text” name=”id” value=”< ?= currentValue(”id”); ?>” />
Title: <input type=”text” name=”id” value=”< ?= currentValue(”title”); ?>” />
<input type=”submit” value=”Update this article?” />
</form>

This block of HTML is something the designer can control. What happens if the designer needs to add an id to any of these HTML elements? They can add it on their own, without bothering the programmer. And what happens if the client decides that the value of the submit button should be reworded? Who does the work go to? The designer or the programmer? Qcodo sends the work to the programmer, but in my example the work can go to the designer. That means Qcodo is bad economics, at least from the point of view of a professional web design firm.

I might ask why something like this is so popular among web application frameworks:

Title: <? $this->txtTitle->Render(); ?>

I believe that the programmers who write pages like this (and I used to be one of them) are hoping to achieve productivity gains by automating some of the work of creating forms. I believe this is a false economy. I don’t see that the number of characters is dramatically less in the Qcodo example than in my example. Whatever small gains Qcodo makes, they are lost as soon as a designer needs to come to a programmer to request a change.

I do believe these frameworks (such as Qcodo) achieve real efficiency gains if the programmer is working as a freelancer, and does both the programming and the designer work. If I was a single individual, unattached to any company, then I would probably use a framework with a template system just like Qcodo (in fact, I did, when I was just starting out). These template systems probably save programmers some time, but they waste time when non-programmers need to edit the pages. In a professional web design firm, one that handles large sites and therefore must have multiple people working on the site, I believe the best bet is to stick as close as possible to pure HTML. HTML is one of the few things that everyone in a web design firm will know.

Qcodo does a much better job of explaining itself and its philosophy than many other PHP frameworks. And from what I can tell, it seems to be well implemented. But it does not trust designers, and it writes too much HTML on it’s own. Consider the way it describes its code generation abilities:

Taking a quick look at the web front-end of codegen, you’ll see that only the top dozen or so lines (including comments) directly deal with the CodeGen object. Everything else past it is simply to paint a pretty HTML page to display the results.

“To paint a pretty HTML page”. I don’t think I’ll give it a try. I’ve a lot of PHP frameworks to test, and I’ve limited time, and this one sounds like it is taking too much power away from the designers.

If a programmer is also their own designer, then I can imagine why they’d want to write all the forms in PHP – it would save them some time. Peter Bowyer summed up that point of view with this post from 2003:

My current project is the biggest site I’ve had to design and build by myself. And in the process of planning the code I’ve realised why so many people have a framework they use. I’m having to do too much stuff here from scratch; code that should just plug in from existing projects. I’ve written a SQLBuilder library, polished up my own take on Fusebox, and am developing the FormProcessor library (the more I use it the more I realise why people do forms the “normal?? – i.e. writing PHP to construct them – way: it’s such a pain writing these forms by hand. Dreamweaver really doesn’t speed up the generation much.

Right. But if the forms do need to be edited by a designer, perhaps using Dreamweaver, then it is better to have the form elements defined as pure HTML. If you are a lone, individual programmer you can do everything to suit your own taste, but when you’re working in a shop where several people work together on one site, then one needs a framework that maximizes how much the designers can do on their own.

This project is tracking all the new upcoming web technologies, like CSS 3, which will allow us to have rounded corners without having to use images. Also, elements such as Canvas. Also, other technologies, such as transparency in PNG images.

Right now, none of these technologies are usable because at least one major browser doesn’t support them. But clearly, over the next year, most of these technologies will become usable in all major, newer browsers. Then the question becomes, at what point do these features have enough market share that you’d want to use them on a commercial site?

The reality that a new version of HTML will soon come out makes me aware of how stagnant things have been for the last 6 or 7 years. Compared to the incredible speed with which things evolved during 1994 to 1999, the last few years have been one of consolidation:

I’ve younger friends who’ve become web designers in the last 4 years and who’ve never seen an evolution of the technology. They learned CSS 2 and HTML 4, and that is what they are still using now.

Universities now offer classes in something called “web design”.

There is now a profession called “web design”.

This profession is now in control of the standards around which the profession does its work.

The usual  balance-of-power conflicts between for-profit corporations and the professions, have become normal for the field.

Consolidation has been the rule. Perhaps it was necessary.

The web is in desperate need of new element options. My friend Chris Clarke likes to joke that HTML 5 will bring web forms up to the level of interaction offered by Visual Basic 1.0. I am hopeful that this next year will rekindle the forward momentum of innovation.

A very interesting post by I Am El Gringo:

For the time-constrained, I submit to you the results of my highly scientific research:

• Yahoo: Minimal Use of tables. I found a picture of Hugh Downs horizontally aligned with it’s caption in a table
• Google Home Page: Not only does Google use tables for it’s iconic home page, it embeds styling in the <td> tags. The horror.
• YouTube: Uses tables for of layout of videos
• Windows Live: Uses tables for footer layout
• MSN: There is one table, but it’s only for stockquotes which is tabular data
• MySpace Semantically pure. MySpace. Whoda thunk it
• Facebook: Does form layout with tables
• Blogger: No tables anywhere on the front page
• Orkut All tables all the time
• Rapidshare: A table with a single <td> for header placement. And again a single <td> table for the central “browse” section. Tsk tsk
• Microsoft: Navigation bar is a table. What did you expect? Unicorns and rainbows?
• Google India: It’s the same Google layout. I wonder if they used copy and paste for the template?
• Ebay: Tables, tables every where
• Hi5: Tables for every thing, pretty much. BTW, I didn’t even know this site existed until last week. Alexa rank 14!?
• Photobucket: Tables for photo gallery layout
• AOL: AOL’s layout is semantically pure! Friggin AOL?
• Google UK: Same GOOG layout. I’m now sure the copied an pasted their html
• Amazon: Now that’s just silly
• IMDB: They used tables for their 3 column layout. What! No CSS framework?
• Imageshack: Semantically pure as the driven snow.
• Finally, even though it’s not on Alexa’s top 20, log in to your Gmail account and look at
  the use of tables

My Hypothesis: Pure CSS design == overcompensation

So, the five companies that use CSS are the web powerhouses–MSN, MySpace, Blogger, AOL and Imageshack. MSN, MySpace and AOL have been maligned for years throughout the web savvy community. My hypothesis is that these companies are overcompensating for the crap that they’ve taken thoughtout the years by designing their site in pure CSS.

Other companies that have more web street-cred like Google and Facebook don’t really have to worry about how the web design community sees them. This leads to things like Google making extensive use of inline styling on their homepage instead of putting it in their stylesheet. I’ve never heard anyone claim that the Google folks are slouches at the web design/development thing. Why is that?

Nikolay Barkhatov feels that a web site should just be an interface to an application:

More I think about web development more I believe there is a clean separation between actual application and web application. Moreover, I believe there is no such thing as web application. Web interface is just an adapter to real application. It’s hard to convince managers to architect application the way where web adapter decoupled from actual application. Managers simply do not see benefits of doing that and do not allocate time for it. Product development ends up with one giant mess called web application where tier separation is a formal thing everybody talks about but nobody can clearly draw the line between them. And finally there is a requirement to quickly come up with remote API for third party developers… Very sad…

It is curious to me that such an idea seems natural, even akin to common sense, to a Java programmer, and yet it has little appeal to non Java-programmers. Why is that? This is a cultural thing. The people drawn to Java are the type of people who want to do everything the theoretically (or academically) correct way.

On some level, Nikolay Barkhatov’s idea sounds very smart. And yet, a lot of other smart people have decided to take another path. David Heinemeier Hansson is a smart person, but he didn’t feel the need to separate the website from the application he was trying to build. Instead, he create the web framework Ruby on Rails, to make it easier to marry code together with a web interface, while keeping things organized. The framework implements an Model View Controller pattern, so that the code and the web interface can work together in an intelligent way. Ruby on Rails was such an obviously good idea that other people began imitating it. Every computer language now has an imitation of Ruby on Rails. For instance, for those using the PHP computer language, there is the Symfony framework.

But having said all that, I admit that I’m fascinated by the idea of separating the application from the web interface. If I wrere to try it, I would write the application in one language, and the web interface in another language. I don’t trust myself to write both in the same language, the temptation would be to cheat, and mix the two together.

If I were to try to separate the application from the website, I might write the application in Java and the web site in PHP. I would use XML to get the data from the Java appliation to the PHP. For instance, suppose I build a site for value investors who want to see whenever a stock falls below a P/E ratio of 10. The application would sit on the server, intake stock information from various sources, look for low P/E ratios, and output the results as XML files. The PHP could read the XML files and construct the website from that.

One advantage of this approach is that the caching is automatic. The PHP only needs to read from static XML files.

How to handle input from users? I suppose that is where the separation of concerns breaks down, and perhaps this is the issue that has made frameworks more popular than tiered approaches to sites. True separation could be maintained if you use normal HTML/PHP forms that are submitted to PHP code that then transforms them into XML files that the Java could then read. But this gets clumsy. I wonder if there are any types of sites where this would be the optimal setup?

In late 2005 I was working at Bluewall and the owner of the company became convinced that some of the sites we were building were about to go super nova, so he got a second server from Rackspace. This was a stupid decision, but it is a a common one. I find clients often get over-excited about their projects. I’m sure some of this excitement is a healthy part of being emotionally invested in the project that you’re trying to bring to life, but that same excitement can lead to needless expense. Dedicated servers from RackSpace are expensive and, as it turned out, Bluewall’s need for extra servers was years in the future. The purchase decision in 2005 represents a lot of wasted money. So why didn’t the second server get shut down? Because once you have even one (low traffic) site on a server, it becomes a pain to shut down that server.

The Second Road had a similar experience. They started off with 3 dedicated servers, plus a firewall and load balancer, from Rackspace. The cost was $1,800 a month. When we got involved with the project, we moved the site to a server from Hostway, which costs us $150 a month (and on which we have several other websites, all sharing one server). So far, this has met all of the Second Road’s needs on the web.

Dedicated servers represent a lot of potentially wasted resources, especially for a small startup which may grown quickly, but which also may NOT grow quickly.  And what is growth, when you’re talking about traffic? If you get mentioned on BoingBoing, and suddenly you are getting 200 requests a minute, and your server collapses under the load, is it time to get another server? What if the spike in traffic lasts 2 days and then dies away and never comes back? If you get another server based on that one spike, you’ll regret it. Unless the spike is permanent. In which case you’ll regret not getting that extra server sooner.

These experiences have me interested in the new cloud services, which promise just the right amount of computing power that your site needs, no more no less. The idea of the “cloud” is that you can scale your resource needs up or down in increments finer than a whole server. So for the next big client we get we will try a cloud service. These are 3 we are looking at right now:

Mosso (this was just bought by Rackspace)

Amazon

IronServers

There has been a lot of investment, by a lot of companies, in cloud services and, really, it is hard to keep up. The fact that as good a company as RackSpace was willing to buy Mosso speaks well about Mosso. Since there is no way to evaluate all the contenders now competing in this field, we are forced to look at the best known, and then one of the somewhat unknowns, just as a test.

For the Amazon services, one of the best known users is SmugMug. They’ve mostly written about the S3 service, but also some about the cloud service.

Amazon is also now moving into the Content Distribution Network business. This is an aspect of building websites that we haven’t yet researched much. My feeling is that neither us nor our clients will ever be interested in the nitty gritty details of getting content to every corner of the world. We don’t want to be up late at night wondering “Is the site running fast enough in Malaysia?” So we’re pleased to see Amazon in this space and we hope they do a good job, because certainly we will give them a close look, if we ever work with a client who really needs that kind of scale.

This is from the article on Ajaxian:

With nearly 2.5 million requests per day to the jQuery website, the jQuery project team is constantly on the look out for ways to decrease hosting costs while still managing the growing number of requests for the site’s resources. Originally leveraging Amazon S3 for many of their static pages, the project has now turned to Amazon’s new CloudFront CDN. The change has allowed for jQuery pages to be globally hosted as opposed to being centrally located in Amazon’s Seattle-based S3 hosting center.

In tests, John Resig, team lead for the jQuery project, noticed substantial performance gains based on the switch:

“I ran a similar test here in Boston and even managed to see a large improvement. I was seeing latency of anywhere from 50-200ms on Amazon S3, but only a latency of 17-19ms with CloudFront.”

What does all of this mean? It means that the jQuery site is going to load even faster than it does now. We already receive some excellent hosting from Media Temple but being able to off-load these static files to the fast-loading servers will only make for a better browsing experience.

In less than 24 hours the project had received almost 2.5 million requests for over 50GB of data. The only drawback is an increase in bandwidth costs but still substantially less than that of a traditional hosting plan. The jQuery project makes use of the Google AJAX API as well and recommends it as choice for linking to the jQuery and jQuery UI libraries.

Darren Hoyt wonders how we might both empower clients to edit their own sites, yet keep them from breaking the sites.

I believe that, eventually, this problem will be solved at the level of TinyMCE. I suspect we are 3 or 4 years away from that. I think eventually visual editors like TinyMCE and FCKEditor will have the same power as Dreamweaver has now. By that point, however, the Javascript libraries will be frighteningly bloated, so the future scenario I’m describing depends on faster computers, faster browsers, faster Javascript engines and faster download times (imagine sites with 8 or 9 megabytes of Javascript – I believe this is the future, even if it sounds freakish to us now).

All of the current valid criticisms of Dreamweaver will apply to the future visual Javascript powered editors, and new complaints will emerge, I’m sure. But aside from all that, I believe that clients will eventually be empowered to design much more freely than they can now, with the Javascript automatically fixing the errors the clients accidentally create. Empty tags, tags that don’t close, incorrect nesting of tags — all this can be fixed automatically. This, I believe, is what will happen in the long run. In the short run, I’m less sure of the path to take.

You can set up a website for free. And Google has spent over $10 billion on its website. So it is safe to say that most websites cost between $0 and $10 billion. How much will your site cost? There is no way to be certain ahead of time, but we can offer some advice.

 All websites fall into two categories:

 1.) they offer content

 2.) they offer a service

 Examples: A blog about computers offers content. A site that lets you put dates into a calendar is offering a service. Many of Google’s websites offer a service: email at Gmail and spreadsheets at Google Docs. If you have an already existing company and you’d like to put up a few pages describing (or selling) your products/services, then you are envisioning a content site. If you’d like to do a magazine style site, then you are still considering a content site.

Price Ranges

 Different kinds of projects have different price ranges. We can only offer very rough guidelines as to the kinds of prices we’ve seen for previous projects. Talk to us about the specifics of the project you are thinking of and we could probably offer you a better idea of how much you need to be prepared to spend.

 1.) A personal or company blog, set up using a free service: $0 (or whatever value you place on one hour of effort). But to keep this going and acquire traffic will probably cost thousands of dollars, at least in terms of your time.

 2.) A small company site that has 5 to 10 pages describing the products/services offered by the company. $500 to $2,000 depending on how prepared you are, and also on how clear in your own head you are about what you want. Disorganization and changing your mind are both expensive.

 3.) A company site describing the products/services offered by the company, and operated by software that allows the staff at the company to continually update the content on the site: $2,000 to $20,000.

 4.) A company site that both describes products/services and also offers some kind of software service. For instance, a real estate firm might allow users to look up listings based on zip code or a manufacturer of skateboards might allow users to design their own skateboard, mixing and matching those variables that can be customized. $10,000 to $40,000 depending on the complexity of the software to be developed. 

 5.) An online store that sells digital downloads of songs or videos: $25,000 to $100,000 depending on the variety of prices/royalties you want to support with your suppliers and how much marketing you want to do. If you want original technology for handling the songs or videos, you might easily burn through another $50,000 on computer programming.

 6.) A company wiki so the staff can document a particular kind of information – perhaps support information for your customers, or technical info for use by your sales people, or a human resources directory aimed at answering common staff questions. This should be on the low end of $2,000 to $30,000. The biggest variable is whether you need some special, custom feature. If you can use unmodified, off-the-shelf software, you can keep things quite inexpensive. But as soon as you get into custom programming, the cost will sky-rocket.

 7.) An online magazine/blog meant to dominate a particular niche: $25,000 to $100,000. For instance, you might want to cover fishing in Alaska. You’ll probably want to establish commercial relationships with 6 or 7 avid fishermen who are also known to be good writers. They might each post something once a week. You might spend $50 or $100 a week on each of them, depending how much news they can commit to delivering. You should also hire a lawyer to create an agreement you want the writers to sign.

 8.) An information resource that, to succeed, must be the best: $500,000 to $5,000,000. One fellow came to us and said “I have $20,000. I’d like to build America’s largest database of foreclosed properties.” We replied: “This is obviously a great idea and, if you can pull it off, the site will surely be a great success. But you should really go out and raise $1 million to get started. You’ll need a team of programmers working constantly to get data into your database, you’ll need a lawyer to work out multiple deals with your sources of information, and you’ll need national marketing. You’ll probably need $2 or $3 million over the first two years, but you probably should not even start unless you can raise $1 million.” The other possibility is to start small and build a database of foreclosed properties for a single metropolitan area. If the site is successful, it should be easy to raise the money to go national.

 9.) An online social network for some demographic niche, for instance, new moms, or hockey fans: $20,000 to $500,000. You can keep costs low by using off-the-shelf software, but you will probably want to customize the look and feel of the site, for the purpose of branding. The more services you offer (upload photos? upload videos? search for other users in the same geographic area?) the more expensive the programming will be. If your target demographic niche is risky (perhaps you want to run a site that offers services to recovering drug addicts) then you’ll want to hire a lawyer to write an air-tight Terms Of Use agreement. Lawyers are expensive, so this will drive up your start-up costs. Also, you’ll need a budget for marketing or you’ll never acquire any traffic.

 10.) A completely original idea for a software service that no one has ever thought of: price unknown. There is a rule in software design that the more original the idea, the harder it will be to accurately estimate the cost of the project. Robert L. Glass has written several books on why this is so. Check out his book “Software Runaways: Monumental Software Disasters”.

 Below are some guidelines about factors that will effect the cost of your website.

How does a site acquire traffic?

 If you are considering a content site, then the site’s most urgent need, and one of your biggest expenses, will the be the effort to acquire traffic.

 Rule #1: a content site, to be successful, needs new material everyday.

 Rule #2: do not underestimate how utterly demanding it is to provide new material daily.

 Rule #3: if you do not yet have the amount of traffic that you want, re-read rules 1 and 2.

 Rule #4: if you find yourself wanting to ask something like “Does the word ‘daily’ include the weekends?” then re-read rule 3.

 Rule #5: providing text-based content is cheaper than providing video, therefore, even if you are focused on video, your site will need at least one writer to provide high-quality, professional content that can act as filler between the videos. Or you need a big budget.

 Rule #6: flows of traffic on the web have become increasingly rigid and hard to change, therefore more effort (and expense) is needed to acquire an audience.

 Rule #7: it is extremely difficult to transfer an offline audience to an online destination. Even authors like Lee Stringer, who’ve spent months on the New York Times best-sellers list, attract little attention when they blog online.

 Rule #8: when you hire a writer who has an already existing online audience, you usually capture a substantial portion of their readership.

 Rule #9: the previous two rules imply that hiring an online writer who already has traffic is a better way to capture traffic than hiring an offline writer, even if they are a star. Sites focused on sales will generally want to aggregate news relevant to whatever is being sold.

What drives software development costs?

 If you are considering a software site, be aware of what drives the costs of software development.

 Rule #1: the more customization you need, the more expensive things become

 Rule #2: the bigger the project, the more customization you’ll need

 Rule #3: when programmers invent things from scratch, the number of bugs increase, and the amount of time that will have to be spent beta-testing

 Rule #4: if you want original technology, then your programmers will need to invent things from scratch

 Rule #5: to control risk, all programmers must live by the “Hit By A Bus” rule, which is, if they are hit by a bus and killed, with a project only half-done, another programmer should be able to easily take their place

 Rule #6: the more original the technology, the harder it will be for other programmers to understand the code (creating possible violations of the “Hit By A Bus” rule)

 Rule #7: cost, risk, complexity and understandability (of the code) can be controlled by the use of a software “framework”. Some of the popular frameworks for web development include Ruby On Rails and Symfony (written in PHP)

 Marketing

There are two main approaches to marketing a site:

1.) The Big Bang (hire a PR agency, buy lots of ads, make a big splash)

2.) Slow and Steady (produce quality material/service, rely on word of mouth)

Slow and Steady is always more cost effective than the Big Bang approach, so the Big Bang approach  should not be pursued unless you have money to burn and you are in a great hurry.

Success

The success of your website will depend on two things:

1.) Clarity

2.) Discipline

In some ways the web is a revolutionary new medium with new potentials and new dynamics, but in many other ways running a website is exactly like running any other kind of business. You must have clarity about what you hope to achieve, and then you must have the discipline to stick to your plan. You should be able to describe the goal of your website in one sentence (and if you can’t, then how will your visitors ever understand what the site is for?). If you are unable to sum up your goals when you are speaking with us, there is a good possibility that you haven’t yet achieved clarity (but we would love to talk to you about your idea – we may be able to help you achieve the clarity you’ll need).

An entrepreneur is in trouble if their description of their idea for a new startup comes out as a confused run-on sentence, such as this: “We’re going to be an agency that puts ads into videos and we’re also going to be funding new video artists with our ad money, because we believe in the possibilities of nurturing these new talents, and we’re going to be an aggregator that accumulates the best new talent in video and our software will offer unique technology to advertisers, such as the ability to keep track of exactly how long each viewer watches each ad.” And yes, this is similar to something that was actually said to us. There are at least 4 different business ideas in that sentence, which is at least 3 too many. (If you have an existing business with many divisions, it is fine to have a site that reflects the diversity of those divisions, but a startup needs to be more focused.)

Clarity controls costs. If you are unsure of what you want to do, you will probably change your mind half way through the project, and that will lead to additional expense.

Imagine you hire a carpenter to build a deck on the back of your house. You tell the carpenter that you want the deck to be 10 by 20 feet, and to be built of oak. The carpenter goes out and buys the needed wood. Now you change your mind, and you tell the carpenter that you want the deck to be 8 feet by 18 feed, and to be built of mahogany. The carpenter has to charge you for all the oak that they bought. Changing your mind is expensive!

When it comes to home renovations, people have an easy time understanding why changes drive up costs. However, for some reason, when it comes to the web, people lack this understanding. It is true that the web is a fluid medium, but still, once work commences, the time invested needs to be paid for. If you hire a web design firm and tell them “I want a simple site that lists the services my company offers” then they will start to build one kind of website. If you change your mind and say “I really need a site where everyone in my company can document their work, so I can track what they are getting done” that is a completely different site and needs completely different code. The work on the first site would be thrown out (but you would need to pay for it) and work would begin on your second idea.

Assumptions to Avoid

Many of our clients have started web projects with (sometimes unconscious) assumptions that, in the end, caused them unnecessary anguish and expense. These are the twelve most common mistaken assumptions that we’ve seen:

1.) I (the client) should be all things to all people.

2.) Feature fetish: “The more features I add, the more popular my site will be.”

3.) If I build it they will come (also know as, “If I have a clever idea, I won’t need a marketing budget, because the site will get mentioned on TechCrunch/Oprah/popular blogs/the local newspaper”).

4.) If one person offers a single piece of off-hand, poorly thought-out, casual feedback, we will immediately re-design the entire site to comply with their feedback. And then tomorrow, when someone else offers some casual, poorly thought-out feedback, we will do the same. And then the next day…

5.) The idea for my site is unique, therefore completing it quickly is urgent. If we are not the very first site that uses this idea, then we will fail. My idea is so good that others will soon imitate it.

6.) Things that move or blink grab my attention, therefore if everything on the page moves or blinks, we will have a truly attention getting site.

7.) Text is boring.

8.) The more unusual and unique the interface, the more interested people will be.

 9.) Websites are fluid so any problems that arise can probably be fixed in a day or two.

 10.) If my site starts off with a narrow focus, then its prospects for growth will be limited. People will pigeon-hole it and I won’t be able to add new content subjects later.

 11.) I can spend my money at an unsustainable rate because 6 months from now my site will be bringing in a huge profit.

 12.) I have no experience on the web, but I have uniquely creative ideas, which I think will be enough for me to build a successful online business.

Disorganization

Disorganization will cost you. If you hire a web design firm to build a site for your company, then they will need certain things from you, such as the company logo, and a description of what you are selling. If you are weeks late delivering these things, you inflict delays on the web design firm. Often, the contract you sign will specify some kind of penalty for the delays you cause. A common penalty is to have your project moved to the bottom of their priority list – which could delay your web site several months (this is how client lateness was handled when we worked at Category4). While there is some text and images that the web design firm can provide on their own, there is usually some core material that only you can provide – a description of what you sell being a good example. Have your images and text ready.

Brainstorming

A gold rush mentality has come to shape expectations of the web. Many people now have the mistaken idea that if they can simply come up with an idea creatively unique enough, they can attain great wealth on the web.

Organizations, like people, can suffer pathologies, and the one we’ve seen most with web ventures is what we call “the endless brainstorm”. While it is fun to engage in a brainstorming session, and while this can be a vital tactic to maintain the innovativeness of an existing business, constantly seeking a new idea can be a waste of resources for a venture that has not yet launched.

We once watched, with a certain degree of horror, as almost $1 million dollars were spent on what turned out to be a year long brainstorming session. Many good ideas were hatched during that year, and each idea was pursued for a few weeks, then abandoned once the first difficulties were encountered. The entrepreneur in this case wanted to find an idea so unique that he would face no competition in the field – but this is not the way the business world works. Generally, any good idea will attract multiple efforts, so competition is to be expected no matter what the basic idea is.

We have a friend who worked at AOL during the early days, and who has since worked with several startups, both successful ones and failures. One of the many good pieces of advice he gave us was: “In a startup, ideas are a dime a dozen. Anyone working at a startup will have 2 or 3 great business ideas a week. The greatness of the idea doesn’t matter. All that matters is execution.”

One of the most common confusions that arises from brainstorming is the desire to build a site that offers both unique technology and unique content. In our opinion, a small firm (less than 25 employees) can either create great content or it can create great software, but never both.

Population dynamics

For the last 3 or 4 years we’ve had clients come to us and talk about building up sites using content contributed by users. This can be a great strategy, but you will find it tough to elicit the kind of participation you seek, so if you are targeting a narrow niche, be aware how the population dynamics will work against you. Not long ago a potential client came to us and said “I’d like to build a video site focused on my hometown of Charlottesville, Virginia, and I’d like most of the content to come from users.” We put together the following estimates to demonstrate the difficulties he would face during the first 6 months of the project.

500,000 people

In the whole extended region (Charlottesville and the surrounding 7 counties), there is only about 500,000 people.

250,000 people

Since your project is focused on video, only those people with broad band connections to the Internet will be able to view your site. And only about half the population has broad-band access to the Internet.

200,000 people

It is reasonable to assume that very small children and many senior citizens will be unable to use your site. Therefore, at most, your target audience will consist of 200,000 people.

20,000 people

If you have some really amazing marketing, maybe 10% of the target population will come check out your site during the first six months.

10,000 people

Assume you’ll have a 50% bounce rate, meaning half the folks who visit you may never visit you again. That is standard. That takes us down to 10,000 people.

100 people

Do you want users to contribute material? That is a high level of interaction, and most sites only get that kind of response from about 1% of their audience. That takes you down to 100 people. Can you engineer your business model to work with 100 active users?

Humility

No one can accurately predict where the stock market will be in 2 years (otherwise they’d already be a trillionaire). For almost the same reason, no entrepreneur can fully know where their startup will be in 2 years: there are simply too many variables. It is important that innovators be humble about the limits of their knowledge. A fantastic insight in one area can carry you a long way, but it will have to adapt to the millions of small facts that will arise, almost daily, and shape the history of your insight as it passes from concept to reality. If you launch a startup, the odds are against you from the start, and you’re only hope of success is to remain open minded about what might work.

If your startup eventually succeeds, it will likely be for reasons you did not initially predict. You must proceed with the attitude of a scientist conducting an experiment, and when you get back the results, you must not argue with them, but rather, you must adjust your mental model of the world. When Sir Alexander Fleming saw that mold on some bread crumbs was keeping bacteria from growing in a petri dish, he said “Astounding! That mold must be producing some substance that kills bacteria! Perhaps that same substance can be used to kill bacteria in humans?” How many of us would be dead right now if he had rejected the evidence, and said simply “That’s impossible! Mold can’t hurt bacteria! Everyone knows that!” Over the previous 60 years, two other researchers had already noted the lack of bacteria in the presence of certain types of mold, yet they failed to reach the conclusion that Fleming did. An entrepreneur would do well to emulate Fleming’s open mindedness.

Peter Drucker, perhaps the greatest business guru of the 20th Century, once remarked that innovators are often disappointed by the manner in which their innovations become popular. In his 1985 book, Innovation and Entrepreneurship, Drucker relates the story of Alfred Einhorn, who invented Novocain, which then became popular with dentists as a local anesthetic. Einhorn held a contempt for dentistry, since it represented such a small niche of medicine. He felt that Novocain should be used by surgeons for all forms of surgery, and so he waged a campaign against the use of Novocain by dentists. In the end, his innovation was successful despite him, rather than because of him. According to Drucker, this pattern, where a product or service is undercut by the entrepreneur who is trying to promote it, is extremely common.

When I first read Drucker’s book I found it hard to believe that an entrepreneur would actively sabotage their own innovation. However, having now spent several years working with startups, I’ve seen that it is, indeed, a common pattern. Many entrepreneurs starts websites at least in part because they consider themselves uniquely creative and insightful, and they want the whole world to see them as they see themselves. The website they launch proves them wrong: their insights are proven false, what works in the end is something unexpected. For instance, in 1992, when Bo Peabody launched Tripod, he was thinking that the site would offer content aimed at college students. His idea failed. The company was saved because some of the programmers at the company had started a side project that allowed anyone to create their own web pages. This then became the future of the company. In his book, Lucky or Smart, Peabody says it is important to be smart enough to know when you are getting lucky. And then, you have to be willing to accept that luck. This takes humility. What’s needed in an entrepreneur is emotional resilience, the kind of strength that allows for openness to the unexpected.

Twice now I’ve seen an entrepreneur sabotage their own website because it became successful for what they felt were the wrong reasons. This emotional resistance to success is nearly always inspired by one of two factors:

1.) The success is with a small niche. The startup was suppose to grow till it was larger than Google, and success with a small niche is, therefore, extremely disappointing. The niche might be big enough to potentially generate several million in revenue, but it won’t ever be enough to catch up with Google.

2.) The success is of a conventional type and, therefore, the entrepreneur regards it as boring. Perhaps the site was suppose to pioneer an altogether new style of interaction among humans, and instead the part of the site that becomes popular is of an old type – for instance, the blog on the site becomes highly successful. The entrepreneur is then disappointed, maybe even angry, to be the owner of a boring success.

Here then, are some fatal traps to avoid. Without open mindedness about the type of success you may encounter, your startup is doomed. And without humility about the limits of your knowledge, your startup is doomed.

Equity

Except for when we were working at Category4.com, every client we’ve worked with has offered us equity in their business. So far, we’ve always turned these offers down. If you have an idea for a web startup, and you’d like our help, I imagine that you too might be thinking of offering of us equity. You’d be injuring your own best interests if you did so. Offering equity means that you’d like enter into a business partnership, and business partnerships are a bit like marriage. Suggesting such a union to people you’ve just met is a lot like proposing marriage on a first date. First of all, it suggests desperation. There is nothing wrong with that in itself – we’ve all read stories in Inc magazine of a venture started with too little money but which succeeds against the odds. However, second of all, and much more seriously, it suggests that you are not thinking very clearly.

A business partnership means that we need to take an interest in every aspect of your business, including all of the flaws. This can be brutal. This is not a good way to make friends.

Recently, an older gentleman approached us with an ambitious idea that he could not finance himself. He already had a successful operation providing hospitality services to movie companies that film on location in Latin America. His idea for a web startup was somewhat related. He suggested that we take equity in the new startup, as a form of payment. We explained to him that a partnership was a bad idea because we hardly knew each other. If we were to take his offer seriously, we’d have to interview his staff and find out if anyone disliked him, and if so, why. We’d have to find out if any of his key people were planning to leave soon. We’d have to talk to his customers and find out if they liked him or hated him. This should not be any of our business, yet if we formed a business partnership it would be, literally, our business.

As it turned out, we were lucky not to form a partnership with this person. We later discovered he was struggling with issues arising from a past history with alcohol, and even though he’d been sober for 20 years, he was still exhibiting some of the behaviors that had sabotaged him before. Incidents such as this are the main reason we are wary of taking equity. But here is the thing – you should be wary too. What about our flaws? What about the stuff we get wrong? If you are offering us equity when you hardly even know us, then you are trusting us too much. And, to us, that indicates that you are not thinking clearly about the risks that your venture will face.

People should know each other at least a year before they think about forming a partnership together. Web startups routinely break this rule, and I suppose that is fine if the company is built to be flipped – that is, you assume that you’ll cash out in a year anyway, so you don’t have to worry about other people’s flaws -  there won’t be enough time for those flaws to come back and haunt you. However, as a general rule, business partnerships should be for people who know each enough that they can trust each other. Even in those scenarios where the company is built to be flipped, you’ll find that having one wrong person on your team, as an equity holder, can significantly hurt your chances of selling the company.

Contact Us

 We have worked on sites that had budgets under $1,000. We have worked on sites that have cost $250,000. We have long associations with a network of professionals which allows us to quickly assemble an agile team with exactly the skillset needed for the construction of your website. Whether your site needs graphic designers, computer programmers, videographers, writers, marketers, photographers or system administrators, we can bring together the talent you’ll need. We can even give you some rough guidance about which situations you’ll need a lawyer for. We are especially able to help web-focused startups whose initial 6 month budget is between $10,000 and $100,000. We can also help with existing sites that have so far failed to attract an audience.

 For every rule we’ve written here, we can think of three qualifiers. If you’d like to hear how the subtleties might apply to your web project, contact us. We are willing to meet once for free, to give you our honest feedback about your idea.

 434-825-7694

contact@teamlalala.com