Closer To The Ideal

There is a reason why Symfony developers are switching en masse to using Doctrine as their ORM, and that reason is the fact that Propel is badly documented. I’ve been using Symfony/Propel for a year now, and this is the most coherent explanation I’ve yet found about creating OR statements:

It is not a mistake but how propel works. I also thought that writing code like yours I will get an OR but $c->addOr() is when you want to include a rule about the same database field. By default $c->add() will overwrite the previous rules if the first argument (the table field) is the same, thus $c->addOr will make an OR statement for this field. The same is true for $c->addAnd() but you thought it works correct because by default it uses AND between fields, but the purpose of $c->addAnd is when you define different criteria for the same field and you want them connected with and AND. Also why are you using Criterion object directly?

danah boyd gets some criticism from her academic ex-peers:

At every academic conference I attend, I hear a constant refrain: “How does it feel to have left academia?” The tone changes dependent on who is doing the asking. Sometimes, it’s pure curiosity or puzzlement, fascination at my choice. At other times, there’s a hint of condescension, as though the question is actually: “Couldn’t make it in academia, eh? Stuck in industry, eh?” I try not to bristle at this but I do find myself getting defensive and trying to explain my position at Microsoft Research over and over again.

…Well, there’s no tenure! What exactly is tenure? The promise that the university will promise you a salary in return for perpetual grant begging? Tenure guarantees a job, but it doesn’t guarantee an enjoyable one. There’s no promise of a pay raise or good classes to teach. Microsoft Research does have the right to fire me but, from what I can see, it’s more common for people to leave when they don’t gel well (just like in universities). The bigger threat is whether or not Microsoft will be around in N years (arguably, also true with many universities). I suspect that my job is just as solid as it would be in most university environments. The difference really comes down to bonuses. At the university, there are no performance-based bonuses. At Microsoft Research, a large chunk of my salary is linked to performance. Thus, I have an incentive to do well. There are also promotions that parallel university levels; Researcher = Assistant Professor, Senior Researcher = Associate Professor, Principle Researcher = Full Professor. This may not offer the on-paper guarantee of tenure, but it is pretty darn equivalent.

It’s not like you have students! Most professors love having students because of the collaboration potential. (Some enjoy the empire building but that’s not my bent.) Of course, this varies by field. Some scholars feel as though they need students to complete their work; in other fields, students are more an opportunity to mentor. My approach to students is more of collaboration and mentorship rather than slave labor. It’s true that I don’t have students, but I have the fortune of being able to take a handful of interns each year for 12 weeks each. These interns are primarily post-quals PhD students who have the skills and passion for collaboratively working on a constrained research project. No, it is not the same as 7-year students that you get to watch grow, but it’s not like I’m not engaged with younger scholars. My time with them is just more constrained and focused. There are also postdocs who come for 1-2 years. And when I’m craving collaboration, I can bring in visiting researchers to work with me. So it’s a bit more hodge-podge, but there’s still tremendous opportunities for engagement with scholars at all levels.

A comment I posted over at Hacker News:

I am concerned by the public’s changing mood, in regards to entrepreneurs. In the 1970s the American government began the process of deregulation, which allowed some important innovations in the industries that were deregulated. Transportation was probably the first big industry effected, followed by many others.

My impression was that there was a stretch when some combination of the public mood and the government’s emphasis conspired to encourage small startups. The 1980s and 1990s were clearly good in this respect.

The mood of the last decade has been increasingly punitive. An increasingly harsh attitude toward flexibility in the law has been evident elsewhere for some time, but in the last 10 years that attitude has increasingly been expressed in the field of business. The Sarbanes-Oxley Act is the most evident expression of the new trend – what once would have been treated as a civil matter is now treated as a criminal matter.

At some point during the next 30 years I expect breakthroughs in biology to allow for a new wave of startups, if the government and the public allow those startups to exist. In matters relating to health, it is worrisome to consider what a punitive legal attitude means – a medical treatment that fails to perform leads to jailtime, rather than a lawsuit.

I think it is urgent that everyone who cares about entreprenurial culture in America to make the argument, to their friends and to strangers, that innovation in business depends in part on tolerance, and that, in practical terms, this means most matters of conflict should be treated as civil rather than criminal cases.

Once upon a time, before the mid 1800s, most Western countries treated bankruptcy as a criminal matter, rather than a civil one. The liberalization of bankruptcy law was one of the factors that allowed our modern economies to gain the dynamic nature they now enjoy. I would guess that most of us know someone who has tried to do a startup, failed on their first attempt, and then later met with greater success on successive tries. Tolerance of failure is the first pre-requisite of a dynamic economy.

Since the public seems at risk of forgetting this fact, we all need to make that case, to whoever is willing to listen.

Someone responded:

When you cross the line into experimenting with medical treatments, you’re not gambling with other people’s money, you’re gambling with lives. You can’t just equate it to any other kind of start up, it has to be held to a higher standard.

I then responded:

Regarding medical innovation.

Other fields also threaten people’s lives. If you build a new kind of jet engine, which gets through testing but which then is responsible for a spectacular crash, then your product has killed a few hundred people. And yet, unless there was fraud in the documentation of the tests, there have not been criminal cases in the past. I worry that in the future there will be criminal cases. Again, the trend that I worry about is that what used to have been treated as a civil matter will, in the future, be treated as a criminal matter. We do not need to focus on the medical field. Many other fields can cause people to die – industrial automation, the transport and disposal of toxic chemicals, the construction of buildings (which could then fail and kill people). All industries are in need of innovation all of the time, yet innovation brings with it risk, including the risk of death. How much innovation will we get if we make these matters criminal?

I should emphasize, just in case people forget, that fraud has always been criminal. It has been criminal for centuries. So the move to criminalize more aspects of business is not a move to make fraud criminal. If you think that the Sarbanes-Oxley Act made fraud criminal, then you are mistaken. Fraud has always been criminal.

Sarbanes-Oxley is representative of the new trend. The overall goal was to encourage greater accuracy in the reporting of a company’s financial health. This goal could have been reached through a variety of methods, including both the carrot (rewards) and the stick (punishments). Rewards could have included tax breaks for meeting some additional level of compliance. Punishments could have included fines levied against companies that failed to meet a higher level of compliance. These approaches would not have raised the risk of jail time for CEO’s. Instead, Sarbanes-Oxley decided to go with the heaviest kind of punishment of all – to treat infractions as criminal offenses, potentially meriting jail time.

This punitive attitude is going to have a chilling effect on the amount of innovation we can expect in any field.

The venture capital model is broken:

Within the past 3 years I’m seeing a huge tsunami of innovation that has nothing to do with technology advantage (a cornerstone of the venture capital market) but instead business model or market demand oriented solutions – technology is become less of a core driver. This is turning the venture playbook upside down.

The article mentioned above talks about what I call “market factors” from a venture perspective- the liquidity market (and regulation mainly liquidity). Because of the breakdown in liquidity there are just to many company that is causing a “backup” in the system. The venture capitalists depend on liquidity to exit out the companies in their portfolio. So if the venture capitalists don’t change their model the trickle-down effect hurts entrepreneurship.

I would like to see new funding models. It would help if the various state governments (in the US) offered standard models for incorporation other than the C-form. I started a company in 2003 where we used the flexibility of the law, in regards to LLCs, to mimic certificates for my tiny company. Then I got all my friends to buy a share for $100. So that company was an LLC that in many ways imitated a C-form corporation. But myself and my business partners burned through $5,000 of legal fees to get the right agreement. I would love to see the government offer that model as a standard, a sort of hybird of C-form and LLC. The goal would be to make it easier for entrepreneurs to raise money from strangers, without having to look toward Wall Street. I realize some people will respond with concerns about fraud – those concerns are valid, but they can be gotten around by other aspects of the LLC model, especially the understanding that everyone has some of the old rights that a partner would have had in a partnership – in particular, the ability to examine the corporate accounts. We now know, from examples such as Enron and more recently with GE, that big companies, even with formal audits, can lie about their finances. The best protection against fraud is transparency.

I wouldn’t say that the model I worked out in 2003 is necessarily the one that the whole world should follow, but I do strongly believe that there are C-form/LLC hybirds out there that can facilitate early stage funding while also lowering the risk of fraud.

I’ve been reading John C. Bogle’s book, Enough. He makes the argument that we live in an era where our business leaders have forgotten what enough is. However much they have, they need more: more cars, more jets, more yachts, more homes, more money. Bogle feels that the ethical lapses we’ve seen in recent years were facilitated by the loss of the concept of enough. It is a good book. People who are interested in questions of business ethics should read it.

On a related note, today the news page over at YCombinator pointed to this old article “How Much Scratch is Enough?” written by Ryan D’Agostino.

Okay, let’s see. Say I give myself eight years—no, better make it ten. Just to be safe. I figure I’ll definitely want a great apartment in Manhattan. Near Central Park. Plus a summer house in, say, the Carolinas. Nothing too big, but nice. Also, enough to put a couple of kids through college. Prep school too. Oh, and Colorado. A condo, on the slopes. Gotta have a nice set of wheels—Beemer—and an SUV (to get around Colorado). Then maybe I’ll open up a little cafe somewhere, or get a boat. Yeah, a boat would be cool. Ten years. Figure $15 million. I think I can do it. But then I’m out. Definitely. Out for good. Just sailing around on my boat.

This is how it starts: with a pledge. A promise to yourself that you will make a certain amount of money—that you will hit your number—by a certain age, and that you will, upon reaching that carefully calculated goal, get out. Go sail your boat. Or open your bookstore or your bed and breakfast, or be a philanthropist or whatever. You won’t have to worry about money. You’ll invest a big, juicy nugget and live off the interest, which will be more than enough.

For some people, though, that word becomes a stumbling block: enough. It makes the calculations tricky, and sometimes, it changes the plan. Enough creeps slowly but steadily upward, like ivy spreading imperceptibly over an entire side of a house, and once it does you can’t picture what the house looked like before. At first, you aim high—way into the millions—and while part of you knows that chances are you won’t really end up with that much, part of you knows there’s a chance you will. You see the number in big, block numerals in your mind, and the corners of your mouth curl up into a little smile, just for a second, when you picture yourself hitting the mark.

Bogle and D’Agostino are both criticizing consumption (cars, houses, jets, etc). I’ve no problem with that. I’m critical of consumption too. But I think it is odd that they both write as if consumption is the only thing that a person might want money for.

For my part, I will never have enough money.

I have in my head an unlimited number of ideas for new businesses. Some are web-based, and of these, some are content sites and some offer a software service. But also, some of the businesses I’d like to pursue have nothing to do with the web. I’ve some software ideas to help biologists and, in particular, to help people learn biology (I’ve just been studying biology myself, so I’m aware of things that could help me learn it better). Some of these ideas are simple, such as a calculator that allows certain kinds of very easy programming (easier than Matlab). And having only recently started studying advanced math, I’m aware that people who take up math as adults, and who are self-taught, may have a set of questions that are different than what high school students ask. And I’ve various creative endeavors I’d like pursue. For instance, I’ve been writing a screenplay for a movie loosely based on the events that occurred at Enron.

The various ideas I’ve got in my head right now could keep me busy for 40 years, and I could easily burn through $200 million pursing them all. I really doubt that I’m going to succeed at all of these endeavors, and I seriously doubt I’ll ever have anything like that kind of money, but I figure I might as well just give it a try and see how far I get. And if a miracle happens, and I end up with $200 million, I’m very certain that by that time my overall goals will have expanded to the point that I’ll need a billion to fund my further ambitions.

I will never have enough money.

D’Agostino suggests that when you finally get the millions of dollars that you’ve been aiming for, you smile: “You see the number in big, block numerals in your mind, and the corners of your mouth curl up into a little smile, just for a second, when you picture yourself hitting the mark.”

I think the opposite is true: the most exciting part of launching a new business is the first dollar that you get. The first few dollars bring a huge thrill, even though the numbers are trivial:

“Our first $100 dollars!”

And then:

“Out first $1,000 dollars!”

And then:

“OMIGOD! Our first $10,000 dollars!”

After awhile the thrill starts to fade. No one celebrates when a new business reaches the $30,000 mark. I assume there is some satisfaction to reaching a $1,000,000 (never been there myself) but I have trouble imagining it is as exciting as the first few dollars that come in, those early dollars that give you your first clue that maybe you’ve a product or service that people will actually want to give you money for.

I do not need multiple cars, houses, jets or yatchs. 10 years from now, I’ll be happy if I have a small apartment in New York City, my current Volvo, which I hope to keep going despite some body rust, and $100,000,000 of software projects that are all going well.

In his fictional scenario (I assume it is fictional) D’Agostino says: “But then I’m out. Definitely. Out for good. Just sailing around on my boat.”

I can’t imagine ever wanting to get out. The thrill of launching and running a business is the most fun thing I’ve yet discovered. If I got out, what would I do? I’d simply get back in.

Brad Delong asks “Why does Berkeley think it should be in the business of building its own content management systems anyway?“.

His complaint:

May I say that a content management system that–if you have been off dealing with another crisis in the middle of a task–decides when you come back and try to save your work that you are no longer logged in and dumps you to a login page after which it dumps you not on the page you were working on but on the root page, LOSING YOUR WORK!!!1!!

Such a content management system is HELLSPAWN!! Is WROSE THAN HILTER!1!!!1!…

He is complaining about bspace, which is based on the open-source
Sakai, a content management system written in Java, and focused on the needs of universities.

I think Delong’s post is a good reminder of how infuriating it can be for users when software fails to behave in the ways users expect. I also suspect this is a good example of an issue that users will regard as a bug, but the programmers will see it simply as a potential future feature that they may or may not add (”Should we catch POST info if a user is not logged in?”).

I should add, WordPress has exactly the same problem. Last week I started writing what I thought would be a short post for this blog, but I got carried away by my theme and wrote a long post. Then I went to get some dinner. I left the browser open, with the post unpublished. I came back after dinner and made some more edits, then hit the publish button – and just like that, my work vanished, because while I was out getting dinner, WordPress logged me out (for some reason I’d assumed that the auto-save feature was also refreshing my session info).

One of the nice things about building my own CMS was that I was free to fix the bugs that bothered me most, and this was a big one for me. I added a feature to my CMS that caught any POST info and showed it on screen, even if the person was logged out. This allowed recovery of the post. I worked on my CMS from 2002-2007 and then abandoned it because I could not keep up with projects like WordPress. Nowadays I force myself to use other people’s open source software, because it is economically rational to do so, but I hate some of the choices they make, and some of the features that they fail to implement.

In the comments, Jacob Davies posts this comment, which I thought was very funny and very on point:

Conversation that has happened more times in my career than I care to mention:

Someone else: “How long of a title shall we allow? 32 characters? 64?”

Me: “FOR THE LOVE OF GOD WHY DO WE NEED TO SET A MAXIMUM LENGTH? IS THIS 1952???”

Someone else: “But what if they put in a really long title and fill up the database?”

Me: “THE VERY NEXT FIELD – THE ‘CONTENTS’ FIELD – IS A FREE-TEXT FIELD WITHOUT A LENGTH CONSTRAINT SO IF THEY WANTED TO FILL THE DATABASE THEY COULD DO IT THERE ANYWAY.”

Someone else: “Won’t it waste space if we allow a variable-length string in the title?”

Me: “OH MY GOD YES A TERRIFYING LOSS OF ABOUT 3 BYTES ON A RECORD THAT IS A MINIMUM OF 1024 BYTES LONG AND OFTEN OVER A MEGABYTE, YOU ARE SO RIGHT.”

Someone else: “Yes but every other system has a length constraint for titles.”

Me: “YES AND I SUPPOSE IF EVERYONE ELSE WAS JUMPING OFF A BRIDGE YOU’D DO IT TOO.”

etc

Computer programmers are subject to some kind of strange mental degeneration in which they rate the potential waste of 0.00001% of the capacity of a modern hard disk as more important than the ability to enter titles longer than 32 characters in length.

This summer I worked on a project where some other programmer had used the Context object in the model classes. This caused ridiculous problems. For instance, to run the command line tools (such as “symfony propel:build-model”) I had to hard-code the loading of a context object into the core classes – what an ugly hack!

So I was intrigued to read of the problems that the Context object can cause in your form classes, and how dependency injection gives us an easy way to avoid needing the Context object.

You might read that and think “woah, this is getting complicated!” but we’re not talking about dependency injection containers here, we’re simply saying that you can make your form object depend on something to run. The thing it depends on should not be the context singleton, it should be the minimum thing that the form needs to operate correctly – which in this case, is a user object that supports credentials.

…The test here is where we are making this form “dependent” on a user object. In this case we are insisting that the object is an instance of sfUser, which you may argue is tying us in to symfony again, but you could use any test here to ensure that the object will have the necessary functionality you need, maybe check for the existence of a “hasCredential()” method for example.

When writing a test for this form class, we now only need to instantiate a user object and load it with some credentials – much easier than doing the same thing and locking into a context singleton. There may be other times when this form could be useful in a lightweight environment, where you can get speedy access to a user object but don’t want the overhead of the symfony context – you might not think of one now, but it’s best to code this way and you’ll have less reasons to kick yourself further down the line.

How to redirect to original destination after login in Symfony. I recall this information was way too hard to find when I first went looking for it:

on first hit to your login action, store referer to the user session:

if(!$this->getUser()->hasParameter(’referer’))
{
$this->getUser()->setParameter(’referer’,$this->getRequest()->getReferer();
}

and then when login succedes, redirect user to stored referer with:

$this->redirect($this->getUser()->getParameter(’referer’);

Anyone alive today grew up in the era of Heisenberg’s uncertainty principle, Max Plank and quantum mechanics. As such, it seems natural to us to think of probability when we think of physics. Einstein may have said “I can not believe that God is dicing with the Universe” but dicing is the view dominates physics today. We are taught these ideas in school. Because of this, we have a hard time remembering what a surprise probability was to scientists working at the end of the 1800s. By that point, Western science had spent 200 years working with Newton’s view of a mechanical universe, and physics was seen as the most pure of the sciences, and therefore the most mechanical, the most free of randomness. Who would have guessed, in 1880, that physics was about to transform from the science most free of probability to the one that depended on it in its most fundamental theories?

Consider this bit of history from “The Human Use Of Human Beings: Cybernetics And Society” written by by Norbert Wiener in 1950.

The beginning of the 20th century marked more than the end of a 100 year period and the start of another. There was a real change in point of view even before we made the political transition from the century on the whole dominated by peace, to the half century of war through which we have just been living. This was perhaps first apparent in science, although it is quite possible that whatever has affected science lead independently to the marked break which we find between the arts and literature of the 19th and those of the 20th centuries.

Newtonian physics, which had ruled from the end of the 17th century to the end of the 19th with scarcely an opposing voice, described a universe in which everything happened precisely according to law, a compact, tightly organized universe in which the whole future depends strictly upon the whole past. Such a picture can never be either fully justified or fully rejected experimentally and belongs in large measure to a conception of the world which is supplementary to experiment but in some ways more universal than anything that can be experimentally verified. We can never test by our imperfect experiments whether one set of physical laws or another can be verified down to the last decimal. The Newtonian view, however, was compelled to state and formulate physics as if it were, in fact, subject to such laws. This is now no longer the dominated attitude of physics, and the men who contributed most to its downfall were Bolzmann in Germany and Gibbs in the United States.

These two physicists undertook a radical application of an exciting, new idea. Perhaps the use of statistics in physics which, in large measure, they introduced was not completely new, for Maxwell and others had considered worlds of a very large number of particles which necessarily had to be treated statistically. But what Bolzmann and Gibbs did was to introduce statistics into physics in a much more thoroughgoing way, so that the statistical approach was valid not merely for systems of enormous complexity, but even for systems as simple as the single particle in a field of force.

Statistics is the science of distribution, and the distribution contemplated by these modern scientists was not concerned with large numbers of similar particles, but with the various positions and velocities from which a physical system might start. In other words, under the Newtonian system the same physical laws apply to a variety of systems starting from a variety of positions and with a variety of momenta. The new statisticians put this point of view in a fresh light. They retained indeed the principle according to which certain systems may be distinguished from other by their total energy, but they rejected the supposition according to which systems with the same total energy maybe clearly distinguished indefinitely and described forever by fixed causal laws.

There was, actually, an important statistical reservation implicit in Newton’s work, though the 18th century, which lived by Newton, ignored it. No physical measurements are ever precise; and what we have to say about a machine or other dynamic system really concerns not what we must expect when the initial positions and momenta are given with perfect accuracy (which never occurs), but what we are to expect when they are given with attainable accuracy. This merely means that we know, not the complete initial conditions, but something about their distribution. The functional part of physics, in other words, cannot escape considering uncertainty and the contingency of events. It was the merit of Gibbs to show for the first time a clean-cut scientific method for taking this contingency into consideration.

I’m reading Ian Stewart’s book, Letters To A Young Mathematician. The book is written as a series of a letters to a fictional “Meg”, who I think is a fictional niece. As Stewart says:

The letters, addressed to “Meg”, follow her career in roughly chronological order, from high school through to a tenured position in a university. They discuss a variety of topics, ranging from basic career decisions to the working philosophy of professional mathematicians and the nature of their subject.

Some ideas were new to me, such as the fact that math can prove that something is unprovable. For instance, there is no reliable method to trisect an angle, and there never will be. Math is unique among the sciences in that is able to know which things are impossible.

This bit about gender in math was especially interesting to me:

Mathematicians are proud to trace their academic lineage through their thesis advisers. Brian was my mathematical father, and Phillip Hall my grandfather.

…Talent must be passed to succeeding generations. I’ve been a thesis adviser to 30 students so far, 20 men and 10 women. Since 1985, the proportions are 50% men and 50% women. I know women are just as good at math as men because I’ve watched both at close quarters. I am particularly proud of my mathematical daughters, most of whom hail from Portugal, where mathematics has long been viewed as a suitable activity for women. All of my Portuguese daughters have remained in mathematics. In fact, most of my graduate students have remained in mathematics, and every single one of them earned a PhD. However, one is now an accountant, several work in computing, and one owns an electronics company, or at least he did last time I heard from him.

The rest of the world is now following Portugals’s lead. In July 2005 the American Mathematical Society released the results of its 2004 Annual Survey Of The Mathematical Sciences. Since the early 1990s, women have been receiving around 45% of all first degrees in math. Women received almost one-third of all US doctorates in the mathematical sciences in the academic year 2003-2004, and one-quarter of those awarded in the top 48 math departments. In all, 333 women received math PhDs that year, the largest number ever recorded.

The percent of computer scientists who are female and who are earning advanced degrees peaked in 1989 and has since retreated. As math is foundational to computer science, this is revealing. Computer science is nearly alone among the professions in seeing women’s participation retreat during the last 20 years. Women have made huge strides in catching up to men in business, law and medicine. Almost 50% of all new doctors are female.

I’d been assuming that women’s retreat from computer science had to be mirrored by weakness in math, but if women are making huge strides in math, then the anomaly of their retreat from computer science is even more puzzling than before.

I’ve worked with entrepreneurs who are obsessed with secrecy about their idea. Some of them ask me to sign non-disclosure agreements, just for hearing the basic idea of their business. This kind of secrecy is always a mistake. If you’ve got an idea for a business, you should talk about that idea with everyone.

A frequent question entrepreneurs have when they are just starting their company is: how secretive should I be about my idea? My answer: you should talk about it to almost anyone who will listen. This includes investors, entrepreneurs, people who work in similar areas, friends, people on the street, the bartender, etc.

There are lots of benefits to talking to people. You’ll get suggestions for improvements. You’ll discover flaws and hopefully correct them. You’ll learn a lot more about the sector/industry. You’ll learn about competitive products that exist or are being built. You’ll gauge people’s excitement level for the product and for various features. You’ll refine your sales and investor pitch. You might even discover your idea is a bad idea and save yourself years of hitting your head against the wall.

TRON was developed in Japan, and it is used in more machines than any other operating system in the world. I don’t know much about embedded technologies, so this was news to me. I have long been puzzled why Japan seems to dominate in so many tech sectors, yet seems so far behind when it comes to operating systems. This articles gives me a piece of the puzzle that helps make the whole worldwide picture more clear – Japan has the most used embedded operating system. Given Japan’s strengths, this makes a lot of sense.

What distinguishes TRON from other operating systems like Windows is the extremely high speed at which data is processed and the fact that the source code is freely available. These are just two of the reasons why TRON is installed in most Japanese-made cellphones and digital cameras, as well as in many other electronic products, such as audio-visual devices, rice cookers, air conditioners, fax machines, karaoke machines, and car-engine control devices. The mini-computers fitted to these devices are called embedded chips, and about 5.3 billion of them are produced worldwide each year. It is thought that about half of these run on TRON. Since only about 150 million PCs are shipped each year, these figures mean that TRON is used in far more machines than the overwhelming leader in the personal computing market, Microsoft’s Windows OS. In fact, in terms of machine numbers TRON is the most widely used OS in the world.

Recently it is TRON’s open-source status that has been attracting attention. Microsoft has never revealed the source code – the blueprint – for Windows, insisting that it is a corporate secret, and has refused to allow others to adapt its OS as they wish. For this reason, the inner workings of Windows have remained a mystery, and some public institutions and corporations have complained at not knowing what is inside the OS. The lack of transparency even led the Chinese government and some public organs in Europe to worry that state secrets might be vulnerable to leaks when held on Windows machines. Some of them have therefore started to switch to the open-source Linux OS developed by Finnish programmer Linus Torvalds instead.

TRON Set for Use in Info-Appliances
In Japan, the E-Life Strategy Research Group, an independent group formed by the Ministry of Economy, Trade, and Industry’s Commerce and Information Policy Bureau and whose members include the president of a major electronics maker and university professors, agreed in March that Linux or TRON should be used in info-appliances (electrical appliances that are connected to the Internet). METI and Japanese electronics makers are keen to move away from Windows and to nurture a homegrown OS in order to secure a leading position for Japanese firms in the info-appliance market. Info-appliances bring to mind images of such products as the air conditioner that switches itself on as you come home from work and the fridge that orders a delivery of beer from the liquor store when your stock runs low.

Interesting article about making a lot of money with Amazon’s affiliate program:

I have earned $119,725.45 from Amazon Associates Program since I began using it as a way to make money online late in 2003. Around half of that amount was made within the last 12 months.

In this post I want to share what I’ve learned along the way on how to make money with Amazon.

While Amazon’s Associates program is not my largest income stream (I rank how I make money blogging here) it was actually the first experiment that I did with monetizing blogs. I began to experiment with it in the last quarter of 2003 (just before I started using AdSense).

I started using it on a personal blog that had been going for around 12 months and had around a thousand readers a day – the first quarter was not spectacular in terms of earnings – I made $31.80 (around 30 cents a day) and almost gave it away.

Paul Graham offers advice about how to keep your startup from dying:

The number one thing not to do is other things. If you find yourself saying a sentence that ends with “but we’re going to keep working on the startup,” you are in big trouble. Bob’s going to grad school, but we’re going to keep working on the startup. We’re moving back to Minnesota, but we’re going to keep working on the startup. We’re taking on some consulting projects, but we’re going to keep working on the startup. You may as well just translate these to “we’re giving up on the startup, but we’re not willing to admit that to ourselves,” because that’s what it means most of the time. A startup is so hard that working on it can’t be preceded by “but.”

In particular, don’t go to graduate school, and don’t start other projects. Distraction is fatal to startups. Going to (or back to) school is a huge predictor of death because in addition to the distraction it gives you something to say you’re doing. If you’re only doing a startup, then if the startup fails, you fail. If you’re in grad school and your startup fails, you can say later “Oh yeah, we had this startup on the side when I was in grad school, but it didn’t go anywhere.”

You can’t use euphemisms like “didn’t go anywhere” for something that’s your only occupation. People won’t let you.

One of the most interesting things we’ve discovered from working on Y Combinator is that founders are more motivated by the fear of looking bad than by the hope of getting millions of dollars. So if you want to get millions of dollars, put yourself in a position where failure will be public and humiliating.

Zed Shaw is moving to San Francisco. He wrote the following comparison of entrepreneurs in San Francisco and New York City:

To give a great example of what it’s like in NYC, I received this email from someone when I posted I was looking for work in SF

I have an idea for a business involving level 2 stock quote analysis that will be able to ask arbitrary questions of the NYSE data stream using indexed log files in Hadoop as well as a backup idea to cannibalize the enterprise search market by doing what they do for cheaper money using this technology and lucene. In both cases, I’ll need a badass coder to help me with the application and deployment parts, ideally using a dynamic JVM language. … I have personal funds to work for 6 months so I’m planning on doing this for 2-3, presenting a prototype to some financial people and ideally getting funding and connections to sell it in one fell swoop. Just registering the idea. Real-time analysis of a whole bunch of email could be an option as well, if you had a business idea.

Yes, I actually did receive this in response to my blog post. This is actually a very common pitch and style for the NYC entrepreneur. This cocky, muscular, alpha-male-wannabe “my dick is bigger than yours come work for me and make me rich” is tiring and the main reason I’m leaving. If NYC entrepreneurs want to know why they can’t find “techies” (another way of saying “fucking nerd”), all you have to do is read the above.

By comparison the companies that contacted me from San Francisco were all incredibly polite, professional, and told me exactly why I should work for them. Each one immediately pointed me to their existing technology. Each one told me their funding levels. Each one had their business model thought out to some degree. There wasn’t a single company that I thought didn’t get the business, or was trying to screw me.

When it comes down to it, the primary difference between the companies who contacted me in SF vs. NYC is this:

* NYC prospects were looking for a badass/ninja/rockstar beta-male “techie” employee to make them rich creating lame applications for giant Finance/Fashion/Marketing companies.

* SF prospects were looking for a partner to get rich with them creating great products for customers.

Now obviously the SF companies are all really looking for an employee to make them rich. The difference is in their approach and how they don’t say it so blatantly or in such a gigantic douchebag way. Their touch was lighter, more collaborative, and definitely not with the attitude of me being some “beta-male” to their “alpha”. That lack of cocky dickwad douchebag in the SF technology scene will be refreshing and interesting.

There is some truth to his account of entrepreneurs in New York City, though I also ran into a lot of the same attitude in Virginia. I’ve run into a lot of clients who have the attitude “I am a genius, I just need some coder to bring my ideas to life.” Invitations to real collaboration are rare. My sense is that a lot of the entrepreneurs that I’ve dealt with on the east coast are at the unconscious incompetence level of learning. They simply do not know how little they know, nor do they know enough to appreciate how much I know.

A bestiary of algorithmic trading strategies

One of the things which confronted me when I got interested in quantitative finance is the varieties of different kinds of quant. Now I realize this is pretty simple. Quants come in three basic varieties.

1. Structurers: people who price complex financial instruments.
2. Risk managers people who manage portfolio risk
3. Quant traders people who use statistics to make money by buying and selling

It took me quite a while to figure this out. I don’t know why people haven’t bothered to state this taxonomy of quant jobs. I suspect it’s because most quants are structurers. Of course, there is often bleed over between these varieties -but it’s a useful taxonomy for looking for work. I’ve done a little of all three at this point (very little, honestly), and have always liked quant trading problems more than the other two varieties. It’s the most ambitious, and the most likely to net you a career outside of a large organization (go me: Army of one!). It’s also the most mysterious, since successful quant traders don’t like to talk about what they do. Structurers and risk managers have to talk about what they do, almost by definition. Quant traders gain little from talking about their special sauce. The ones who have spilled the beans are guys like Ed Thorp -who only talk about old strategies, or guys like Larry Harris, who wrote the best book there is on trading, though he wrote it without any interesting equations in it. Of course, there are going to be quant jobs which don’t fit exactly into these categories; there is a lot of overlap between traders and risk managers, for example: I’m only presenting them as a useful framework to hang some thoughts on.

Very interesting thread about some of the limits of the Amazon Elastic Load Balancer system:

We have tried to use ELB in a live ad-serving application for the last two weeks, and unfortunately, we are going to have to revert to a colocation setup because the reliability has been unacceptable.

Our setup was two EC2 instances behind an Amazon load balancer. The system served 45-50 million requests a day scattered around the world. The traffic pattern was “real world,” as you would expect for a live round-the-clock application.

We were seeing a lot of 503 errors in our test clients after initial launch and setup a heartbeat monitor that sent a test request every second. After running for a week, we found that a full 1% of our heartbeat test requests received 503 errors. Our monitoring on the EC2 instances behind the load balancer shows that they never returned any errors and never achieved high levels of load either. All of our investigations pointed to the conclustion that the failures were completely attributable to the load balancer.

Not Even Wrong asks an interesting, though misguided, question:

The problem Lee addresses seems to me to be an extremely important one. Pretty much every knowledgeable particle theorist that I talk to these days, string theorist and non-string theorist, agrees that current ideas about how to go beyond the standard model are not working very well. Everyone hopes that some big new idea will come along and show the way forward, with people often wistfully speaking about how maybe some bright post-doc out there may be at this very moment working on the needed new idea. The problem with this is that what is needed is probably something quite different than any of the current popular research programs, and finding it may be difficult enough to require someone’s concerted effort over quite a few years. If this is so, it’s very hard to see how anyone on the standard career path in the US is going to be able to do this. A young post-doc here generally only has a couple years in between needing to apply for new jobs, and if he or she were to devote those years to working hard on a very speculative new idea, this would most likely be suicidal for their career.

Yes, why can’t the current generation of post-docs take big risks, just like Einstein? Oh wait, Einstein was never a post-doc. He worked at the patent office. He found that school stifled his creativity:

His father intended for him to pursue electrical engineering, but Einstein clashed with authorities and resented the school’s regimen and teaching method. He later wrote that the spirit of learning and creative thought were lost in strict rote learning. In the spring of 1895, he withdrew to join his family in Pavia, convincing the school to let him go by using a doctor’s note.

More so, Thomas Kuhn has argued that most of the really big paradigm shifting breakthroughs in science are caused by someone coming into a field from the outside.

This is the smartest sentence in the post:

A young theorist would be more likely to be able to find the necessary time if he or she went to work as a night-time security guard.

Here is a law of science: post-docs work on normal science (what Thomas Kuhn called normal science – the science that does not replace the currently dominant theory with a new one). They don’t work on new paradigms.

The only way to get more paradigm shifting breakthroughs is to provide people (every citizen in society) with a comfortable existence, so that they are not destroying their life when they decide to pursue something all together outside of the mainstream. In other words, we need more jobs like what Einstein had at the patent office – jobs that provide people with a safe existence, and with time to think. With more such jobs, we would have more big breakthroughs (whether society needs more big breakthroughs is a separate subject).

Git handles merges better than Subversion:

Subversion has worked for us up until now on the project, but it is falling short of meeting our process and productivity needs. Without effective merging, branches won’t be used no matter how cheap they are to create. The only Subversion branches we have ever created have rarely stayed in sync with upstream and always led to dramatic merging ceremonies, worries about regression and code loss. To make a long story short, Subversion supports parallel development, but makes it so difficult to use effectively that you are better off avoiding it altogether.

Git handles our roughly 500,000-line Enterprise Java code base with ease. It is faster for one developer to git svn clone and then have his peers git clone than it is for each developer to checkout simultaneously from Subversion. Once done it gives every developer on the team offline access to the entire project history. The offline access allows developers to do complex historical comparisons that would be slow enough in Subversion to discourage developers from doing them. Git not only makes certain reporting operations fast, it makes them practical and productive to use.

People who attempt suicide, and survive, see big gains in their income:

Last year Dave Marcotte, a professor of public policy at the University of Maryland, Baltimore County, pushed the field forward when he wondered what happens to people like Jones who attempt, but do not achieve, suicide.* There are about 20 attempts for every successful suicide. (Approximately 2.9 percent of the U.S. population has attempted suicide—1,760 attempts per day.)

…Marcotte couldn’t test the relative “life improvement” of successful suicides—since they were, of course, dead—but he could study those who had failed at suicide to determine if their lives improved after the attempt. The results are surprising. Marcotte’s study found that after people attempt suicide and fail, their incomes increase by an average of 20.6 percent compared to peers who seriously contemplate suicide but never make an attempt. In fact, the more serious the attempt, the larger the boost—”hard-suicide” attempts, in which luck is the only reason the attempts fail, are associated with a 36.3 percent increase in income. (The presence of nonattempters as a control group suggests the suicide effort is the root cause of the boost.)

Wikipedia’s growth is slowing. But why?

Still, there remain unanswered questions. Could its growth ever halt completely? How big will the site be when the editors decide that the sum of human knowledge is catalogued? Could a new website take Wikipedia’s place by toeing an inclusionist line?

Parc’s research doesn’t give any answers, but Chi has identified one model that Wikipedia’s growth pattern matches. “In my experience, the only thing we’ve seen these growth patterns [in] before is in population growth studies – where there’s some sort of resource constraint that results in this model.” The site, he suggests, is becoming like a community where resources have started to run out. “As you run out of food, people start competing for that food, and that results in a slowdown in population growth and means that the stronger, more well-adapted part of the population starts to have more power.”

This is a great article about managing UI complexity:

Progressive disclosure is the most popular means of managing complexity. The idea is that clutter and cognitive overhead can be reduced by hiding less frequently used elements behind some avenue of accessing those elements, like a mouse click or a keyboard shortcut. It requires that the designer accurately determine which elements are frequently and infrequently used and to what degree.

Quite a bit of care needs to be put into the progressive disclosure hierarchy and the mechanisms used for disclosure. Poorly considered use of progressive disclosure can achieve the opposite of the intended effect by making the interface even more complex. As an example, Microsoft Windows has been trending towards removing the menu bar from individual windows and instead packing each function into the main interface (often using pull down menus), which has some issues.

Sacha Chua describes herself as a shy connector, and offers some suggestions:

[I] Hate going out

[I am] Comfortable with hosting people

I sometimes feel overwhelmed in places people like going to “hang out”, such as busy restaurants and bars. I prefer to host small get-togethers at home, where I can keep group numbers low and I’m in familiar territory.

An article about setting up private/public key pairs so you can ssh to your instances on Amazon EC2.

Also, the Scalr service looks interesting. I have the impression they are putting a much better interface over AWS.

I recently oversaw a project that had me working with a distributed team (in the US, Europe and India). The same project also worked exclusively with independent contractors, whose work was hired on an assignment basis. The work was for a new web-startup. I have some concerns about such an arrangement being used for a new, innovative project. I’m going to attempt to explain my concerns in this essay.

First, I’ll talk about using freelance, independent contractors for a web start-up. The seeming advantage of having no permanent employees is that the operation has no fixed costs. An advantage? It depends. If you really want to avoid all fixed costs, simply close your business. Being in business means having expenses, the real issue is whether you are making a profit – does the labor you hire help you make more money than the cost of the labor?

The project I oversaw moved slowly, and some of the slowness can be attributed to the freelance nature of the labor we hired. The programmers would do whatever we asked them to, and they all did high quality work. However, because we made no long term commitment to them, they of course had to keep busy with other work. Sometimes we needed them but they were busy with their other clients. Also, they were not part of the conversations we were having in New York City, so they lacked a broad overview of the project and its goals, so they could not easily take the initiative to move ahead on their own.

If a project has no particular deadline, or if it is simple, then working with independent contractors is a great way to keep fixed costs low. However, if a project is suppose to move fast, or if it needs to achieve certain deadlines, and especially if it is very complex, then I would recommend against the use of this kind of arrangement. For fast moving, complex projects, you need full-time, committed programmers.

Now I’ll offer some thoughts about agility, and the importance of physical proximity.

Agile practices have 2 main hopes:

1.) to allow a fast-forming team to quickly become highly-productive.

2.) to control costs by ensuring discipline in regards to goals and deadlines.

I used the word “hopes” instead of “goals” because the ultimate aim of agile practices is more of an ideal to be aspired to rather than a reality that can be achieved, and by that, all I mean is that we are constantly learning more about what these practices truly mean, and in the future we can reasonably hope to do better than we are doing today. I’m speaking for the entire profession of computer programming when I say this.

I am in the United States, in New York City. I’ve been working on websites since 1996. I’ve worked on many small teams, and over the years I’ve used some agile practices, such as frequent iterations, frequent in-person consultations with the client, and unit tests. When I started on this last project, I first thought it would offer the chance to experiment with a fuller range of agile practices. However, I came to realize that most agile practices depend on physical proximity. The programmers need to be able to meet, preferably every day (”every day” in the sense of casual conversation, not “every day” in the sense of “let’s have a formal meeting”). Consider some of these practices:

1.) pair programming (two people looking at the same screen, checking for errors)

2.) the use of index cards to map objects

3.) stand up meetings (meetings kept short and lively because everyone is standing)

None of these make any sense if the programmers are in different countries.

I’d also add code reviews. One can do a code review at a distance, but I think in person is much more effective. And code reviews don’t make much sense when you are working with independent contractors. Code reviews are meant to develop each programmer’s long-term abilities, and also they are useful for helping the whole team agree on certain programming habits. But one would have to have a long-term relationship with the programmers before doing such a thing would make sense.

Programming is an art, and art needs passion. Many agile practices are aimed at soliciting a greater emotional commitment from the programmers. I might write “For an agile project to go well, the programmers need to emotionally commit to it” but that isn’t quite right. What would be more correct would be “For an agile project to go well, the programmers need to emotionally commit to their own professional ideal.” That is, they need to care about the craft of programming. They need to be constantly pushing themselves to always be better programmers.

I would never argue that agile practices are needed on every project. If a project is routine, then long-distance, non-agile practices might be perfect. For instance, suppose you decide to launch a new online magazine. After researching your options, you decide you will use WordPress as the software that powers your site. I’d suggest you find a good designer, local to you, and work with them to come up with the design. Once you have the design, the rest of the process is mundane. You could certainly hire a team in India to install WordPress and implement your design for you. And you would not need agile practices for such a project. In fact, you do not even need great programmers for such a task. Mediocre programmers can do a reasonable job with tasks that are standard, routine and mundane.

However, I would argue that agile practices are needed at projects that need to be fast moving (this would include most start-ups), or at any project where the aim is to create something altogether new and unique. If your project is venturing out into the great unknown, if you are going where no one has ever gone before, you will need a team of excellent, committed adventurers to go along with you.

Let’s look at the history of agile practices, going back centuries.

In the 1990s, DARPA invested some money in agile research, which lead to such books as that by Goranson: Agile Virtual Enterprises.

Goranson looks back at the whaling industry of the mid-1800s and examines how 2 small towns in Massachusetts were able to capture 90% of the whaling industry, when most of the actual hunting was done in the South Pacific. He found that the patterns of hiring and team building had much in common with what start-ups aspire to nowadays – a tight-knit, fast-forming team of highly competent professionals held together by a commonly understood body of professional ethics and expectations.

Goranson emphasizes the role of geographical concentration (and therefore physical proximity), to the evolution of culturally unique traits, leading to competitive advantages:

Geographical and professional isolation allowed for the evolution of a unique and very effective system which for many years put a brand new, fairly high risk/high payoff, virtual enterprise in the water at the rate of one every two weeks. The return of each whaling expedition triggered the formation of another. It was considered an invitation to bad luck to reuse the same combination of partners, so during the six to nine months it took to recondition the ship for another voyage, the owner of the boat assembled a new group of key players who would join him in setting up the basic physical and social conditions needed for a successful venture.

The primary partners required to launch a voyage consisted of a ship owner, an insurer (of the ship cargo), a provisioning financier to supply the expedition with food and other consumables, a captain, and often a manufacturer who agreed to buy the oil at a set price. This component of the partnership was formed in the first couple of months, the partners being determined partially by availability.

A month or less before the ship was ready to sail, a secondary group of partners, the crew, was formed. They shared a distinct cultural background; almost none of them had, or would ever, serve in the navy or the merchant marine. This professional distinctiveness, coupled with an intense geographical concentration, fostered the development of a unique culture based on the virtual enterprise.

Goranson uses the word “agility” to refer to fast forming supply chains, and also (relatedly) the ability to overcome the administrative costs that might keep a new supply chain from forming (in the early 90s, when DARPA funded Goranson’s work, one problem seemed especially prevalent: the fact that the most innovative technologies in America were being developed at small start-ups, and yet large Fortune 500 companies were having difficulty integrating small start-ups into their supply chains, because of the administrative hassles of developing relationships with a multitude of small companies). He uses the word “virtual” to refer to an organization that is project-specific and not meant to last. For instance, in the modern context, the best examples of agile, virtual enterprises are in the movie industry. Each movie is its own independent business, and the production of each movie entails the fast formation of a team of highly competent professionals who understand they are being hired to work on a specific, time-limited project, which will disband as soon as the project is done.

Whaling ships and modern web start-ups share certain traits::

Every voyage included a team of skilled craftsmen – carpenter, blacksmith, cooper, and a sailmaker (often a boatwright, a rigger and a cook were also in this class) – whose combined expertise allowed the enterprise to respond effectively to a broad range of situations. Each of these professionals, along with the tools, supplies, and sometimes apprentices they brought abroad, formed an essentially self-contained business which was integrated into the enterprise as a whole. In these cases, it was not just the person who signed up for the voyage, but their business. From the shipowner to the cook, everyone was paid with a pre-arranged percentage of the take.

In other words, every ship was its own little agile, fast-forming start-up.

Goranson’s use of the word “agility” is somewhat different than the way the word “agility” is used by its proponents in the software industry, but the meaning of the word overlaps for both groups in the sense of “the ability to get supply from a new source quickly”. Consider the words of the Agility Manifesto:

We have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

With only minor tweaks of the wording, you could use the above as a statement of the professional culture that pervades Hollywood. The emphasis is on doing what is necessary to get something working into production fast. By contrast, the slow moving process of low trust, heavily documented, laborious negotiated, carefully vetted contracts are anathema in this context.

(Stephen M. R. Covey has a written a book called The Speed Of Trust, which does an excellent job explaining how much a project speeds up when the participants can trust each other, and therefore how important it is to work toward facilitating the long-term development of trust in any business relationship.)

Goranson emphasized that one of the legal traditions that the whaling industry and movie industry share is a very strong reliance on verbal contracts, as opposed to written contracts and detailed documentation. Goranson points to the bankruptcy of Kim Bassinger as an outstanding example of the importance given to verbal contracts in the movie industry. (Bassinger met some producers for dinner to discuss the possibility of her starring in their movie Boxing Helena. She promised, over dinner, to star in the movie. When she later backed out of her promise, they were able to sue her for $8 million.) The Agility Manifesto de-emphasizes contracts, but still puts a heavy emphasis on the importance of verbal communication, a point made clear by “Customer collaboration over contract negotiation”.

Goranson emphasizes the role that geographical concentration played in allowing this culture of virtual enterprises to flourish. I believe that geographic concentration still plays the predominate role in allowing the continued evolution of unique agile practices.

I’ve had it suggested to me that the cheapness of overseas programmers (in places like India) can speed a project up, simply because, since they are cheap, you can hire a lot more of them. This is flatly contradicted by Fred Brooks thesis in The Mythical Man-Month. When the System 360 project was running late at IBM, Brooks tried to speed up the project by adding more programmers to the team. But he found that the additional programmers actually slowed the project down. Each new programmer became one more person who needed to be kept up to date with the latest documentation – and every few additional programmers required that a new manager be brought into the project to manage them. The growing complexity of communication across a growing mass of workers helped to slow the project down, rather than speed it up.

In this sense, the cheapness of overseas programmers actually encourages bad habits. Because they are cheap, you may get tricked into thinking that you can hire a lot of them. But the more you hire, the more time you will need to invest trying to keep everyone unified in their understanding of what the code is suppose to do. And after a certain number, you’ll need to start hiring additional managers.

If you are a project manager, and you rely on the cheapness of your programmers to keep a project’s costs under control, then you need to fire yourself. There is only one correct way to keep costs under control, and that is through disciplined focus on goals and deadlines. Agile practices have evolved to make it easier to keep complex projects on track. To the extent that geography matters to agile practices, then geography matters for the goals and deadlines of your project.

Nicole Radziwill approaches this subject from a different angle, but she makes a similar point about the importance of geography:

Space does matter. We know this when we are designing facilities and plant layouts, for example, because one of our common considerations is to minimize traffic between areas and departments. More often than not, we do this to minimize the time spent moving people or equipment around a plant, so that time is not wasted. But the same concept could apply to our supply chains. Why aren’t we minimizing the time that components or goods spend traveling through the supply chain, when it could lead to reductions in energy costs? Furthermore, why aren’t we shortening our supply chains to strengthen local and regional businesses, and train the next generation of skilled workers (who can actually do something useful for the regional economy)?

The logic has been something like this: energy is cheap, therefore transportation is cheap, and transportation is easily available and accessible through third-party providers like FedEx and UPS. But I can’t shake the feeling that “supply chain status quo” is not good for quality in the long-term – because it encourages us to source the products and components that are most affordable, rather than the ones that might help us cultivate a quality consciousness in our local areas.

To offer a partial answer to the question that Nicole poses, I think the reason why products get sourced to where they are most affordable (at the current moment, in the short term) is that people tend to fear losses more than they value potential gains. There is an abundant literature on this subject. In regards to sourcing, buyers go with what is cheapest in the short-term, rather than developing long-term innovations that might lead to much greater gains in the future. Consider:

Prospect theory… argues that people center their expectations around a variable and moveable reference point. The reference point is most easily thought of as the status quo, though it is not always so. In most situations, people are more likely to ensure that they preserve their current situation, than to risk losses by seeking potential improvements. Indeed, people tend to sacrifice a great deal in order to maintain their current situation. As Levy notes, “people tend to be risk-averse in choices among gains, but risk acceptant with respect to losses.” This is often the problem in intractable conflicts; the risks required to resolve the conflict seem much too great compared to the potential losses. This sort of framing is what leads ultimately to hurting stalemates. Loss-oriented framing is, ultimately, this tendency to minimize losses while missing opportunities to improve one’s situation.

The status quo of any culture, in the short-term, can be regarded as an under-developed blandness, relative to the unknown brilliance of some unknown future innovation. It takes a leap of faith to believe in that unknown future brilliance. It is easier, and safer, to go with what is cheapest in the short-term. This is cowardice, but it is extremely common. True business leadership is rare.

In terms of developing some unknown future brilliance, I’m thinking especially of those cultural traits that, in part, give rise to comparative advantage. There is, at this point, a mountain of literature about the advantages that agile teams have when it comes to fast-forming, fast-moving projects. Agility is a cultural innovation that deserves further refinement, and each nation, depending on its legal traditions and its labor laws, is going to develop agility in a unique direction. The future development of these potential innovations is undermined by the extended supply chains that send work to distant parts of the world. I’m arguing that, just as Goranson stressed geographical isolation as playing a role in the evolution of a uniquely competitive culture in Massachusetts, then the dispersal of work from a geographic center must lead to the opposite – the dissolution of uniquely competitive cultural traits.

I can imagine someone reading this essay and raising certain objections. What about Linux? What about Apache? Certain open source projects have been successful with teams that are scattered all over the world. But then, these were not fast moving projects. Teams that never meet in-person can become highly productive, but it takes years. And by the time these projects hit their strides, much of the core team had, in fact, traveled to different countries and met each other in-person.

One other fact argues for the importance of physical proximity. For a team to be highly productive, it needs to have both high levels of communication and high levels of trust. Body language is important to trust. In a normal, in-person conversation, the bulk of the communication is non-verbal. Below I’m going to quote from an article that is aimed at teaching salespeople how to make more sales. On a start-up, the team members are not trying to sell products to each other, but they are often in the position of having to try to sell a given strategy to each other. One team member may feel that their site should be pitched to consumers, another team member may feel strongly that the site needs to be pitched to businesses. The success of the start-up depends on all of the team members being to able to listen to all points of view, take in the maximum amount of information and opinion, and reach the best decision.

It really doesn’t matter how knowledgeable you are about your product line or how many closing techniques you have mastered, unless you earn your prospect’s trust and confidence you’re not going to make the sale.

…Body language is a mixture of movement, posture and tone of voice. Research indicates that in a face-to-face conversation, more than 70 percent of our communication is nonverbal.

Our body language reveals our deepest feelings and hidden thoughts to total strangers. In addition, nonverbal communication has a much greater impact and reliability than the spoken word. Therefore, if your prospect’s words are incongruent with his or her body language gestures, you would be wise to rely on the body language as a more accurate reflection of their true feelings.

On a slightly humorous note, I’d like to offer a movie review. The best movie ever made about an agile, virtual enterprise was Akira Kurosawa’s 1954 movie, The Seven Samurai. Of course we need to translate the movie into business-speak before the comparison becomes clear. I’m only partly kidding when I write this. I really do believe this movie offers a good example of some of the above ideas.

In the opening scene, we meet the client (poor village farmers). We then come to see their problem: bandits who plan to raid the village as soon as the harvest is in. The villagers then go to a big city to hire a contractor who can solve this problem for them. They meet Kambei, a samurai. Though the villagers can pay very little, he takes pity on them and agrees to help them. He therefore steps into the role of project manager. He must then assemble a team, and this part is pure agile enterprise in practice. The city is full of many samurai, and out of this sea of potential recruits, a group of 6 is selected. These are highly skilled workers with a strong emotional commitment to their own professional ideal – that is, they all strive to be the best samurai that they can be. The 7 samurai then set out for the village. They almost instantly form a tight-knit, highly cohesive team, because they share a commonly understood body of professional ethics and expectations. The enterprise is “virtual” in the sense that Goranson uses the word – it is project-specific, and will disband as soon as the project is complete. At the village, the samurai work in close consultation with the client to ensure the goals of the project (the defense of the village). Finally, after a decisive fight, the goals of the project are achieved – the bandits are defeated. The client is happy. The surviving samurai then disband.

In this movie, the samurai are recruited in the same fashion as sailors might have been recruited for a whaling ship, or the same way that a computer programmer might be recruited for a web start-up.

To recap, my current experience teaches me that:

1.) Agile teams should be small (as per Brooks).

2.) They should meet in person several times a week.

3.) Everyone on the team must be highly competent.

4.) Everyone on the team must be emotionally committed to their own professional ideal.

5.) For the duration of the project, everyone on the team should be full-time

Furthermore, on projects that aim to be unique or innovative, or on projects that are complex, or on projects that need to be fast-moving, I will be discouraging clients from these practices:

1.) part-time, independent contractors

2.) distributed teams, with workers geographically dispersed.

One last point, unrelated to the rest. A number of people have said to me that they feel the cheapness of overseas workers (in places like India) is driving down wages in America. I do not believe this is true. Wages in America have declined over the last 40 years, but that is because the American people have themselves become unwilling to invest in their own country. Consider the fact that in 1955 the average middle class family saved 17% of their income, but in 2000 the US savings rate hit 0%. Wages will decline in any nation that does not invest in itself. By contrast, consider Germany, which has a high investment rate. Not only does Germany have higher average wages than America, it has also had more economic growth over the last 30 years. And right now China is saving and re-investing 40% of its income. I do not know a single American family that saves 40% of its income. Most of my friends, even those with college educations and good paying jobs, only save 10% or 15% of their income. It is wrong for Americans to first engage in an orgy of over-consumption and under-investment, and then blame poor nations like India and China for the obvious problems that must result from such reckless behavior.

Penelope Trunk, on the folks who do not understand her, and the entrepreneurs who do:

I’m going to answer the question people ask me most often: “What do I do when my company is out of money?”

Here is the answer:

1. Lay people off to save money
2. Forgo salaries to save money
3. Make a sale to generate money
4. Cut back your family’s spending to redirect money to the company
5. Kill your personal credit to redirect money to the company

All those options suck, of course. I should know. I did them all. And each time I wrote a blog post about how I was going nuts from funding, or making my family crazy from funding, commenters would tell me I look too crazy for anyone to fund.

But entrepreneurs wrote to me to tell me that they understand. Because most entrepreneurs have experienced something similar. They just don’t write about it. And investors are not stupid: They know this is what happens to entrepreneurs. That’s why investors are investors and not entrepreneurs.

Lots of people told me to throw in the towel, but entrepreneurs never did. Because entrepreneurs knows that having a successful startup is really about not quitting. You never get the business model right on the first try. You never feel like you know what you’re doing, and you always have to adjust and adjust until you find what works. If you have passion and energy and faith, you keep going.

But it has to be an insane amount of passion and energy and faith, because there is no sane reason to have a venture backed start-up. A start-up does not get funding if it is a reasonable business model. A start-up gets funded because it’s shooting for the moon, and investors fund companies to have a lottery ticket to the moon. So the odds are terrible that any entrepreneur will succeed. It is always a more sane financial decision to work at a corporate job for a paycheck.

Dan Bezdek posted on LinkedIn, suggesting that it was impossible to launch an online business nowadays due to the hyper-competition on the web. He wrote:

I still think there is a huge difference between an online business and a traditional business. We have this grocery store in the neighborhood which is right next to Safeway. Now, you’d imagine there is no hope for this store; however, not only it is surviving, but in fact has so many customers.

He also wrote:

The main problem is that the culture of internet is free service, and this problem will remain with Internet forever. You might spend a couple of years in development, and provide a great service; but you can’t charge users even a $1 except in very few niche sections.

I wrote a response, but LinkedIn limited me to 4,000 characters, so I was not able to post my whole response. I post it here, instead.

- – - – - – - – - – - – - – -

Dan Bezdek, I’m afraid I’m unable to understand your reasoning. I’m going to try to paraphrase what I think you are saying.

You write “It is so easy for anyone in the world to compete with you even in places that $100 is a week pay.”

I think you are trying to say that because computer programmers can be hired cheaply, it is more difficult for you to build a business? But the opposite is true: cheap computer programmers make it easier to launch a new business. I can not think of a single case in economic history where the falling price of a supply factor made it more difficult to launch a new business. A few examples:

1.) During the late 1600s, in Europe, the dramatic fall in the price of paper allowed for the creation of the first modern newspapers. The Spectator launched on March 4, 1712.

2.) During the mid 1700s, in Europe, the dramatic fall in the price of coffee beans allowed for the first coffee houses to open. Paris had hundreds of small coffee stands at the outbreak of the Revolution.

3.) During the late the 1800s, in America, the dramatic fall in the price of communications (the telegraph and the telephone) allowed for the organization of businesses over a distance, and at a scale, never before seen.

4.) During the late 1900s, all over the world, the dramatic fall in the price of computing power brought several new technologies into the mass market, including cell phones, personal computers, and personal (non-business) software.

As a general rule, the cheaper supplies get, the easier it is to start a business.

You also wrote this:

“In any case, the point of all this is to say that developing even a barely successful online business is probably 10 times more difficult than setting up a grocery shop.”

You didn’t mention which country you are in. If you are in the United States, then your statement is incorrect. The retail sector in the US is overbuilt and is expected to consolidate over the next 10 years. Many grocery chains are expected to go bankrupt. Meanwhile, the Commerce Department projects that the software industry will continue to grow, and much of that growth will be happening on the web.

If you do not live in the US, then I’d have to know what country you are in, to know if there is any truth to what you are saying. Some countries, such as France and Italy, offer strong legal protections to small firms. In those countries, small groceries have some hope of surviving, but that is because of government protection, not economic fundamentals.

I think the concept that you are trying to get at is what economists would call “barriers to entry”. That is, what barriers keep competitors from entering your market and competing with you? And I think what you are thinking is that a local grocery store, because it is grounded in a specific geographical space, has some immunity from competition, whereas a web site has to compete with every other web site on the web.

Again, if you are speaking about the US, you are plainly wrong. The history of post-war economic development in the US is the history of retail consolidation. Mom-n-Pop stores have been relentlessly replaced by big chains such as Wal-Mart. Massive numbers of bankruptcies have happened in every state. The small-scale grocery was once common, and now is nearly extinct. They’ve been hunted to extinction through the relentless competitive pressures of market consolidation.

Consider the graph that the US government has posted here:

In the late 1990s, a number of leading grocery retailers went on a buying spree. Between 1997 and 2000, more than 4,100 stores were acquired, amounting to almost a fifth of all U.S. supermarkets. Mergers and acquisitions by large grocery retailers, including Kroger, Albertson’s, Ahold USA, and Safeway, produced a significant increase in the share of grocery store sales by the largest firms. By 2005, the 20 largest retailers accounted for 61.6 percent of total U.S. grocery store sales, up from 40.6 percent in 1995.

If you look at the facts, you’ll admit that small-scale groceries have mostly been wiped out. And this happened during some of the same decades that saw the explosive growth of the US software market.

Clearly, it is possible to make a lot of money on the web. 37 Signals makes millions of dollars in sales, and Amazon makes billions of dollars in sales. AOL is making a shockingly large gamble on weblogs.

You include this surprising comparison:

Now, compare this store to many 2-3 men operations online trying to make it online offerring some service; there are thousands of such operations which wish they were making as much as that grocery store.

I assume you meant “people” where you wrote “men”. A 3 person web start-up will cost less than running a grocery store. Even a small grocery store is going to have more than $50,000 worth of inventory in it, then it will have labor costs, rent or real estate taxes plus land cost, licenses for health, fire, safety, etc. If you are going to compare businesses that have different capital requirements, then you might as well write “I tried to run a lemonade stand, but it never made as much money as Toyota.” The comparison is absurd.

Of course, as segments of the web mature, the capital requirements for starting a web site go up for that particular segment. Once upon a time, a long time ago, you could start a successful weblog for free. Nowadays, if you are starting a weblog which you hope will develop a mass audience, then you should probably have $100,000 for marketing and writers. Likewise, if for some crazy reason you decide to launch a competitor to YouTube, you should start with $100 million in the bank. In the future, you may need millions of dollars to get into any established segment. But some segments will remain open to the sudden hit that comes form nowhere. Weblogs, for instance, began to consolidate some time ago, yet new weblogs still occasionally burst forth and become large-scale hits. And one of the great things about the web (so far) has been the speed with which new segments emerge.

I’ve already written extensively about the costs of building a web site. And I’ve helped launched a number of successful sites that cost less than $100,000 to get going. And I continue to work with start-ups that have budgets under $100,000, and I’ve great faith that a number of these will become successful.

If you are thinking of starting a new business, you’d be wise to start it online.

Apparently 800 people not only read Jesse James Garrett’s blog post that introduced the concept of Ajax, and they went to the trouble of linking to it and tagging the URL, but many of them didn’t tag it with the term “ajax”.

Popular tags get set quickly, but not in stone

It only took 10 users for ‘ontology tags folksonomy’ to sort to the top of the tag list, meaning that even a small group of users can pretty quickly create much of the consensus value around a given link. This is in keeping with the idea of lowest common denominator tagging. However, though this consensus was established quickly, it was not frozen, with the positions among those three words varying, and with tags eventually replacing tagging.

I’ve only found one popular link so far that violates this idea, for the original Adaptive Path piece on Ajax. For this link, the tag ‘ajax’ is overwhelmingly #1, with 1171 occurrences from 2352 taggers. (Second place is ‘javascript’, with a mere 644 tags.) Yet over 800 people, more than a third of the total, tagged it before ‘ajax’ hit the #1 spot — it’s as if you can see Ajax becoming a real term as enough people read the article. The Ajax article may be a one off, or there may be some small but instructive number of links whose consensus view changes slowly, documenting the rise of some new concept.

This is a good example of how assigning meaning to a term, or a word, is a social activity, one that evolves over time. The importance of an article depends on how many people cite it, but people will start citing an article once other people have agreed that the article is important. I’m fairly sure this process of feedback will be found for any word, idea, or article that people have agreed is important, and in fact, the feedback itself is pretty much the same thing as social recognition of importance. (Obviously, for technical terms, only those with the technical skill to follow the conversation can agree or disagree about somethings importance.)

Odd. Here is a blog post, nominally written in 2005, that uses the phrase “irrational exuberance” and links to a page that explains that the phrase “irrational exuberance” comes from a speech in December of 2006. I wonder how Gene Smith, in April of 2005, knew that Allan Greenspan would coin this phrase 20 months later?

Turning a blind eye to those problems is what turns strange zeitgeist into irrational exuberance.

Posted by Gene Smith on Apr 20, 2005

Just being yourself is a creative act:

But why lie to appear competent or likable? Why not just be yourself? In fact, “just being yourself”, if we examine it closely, takes creative effort. Our expression of who we are involves choices that reflect social and interpersonal context, our mood, our personality, our need to maintain our self-image and so on. If we consider self-presentation as a creative process, we can see how it can easily slide into deception. Every interaction involves decisions about which attributes to emphasize and which to minimize, which impulses to follow and which to ignore. At some point, we may not be choosing among our actual traits and our sincere reactions. We may simply fabricate the traits and reactions the social situation calls for, or that we think it calls for. In other words, we might lie.