Closer To The Ideal

Nati Shalom writes about the crisis of scaling that hits startups once they become successful. Most startups are unsure if their idea will be successful, so they do not, initially, waste any energy planning to scale up. Instead, they do what is necessary to get to market as fast as possible. If they are then successful, they must then re-engineer their architecture to take scaling into account. Shalom offers some tips on how to do this. He also feels the problem can be avoided from the start:

Are we all doomed to go through this painful process when we are successful?

We seem to take it for granted that dealing with scalability is complex. When we start a new application it’s hard to know whether we’re ever going to be successful to the point where the investment in scalability is worth the effort. At this initial stage the important thing is time-to-market. We want to get our idea out there as quickly as possible. This is a reasonable desire as indeed most projects don’t take off.

Now imagine what would happen if dealing with scalability wasn’t that difficult. That would have change the entire decision making process, and would enable Twitter and many others to start with a scalable architecture from day one, avoiding this painful process.

So the question is what would is required to simplify building a scalable application to the point in which it is as simple as building it for a single machine?

From my experience, most challenges have already been faced by and dealt with others - so the first thing that I did was look at how others (not necessarily in the same industry) addressed this issue.

In this case, storage virtualization is a good example. At first, we used local disks. Local disks tend to get filled-up quite fast. It was very hard to deal with this problem as it required replacing the disk with a bigger one every time full capacity was reached. IT had to go through this process for every user and every application — very painful and costly. The solution came in the form of NAS and SAN, or network-attached storage. Instead of using a local disk, use a virtual disk that resides somewhere on the network. The user and the application don’t need to be aware of it, because they use a local disk driver that virtualizes the network devices to make them look as if they were just another local disk. The application scales but hasn’t changed as a result. We can add and remove devices as we wish with no changes to the application. Later on, if there is a more cost effective solution available, we can easily replace the devices.

We can apply the same concept of virtualization to the middleware stack — namely the data, the messaging and the processing — with the same degree of simplicity. The application interacts with a “proxy” that hides the details of how a message or update operation is routed, how fail-over is handled, how data is partitioned and so on.

With services such as Amazon EC2, and other cloud environments, this can be made even simpler, as we can have a pre-configured image and hardware ready for deployment. All we need to do is just deploy our business logic. (See an EC2 example here).

With today’s frameworks architectures, we don’t have to go through the same painful experience. We can build scalable architectures from the get-go. I would even argue that it takes less time to build applications with this approach than the traditional client-server approach.

I suppose Sam Ruby likes Erlang for these same reasons - automatic concurrency.

I do think that Shalom is under-estimating the degree to which emotions, self-esteem and uncertainty combine to ensure conservative choices that may be painful later on. His example of disk-virtualization is a fine representative - none of the start-ups I’ve worked with have choosen to use Amazon S3, even though they all might benefit from it later on, and it is relatively simple to use. But a lot of people starting a new venture will collect together people they like working with, and the entrepreneur who gets the project going will make use of the skills that his/her team already has, rather than asking them to learn a new skill, like amazon S3.

 Zed Shaw has posted a rant about everything he hates in the Ruby community. The rant is so over-the-top that is impossible not to find it entertaining. It is rare to see anyone with 21 years experience post anything like this in public. Most people in his position would be worried about their careers. He, apparently, is beyond the point of caring.

At times, the rant is almost incoherent:

Alright people, time to get a huge grip on reality’s collar and hold on tight.

Ruby on Rails is not a mother fucking industry!

Jesus fucking christ on a goddamned pike you absolute mother fucking donkey dick sucking morons get a fucking grip!

You are not in an industry. You are a bunch of people barely scraping by in a tiny little sector of a moderate sized piece of the economy. Gaming alone makes you all looks like the pathetic little crumbs I brush out of my toaster when it smells bad.

He makes a few good points though:

Where I work the company is willing to blow huge amounts of money on a consulting firm or hardware, but ends up firing people when times get tight. It’s a universal mass hysteria that paying $100 – $200 per hour for a group of consultants is preferable to simply hiring good employees. At the rates companies pay these consultants they could hire 4 full time employees.

Consultancies used to provide a service by managing the entire project so you didn’t have to do much. Now with Agile and Pair Programming the consulting firms can dupe clients into helping them make the sausage, provide little to no services, yet still charge insane rates. What’s impressive is these consulting firms somehow charge rates that are 5 or 6 times what they pay their employees.

…

Let’s take ThoughtWorks as a classic example of the hysteria. They decided to get into the Ruby on Rails game and went full bore. I was telling people right when Rails came out that doing it for internal projects at big companies would be a huge money maker. Nobody believed me, and now rather than all my smart friends working on cool applications for big money I have ThoughtWorks fucking up my party.

Before you continue this part of the rant ask yourself a question:

How did ThoughtWorks go from 0% Rails business to 60% Rails in just a few short months, but somehow didn’t hire that many top notch Ruby guys? Remember, if 60% of your business is Rails then 60% of your people need Rails training or else you have to hire more people. If they didn’t hire any more people than that means…the people they had were retrained. With two week training courses. Huh? How does that make them experts?

What happens if you do that is you have a group of former C# and Java guys running around writing shitty Ruby code and training on the client’s dime for huge fees.

Some of the post seems to border on libel:

In the two projects I’ve taken from ThoughtWorks I found mountains of horrible, horrible code. They of course try to pull the classic “there’s many ways to do everything in programming” but this time they kind of get caught because Ruby on Rails means stay on the Rails. There is an established best practice way to build web applications with Rails and that’s the entire point of the system. When ThoughtWorks fucked up these projects they did it in such a completely deviated way that it was impossible to defend.

Additionally, the people they placed on these projects were not well trained at all, had no idea about simple Ruby idioms let alone good design, and spent more of their time drinking and having fun than actually getting shit done. At the last project they actually had bottles of Pedialyte in the fridge to help with their hangovers after wild nights partying.

I need to post something to inaugurate this blog, so I will start by reposting some comments I wrote over at Burningbird.

Shelley Powers suggests one of the things that leads to gender disparities in the tech world:

If Tim Bray wants to get a better idea of why there aren’t as many women seemingly interested in Ruby, Rails, or associated technologies such as Ajax, he needs look no further than sites like Ajaxian. In all of the time I’ve been reading the site, it covered women–our contributions, as writers, as techs, in any capacity–a grand total of three times. Which is why I decided to unsubscribe from the site today. It’s not for everyone interesting in Ajax–only the boys interested in Ajax.

Someone in the comments suggested:

I think part of it, at least in the RoR world, is that there’s just so much focus on who the hotshots are, and so so much cronyism.

My thoughts:

I don’t know if it’s useful to look at this issue on the micro level. Questions like “Does the RoR community have enough women?” miss the point. The problem is large scale. The number of women receiving advanced degress in computer science peaked in 1989 and has since been in steady decline. That number is, I think, a useful metric for measuring women’s participation in computer science. Whereas women have made dramatic gains in the legal profession, the medical profession, and in managing large enterprises, they have lost ground in the computer industry.

I’ve yet to hear a great explanation of what’s chased women away from the tech industry. The decline since 1989 is easy to measure, yet most of the explanations that I’ve heard are contradicted by the dramatic gains women have made in the law and medical industries.

Consider a few of the most simplistic explanations:

Women don’t like hard science? Nothing is tougher than organic chemistry, which tens of thousands of women have had to master on their way to becoming doctors or “nurse practitioners” (the highest level of nursing).

Women don’t like long, brutal hours? I’ve a friend who just went through the UVA medical school. During her last year of school her average work week was 90 hours. Sometimes it was longer. 90 was the average.

Women are no good at math? I could repeat my comment, above, about organic chemistry, which involves some tough math, but I’d also like to add that I’ve been doing programming in script languages for 12 years now, and I’ve never needed advanced math. Not all programming involves gyroscope stabalized frictionless vector navigation for NASA space craft.

Women require balanced lives, and are unwilling to spend time away from their families? Relative to law and medicine, the tech industry provides more flex time and more chances to work at home. Also, women do put in long hours in the fields of law, medicine, management and marketing. It’s just tech, in particular, where women have been driven out.

Women are drawn to the more artistic side of things, they don’t like technical stuff? Pure sexism, but even if it was true, it wouldn’t explain why graphic design for the web (the artistic side of web work) has become so male dominated, with stars like Eric Meyer and Dave Shea keynoting every design conference.

It’s clear that women have been chased away from the tech field since 1989. The numbers are clear about that, and match what anecdotal evidence most of us can offer. But why? I’ve yet to have a eureka moment, where someone offers an explanation so clear that I simply jump and go “That’s it! That explains it all!”