Closer To The Ideal

Our biggest project of the last 8 months has been The Second Road. This site is built on top of the framework that I started developing back at Category4.com (which they released as open source). We had a big roll-out on May 29th, which we completed and clarified much of the functionality. The site just got a nice write up in the local paper:

Often it is late at night, when there is neither a 12-step meeting to attend nor anyone awake to phone, that the craving for a drink is strongest.

And it is times like these when Ginger Bauler goes online to reach out to others recovering from addiction, finding solace in their tales of success and providing encouragement for those trying to break the shackles of dependency.

Bauler, who used to manage a research laboratory in Charlottesville, writes a blog about her struggles with alcoholism and her quest for sobriety on The Second Road — a new online support community for drug and alcohol addicts started by Charlottesville residents.

Writing about her battles enables Bauler to become “accountable” for her recovery, she says. And meeting and keeping in touch with those dealing with similar experiences gives her strength.

“I develop these relationships with total strangers, but with whom I’m completely connected because of this disease,” said Bauler, who has become a managing editor of the site.

The Second Road is the brainchild of local documentary filmmaker Melissa Shore, who partnered with Chip Ransler, owner of a digital publishing firm, to launch the site in November. By late May the social networking site had 225 members and more than 1,400 different people had visited it in a recent 10-day stretch.

Each member of The Second Road has his or her own profile page, similar to sites such as Facebook and MySpace, where they can post information and display customized features. The site also includes a series of blogs, chat groups, a “sharing wall” for inspirational quotes and testimonial videos from recovering addicts.

Shore, who grew up watching family members battle addictions, noticed that there was a gap in services for people in recovery and a need for round-the-clock services.

People in recovery do not always have access to meetings or counseling services, and some may have no one to turn to in times of crisis, Shore realized. That is especially true in rural communities, where social services either might not be readily available or are far away.

“There are hours of the day when you can’t or don’t feel comfortable reaching out for help,” Shore said. “The beauty of the Internet, of course, is that it’s 24-7.”

The site has won praise from many in the local mental health community as a vital tool to help round out recovery services.

“The concept is absolutely brilliant,” said Jeff Gould, administrator of the Charlottesville/Albemarle Drug Court. “This type of online recovery network is just perfect for people who can’t get to meetings.”

Beth Elliott, a retired social worker who advises the site’s creators, says that members are using The Second Road to implement the treatment plans they develop with counselors, through making lists and blogging about their successes and missteps.

Nati Shalom writes about the crisis of scaling that hits startups once they become successful. Most startups are unsure if their idea will be successful, so they do not, initially, waste any energy planning to scale up. Instead, they do what is necessary to get to market as fast as possible. If they are then successful, they must then re-engineer their architecture to take scaling into account. Shalom offers some tips on how to do this. He also feels the problem can be avoided from the start:

Are we all doomed to go through this painful process when we are successful?

We seem to take it for granted that dealing with scalability is complex. When we start a new application it’s hard to know whether we’re ever going to be successful to the point where the investment in scalability is worth the effort. At this initial stage the important thing is time-to-market. We want to get our idea out there as quickly as possible. This is a reasonable desire as indeed most projects don’t take off.

Now imagine what would happen if dealing with scalability wasn’t that difficult. That would have change the entire decision making process, and would enable Twitter and many others to start with a scalable architecture from day one, avoiding this painful process.

So the question is what would is required to simplify building a scalable application to the point in which it is as simple as building it for a single machine?

From my experience, most challenges have already been faced by and dealt with others – so the first thing that I did was look at how others (not necessarily in the same industry) addressed this issue.

In this case, storage virtualization is a good example. At first, we used local disks. Local disks tend to get filled-up quite fast. It was very hard to deal with this problem as it required replacing the disk with a bigger one every time full capacity was reached. IT had to go through this process for every user and every application — very painful and costly. The solution came in the form of NAS and SAN, or network-attached storage. Instead of using a local disk, use a virtual disk that resides somewhere on the network. The user and the application don’t need to be aware of it, because they use a local disk driver that virtualizes the network devices to make them look as if they were just another local disk. The application scales but hasn’t changed as a result. We can add and remove devices as we wish with no changes to the application. Later on, if there is a more cost effective solution available, we can easily replace the devices.

We can apply the same concept of virtualization to the middleware stack — namely the data, the messaging and the processing — with the same degree of simplicity. The application interacts with a “proxy” that hides the details of how a message or update operation is routed, how fail-over is handled, how data is partitioned and so on.

With services such as Amazon EC2, and other cloud environments, this can be made even simpler, as we can have a pre-configured image and hardware ready for deployment. All we need to do is just deploy our business logic. (See an EC2 example here).

With today’s frameworks architectures, we don’t have to go through the same painful experience. We can build scalable architectures from the get-go. I would even argue that it takes less time to build applications with this approach than the traditional client-server approach.

I suppose Sam Ruby likes Erlang for these same reasons – automatic concurrency.

I do think that Shalom is under-estimating the degree to which emotions, self-esteem and uncertainty combine to ensure conservative choices that may be painful later on. His example of disk-virtualization is a fine representative – none of the start-ups I’ve worked with have choosen to use Amazon S3, even though they all might benefit from it later on, and it is relatively simple to use. But a lot of people starting a new venture will collect together people they like working with, and the entrepreneur who gets the project going will make use of the skills that his/her team already has, rather than asking them to learn a new skill, like amazon S3.

It is impossible to keep up with all the new technologies that came out over the last year, especially one’s that I probably won’t ever use. I admit, I was confused, till now, regarding Adobe Flex, Microsoft Silverlight, JavaFX. Apparently these were all aimed at the same basic market, the same one that Google Gears aims at: ????????building applications that have a front-end that lives and runs as a desktop app, but pulls data from the web. I’m pleased to now at least understand what all these are about. I can’t see myself building this kind of software in the near future, so I guess I can ignore these technologies. If I do end up doing this kind of software, I’m sure I’ll use Google Gears, simply because I already have some slight introduction to the Google API.

Blaine Cook has either been fired or has quit Twitter. Most people assume he’s quiting/fired because of the difficulties that Twitter has had with scaling. Twitter has been a favorite weapon in the hands of Java’s defenders – they use Twitter to bash Ruby On Rails. The media attention given to Cook’s departure reminds me of an incident from a year ago that I wanted to blog about, but didn’t at the time.

The incident last year started when Radical Behavior interviewed Alex Payne, who was one of the developers at Twitter. Alex said:

By various metrics Twitter is the biggest Rails site on the net right now. Running on Rails has forced us to deal with scaling issues – issues that any growing site eventually contends with – far sooner than I think we would on another framework.The common wisdom in the Rails community at this time is that scaling Rails is a matter of cost: just throw more CPUs at it. The problem is that more instances of Rails (running as part of a Mongrel cluster, in our case) means more requests to your database. At this point in time there’s no facility in Rails to talk to more than one database at a time. The solutions to this are caching the hell out of everything and setting up multiple read-only slave databases, neither of which are quick fixes to implement. So it’s not just cost, it’s time, and time is that much more precious when people can['t] reach your site.None of these scaling approaches are as fun and easy as developing for Rails. All the convenience methods and syntactical sugar that makes Rails such a pleasure for coders ends up being absolutely punishing, performance-wise. Once you hit a certain threshold of traffic, either you need to strip out all the costly neat stuff that Rails does for you (RJS, ActiveRecord, ActiveSupport, etc.) or move the slow parts of your application out of Rails, or both.It’s also worth mentioning that there shouldn’t be doubt in anybody’s mind at this point that Ruby itself is slow. It’s great that people are hard at work on faster implementations of the language, but right now, it’s tough. If you’re looking to deploy a big web application and you’re language-agnostic, realize that the same operation in Ruby will take less time in Python. All of us working on Twitter are big Ruby fans, but I think it’s worth being frank that this isn’t one of those relativistic language issues. Ruby is slow.

David Heinemeier Hansson, the inventor of Rails, had an extremely defensive response:

Rails makes the act of developing such a pleasant experience that when you need to follow the same scaling path as every other shared-nothing stack, the contrast can feel stark. And perhaps it’s a natural reaction to feel a need to blame something for that contrast, however natural it is.

[…]

Second, when you work with open source and you discover new requirements not met by the software, it’s your shining opportunity to give something back. Rather than just sit around idle waiting for some vendor to fix your problems, you get the unique chance of being a steward of your own destiny.

[…]

Once the stress of having to deal with that in the moment subsides, I’m convinced that the team will grow beyond the blame game, get their hands dirty as full participants in an open source community, and contribute back their advances to the framework.

Gluttonous had what was, by far, the best reaction to David’s post:

The conversation shifts from where bottlenecks come from when scaling Rails to “you’re saying it’s Rails’ fault, but it’s not”. This is where we start to slide down the slippery slope. It’s easy to argue about blame. David pulls out his “I’m not a vendor, I don’t owe you shit” speech, and says that because of the beauty of open source, Twitter is in charge of their own destiny.

He’s absolutely right. He’s also a big jerk about being right. David seems to imply that Twitter has been sitting on their hands, doing nothing, refusing to solve their own problems, and not contributing back to the community. This is false. It’s also completely beside the point.

Early in the discussion we started to imagine blame where it didn’t exist. Alex never says that Rails should support doing 11k requests a second. He simply said it doesn’t, which is something no one will dispute.

So, we’ve moved into rough territory, but someone goes and does something nice (Woo hoo!). Dr Nic writes a plugin to use multiple databases with Rails. Way cool. Really way cool. Awesome job Nic. This is the part of open source that I love, when people help each other just because it’s an interesting problem and they’re a nice person. Hugs and kittens all around.

DHH tosses down an ”I told you so”. David, what are you doing man? Yes, Dr Nic is awesome, and yes, Ruby is wondrously hackable, and yes, Rails allows plugins easily, BUT why do you keep beating this dead horse? No one said anything about a critical flaw. No one said Rails is inherently flawed and can never be adapted for high traffic sites. Just the opposite is true! Alex and the other guys at Twitter are those who are pushing Rails further than it’s ever gone. These are the pioneers. They’ve gotten to the hard part, and they’re still going, despite the difficulty. Really, does anyone think that sites at that load are running out of the box copies of Rails? It’s just not the case. That’s ok. The fact that Rails can be modified is one of its strengths. There’s no need to be defensive here. That’s the big problem we all need to face.

People often say that DHH has a big ego. That is no crime in itself – in some situations, having a big ego is healthy and even useful. But this behavior – where someone points to a weakness that needs to be improved, and their remark is interpreted as an attack that needs to be counter-attacked – is a major problem. This is the kind of behavior that cripples organizations, leads to pointless in-fighting and turf wars, and saps the energy out of projects.

The solution? People (especially those in leadership positions) should welcome attacks. That way, even when no attack actually exists, as in this case, the original speaker will still get a welcoming response. There are 3 advantages to maintaining a welcoming attitude to attacks:

1.) When an attack really does exist, a welcoming response will often soften the attitude of the person making it.

2.) An open attitude to criticism can not be demonstrated in the face of praise, it can only be demonstrated in the face of criticism. Every product or idea, no matter how good, will have some flaws, and so it is important to be open to negative feedback.

3.) Welcoming attacks saves you from the embarrassment of reacting in a defensive way, when, in fact, no one was ever attacking you.

 Zed Shaw has posted a rant about everything he hates in the Ruby community. The rant is so over-the-top that is impossible not to find it entertaining. It is rare to see anyone with 21 years experience post anything like this in public. Most people in his position would be worried about their careers. He, apparently, is beyond the point of caring.

At times, the rant is almost incoherent:

Alright people, time to get a huge grip on reality’s collar and hold on tight.

Ruby on Rails is not a mother fucking industry!

Jesus fucking christ on a goddamned pike you absolute mother fucking donkey dick sucking morons get a fucking grip!

You are not in an industry. You are a bunch of people barely scraping by in a tiny little sector of a moderate sized piece of the economy. Gaming alone makes you all looks like the pathetic little crumbs I brush out of my toaster when it smells bad.

He makes a few good points though:

Where I work the company is willing to blow huge amounts of money on a consulting firm or hardware, but ends up firing people when times get tight. It’s a universal mass hysteria that paying $100 – $200 per hour for a group of consultants is preferable to simply hiring good employees. At the rates companies pay these consultants they could hire 4 full time employees.

Consultancies used to provide a service by managing the entire project so you didn’t have to do much. Now with Agile and Pair Programming the consulting firms can dupe clients into helping them make the sausage, provide little to no services, yet still charge insane rates. What’s impressive is these consulting firms somehow charge rates that are 5 or 6 times what they pay their employees.

…

Let’s take ThoughtWorks as a classic example of the hysteria. They decided to get into the Ruby on Rails game and went full bore. I was telling people right when Rails came out that doing it for internal projects at big companies would be a huge money maker. Nobody believed me, and now rather than all my smart friends working on cool applications for big money I have ThoughtWorks fucking up my party.

Before you continue this part of the rant ask yourself a question:

How did ThoughtWorks go from 0% Rails business to 60% Rails in just a few short months, but somehow didn’t hire that many top notch Ruby guys? Remember, if 60% of your business is Rails then 60% of your people need Rails training or else you have to hire more people. If they didn’t hire any more people than that means…the people they had were retrained. With two week training courses. Huh? How does that make them experts?

What happens if you do that is you have a group of former C# and Java guys running around writing shitty Ruby code and training on the client’s dime for huge fees.

Some of the post seems to border on libel:

In the two projects I’ve taken from ThoughtWorks I found mountains of horrible, horrible code. They of course try to pull the classic “there’s many ways to do everything in programming” but this time they kind of get caught because Ruby on Rails means stay on the Rails. There is an established best practice way to build web applications with Rails and that’s the entire point of the system. When ThoughtWorks fucked up these projects they did it in such a completely deviated way that it was impossible to defend.

Additionally, the people they placed on these projects were not well trained at all, had no idea about simple Ruby idioms let alone good design, and spent more of their time drinking and having fun than actually getting shit done. At the last project they actually had bottles of Pedialyte in the fridge to help with their hangovers after wild nights partying.

I’ll link to this article because I know I’ll want to refer to it later. I very much enjoy its dicussion of the negative effects of “network effects”. Like credit in a stock market, “network effects” exaggerate both the upward and downward swings.

You see this happen all the time at dinner parties or events. Things are great until one or two people announce the intention to leave. If those folks are fun and entertaining, there’s an immediate realization that the quality of the experience is about to go down. And yet more people announce their intention to leave, and so on, until you are left with the party hosts and a big mess ;-)

Advanced discussion: Social Network Death Spiral
Now let’s do a more advanced discussion using the concepts above – for some new readers, this discussion might completely be incoherent ;-)

Let’s consider a specific scenario where a social network could easily start to “Death Spiral” – here’s some set up on the scenario:

• You have a bunch of users, let’s call the total number N
• The total number of users in the ecosystem, called the carrying capacity, is variable C
• These users all individually require some utility value on a site, let’s call this V_required
• Then there’s a retention %, called R, which depends on two factors:
  • If the utility value for users is satisfied, that is, V > V_required, then R close to 100%
  • If the utility value drops under V_required, then R is crappy, closer to 0%
• And to borrow Metcalfe’s Law, the value of the network is calculated at V = N^2

So the scenario is that as the total users for the application reaches the carrying capacity, you basically hit a point of maximum saturation – this is defined by the ratio N/C. Sometimes this ratio can also be referred to as the “efficiency” of a user acquisition process, which relays how many people you actually acquire versus the universe of all users. (Obviously you want this to be as large as possible)

Once you hit the carrying capacity and acquire all possible users, N is at the highest point, and thus the network value is also at its highest point, V = N_max^2. Similarly, because the network value V is at its highest, the retention reaches its highest point as well.

The question in this scenario is, at any point during the growth of the network, does the network value V exceed the required value of the site, which we call V_required? Does the network break through the critical mass of value?

If so, retention should be great, as defined by the explanation above. In fact, maybe you reach V_required early on during the growth of the site, which makes the acquisition process much more efficient. Early on, maybe the userbase wasn’t sticking, but a critical mass threshold is met, and suddenly the entire userbase sticks, which creates a long-term creation of ad impressions and company value.

However, if you don’t reach the required value in the network, then you’re pretty much screwed. Then the retention sucks, since the users aren’t finding value, and some percentage of them will leave. This will then remove more value from the system, causing yet another round of users to leave. This continual loss of users is a death spiral that collapses your network in fine Eflactem’s Law style.

A very interesting variation of this is when you apply Metcalfe’s Law not to the entire network of users, but rather think of a social network as a loosely grouped set of connections. In that case, some local networks might have achieved critical mass, and if they are big enough, they will be retained. However, if the smaller networks around any given group start collapsing, then sometimes even the large networks will get pulled down with them.