Why do people sometimes feel attacked, even though no one is attacking them?
Blaine Cook has either been fired or has quit Twitter. Most people assume he’s quiting/fired because of the difficulties that Twitter has had with scaling. Twitter has been a favorite weapon in the hands of Java’s defenders - they use Twitter to bash Ruby On Rails. The media attention given to Cook’s departure reminds me of an incident from a year ago that I wanted to blog about, but didn’t at the time.
The incident last year started when Radical Behavior interviewed Alex Payne, who was one of the developers at Twitter. Alex said:
By various metrics Twitter is the biggest Rails site on the net right now. Running on Rails has forced us to deal with scaling issues - issues that any growing site eventually contends with - far sooner than I think we would on another framework.The common wisdom in the Rails community at this time is that scaling Rails is a matter of cost: just throw more CPUs at it. The problem is that more instances of Rails (running as part of a Mongrel cluster, in our case) means more requests to your database. At this point in time there’s no facility in Rails to talk to more than one database at a time. The solutions to this are caching the hell out of everything and setting up multiple read-only slave databases, neither of which are quick fixes to implement. So it’s not just cost, it’s time, and time is that much more precious when people can[’t] reach your site.None of these scaling approaches are as fun and easy as developing for Rails. All the convenience methods and syntactical sugar that makes Rails such a pleasure for coders ends up being absolutely punishing, performance-wise. Once you hit a certain threshold of traffic, either you need to strip out all the costly neat stuff that Rails does for you (RJS, ActiveRecord, ActiveSupport, etc.) or move the slow parts of your application out of Rails, or both.It’s also worth mentioning that there shouldn’t be doubt in anybody’s mind at this point that Ruby itself is slow. It’s great that people are hard at work on faster implementations of the language, but right now, it’s tough. If you’re looking to deploy a big web application and you’re language-agnostic, realize that the same operation in Ruby will take less time in Python. All of us working on Twitter are big Ruby fans, but I think it’s worth being frank that this isn’t one of those relativistic language issues. Ruby is slow.
David Heinemeier Hansson, the inventor of Rails, had an extremely defensive response:
Rails makes the act of developing such a pleasant experience that when you need to follow the same scaling path as every other shared-nothing stack, the contrast can feel stark. And perhaps it’s a natural reaction to feel a need to blame something for that contrast, however natural it is.
[…]
Second, when you work with open source and you discover new requirements not met by the software, it’s your shining opportunity to give something back. Rather than just sit around idle waiting for some vendor to fix your problems, you get the unique chance of being a steward of your own destiny.
[…]
Once the stress of having to deal with that in the moment subsides, I’m convinced that the team will grow beyond the blame game, get their hands dirty as full participants in an open source community, and contribute back their advances to the framework.
Gluttonous had what was, by far, the best reaction to David’s post:
The conversation shifts from where bottlenecks come from when scaling Rails to “you’re saying it’s Rails’ fault, but it’s not”. This is where we start to slide down the slippery slope. It’s easy to argue about blame. David pulls out his “I’m not a vendor, I don’t owe you shit” speech, and says that because of the beauty of open source, Twitter is in charge of their own destiny.
He’s absolutely right. He’s also a big jerk about being right. David seems to imply that Twitter has been sitting on their hands, doing nothing, refusing to solve their own problems, and not contributing back to the community. This is false. It’s also completely beside the point.
Early in the discussion we started to imagine blame where it didn’t exist. Alex never says that Rails should support doing 11k requests a second. He simply said it doesn’t, which is something no one will dispute.
So, we’ve moved into rough territory, but someone goes and does something nice (Woo hoo!). Dr Nic writes a plugin to use multiple databases with Rails. Way cool. Really way cool. Awesome job Nic. This is the part of open source that I love, when people help each other just because it’s an interesting problem and they’re a nice person. Hugs and kittens all around.
DHH tosses down an ”I told you so”. David, what are you doing man? Yes, Dr Nic is awesome, and yes, Ruby is wondrously hackable, and yes, Rails allows plugins easily, BUT why do you keep beating this dead horse? No one said anything about a critical flaw. No one said Rails is inherently flawed and can never be adapted for high traffic sites. Just the opposite is true! Alex and the other guys at Twitter are those who are pushing Rails further than it’s ever gone. These are the pioneers. They’ve gotten to the hard part, and they’re still going, despite the difficulty. Really, does anyone think that sites at that load are running out of the box copies of Rails? It’s just not the case. That’s ok. The fact that Rails can be modified is one of its strengths. There’s no need to be defensive here. That’s the big problem we all need to face.
People often say that DHH has a big ego. That is no crime in itself - in some situations, having a big ego is healthy and even useful. But this behavior - where someone points to a weakness that needs to be improved, and their remark is interpreted as an attack that needs to be counter-attacked - is a major problem. This is the kind of behavior that cripples organizations, leads to pointless in-fighting and turf wars, and saps the energy out of projects.
The solution? People (especially those in leadership positions) should welcome attacks. That way, even when no attack actually exists, as in this case, the original speaker will still get a welcoming response. There are 3 advantages to maintaining a welcoming attitude to attacks:
1.) When an attack really does exist, a welcoming response will often soften the attitude of the person making it.
2.) An open attitude to criticism can not be demonstrated in the face of praise, it can only be demonstrated in the face of criticism. Every product or idea, no matter how good, will have some flaws, and so it is important to be open to negative feedback.
3.) Welcoming attacks saves you from the embarrassment of reacting in a defensive way, when, in fact, no one was ever attacking you.