Closer To The Ideal

I admire Jeff Bezos more than any other business leader today. Most business leaders talk about innovation, but Bezos really does it, over and over again. Amazon is releasing a developer kit, so 3rd party developers can develop for the Kindle.

Possibly I’m a reactionary, but stuff like Amazons cloud computing failures makes me want to continue to manage my own boxes:

“[Normally] you need to assume that anything can fail — where we didn’t go far enough was to assume that everything can fail,” he said.

Teich said that while Heroku had designed for redundant servers and failover capacity, this was a novel kind of blackout for a hosting provider. A server failing was normal, he said, but it was unheard of for a whole class of resources to suddenly vanish.

Heroku had recently moved its front-end servers onto the high memory m2.2xlarge instances, and some of those instances were already running “core back-end stuff.”

Teich also said that all of Heroku’s m2.2xlarge instances were running in a single availability zone, which was a mistake. He stressed that Heroku had failover built in already — if 21 instances had failed instead of 22, or if it had spread instances across several zones, “we wouldn’t be talking [about the outage],” he said.

Nevertheless, on Friday, January 2, every m2.2xlarge instance in that availability zone suddenly vanished, despite all other types of EC2 instances running as normal. That’s unheard of in traditional hosting. It would be like every server with a given amount of RAM suddenly shutting down, regardless of operating system, age, brand, hardware or location in the data center, with no effect on its neighbors.

“For us, there’s the stuff you plan for and then there’s the stuff you don’t even know about,” Teich said.

An event like this was an “unknown unknown” that nobody planned for because nobody imagined it. He chalked it up to the learning process and pointed out that everybody in Amazon Web Services was flying by the seat of the pants at least part of the time.

“It’s not like there’s ‘best practices’ for cloud computing yet,” he said.

50 people say “hi”, for $1 each. Project launched via Amazon’s Mechanical Turk.

Stuff like this makes me afraid to ever rely on anyone’s cloud services.

BitBucket got hit with a denial of service attack. But they host everything on Amazon Web services (EC2, EBS, etc). So they were dependent on Amazon to figure out what the problem was. And Amazon took 16 long hours to figure out what the problem was. And in the mean time, Amazon kept telling BitBucket that everything was fine:

What came from that, was 5 or 6 hours of advice, some of which were obvious timesinks, while others were somewhat credible. What they kept coming back to was that EBS is a “shared network resource” and performance would vary. We were also told to use RAID0 to distribute our load over several EBS instances to increase the throughput.

At this point, we were getting less throughput than you can pull off of a 1.44MB floppy, so we didn’t accept this for an answer. We did some more tests, trying to measure the bandwidth of the machine by fetching their “100mb.bin” files, which we couldn’t do. We again emphasized that this was in fact, in all likelihood, a network problem.

At this point, our outage was well known, especially in the Twittosphere. We have some rather large customers relying on service with us, and some of these customers have some hefty support contracts with Amazon. Emails were sent.

Shortly after this, I requested an additional phone-call from Amazon, this time to our system administrator. He had been compiling some rather worrying numbers over the past hours, since up until now, the support had refused to acknowledge a problem with the service. They claimed that everything was working fine, when clearly, it was not.

This time, a different support rep. called, and this time, they were ready to acknowledge our problem as “very serious.” We sent them our aggregated logs, and shortly thereafter, they reported that “we have found something out of the ordinary with your volume.”

We had been extremely frustrated up until this point, because 1) we couldn’t actually *do* anything about it, and 2) we were being told that everything should be fine. It felt like there was an elephant right in front of us, and a person next to us was insisting that there wasn’t.

Interesting article about making a lot of money with Amazon’s affiliate program:

I have earned $119,725.45 from Amazon Associates Program since I began using it as a way to make money online late in 2003. Around half of that amount was made within the last 12 months.

In this post I want to share what I’ve learned along the way on how to make money with Amazon.

While Amazon’s Associates program is not my largest income stream (I rank how I make money blogging here) it was actually the first experiment that I did with monetizing blogs. I began to experiment with it in the last quarter of 2003 (just before I started using AdSense).

I started using it on a personal blog that had been going for around 12 months and had around a thousand readers a day – the first quarter was not spectacular in terms of earnings – I made $31.80 (around 30 cents a day) and almost gave it away.

Very interesting thread about some of the limits of the Amazon Elastic Load Balancer system:

We have tried to use ELB in a live ad-serving application for the last two weeks, and unfortunately, we are going to have to revert to a colocation setup because the reliability has been unacceptable.

Our setup was two EC2 instances behind an Amazon load balancer. The system served 45-50 million requests a day scattered around the world. The traffic pattern was “real world,” as you would expect for a live round-the-clock application.

We were seeing a lot of 503 errors in our test clients after initial launch and setup a heartbeat monitor that sent a test request every second. After running for a week, we found that a full 1% of our heartbeat test requests received 503 errors. Our monitoring on the EC2 instances behind the load balancer shows that they never returned any errors and never achieved high levels of load either. All of our investigations pointed to the conclustion that the failures were completely attributable to the load balancer.

An article about setting up private/public key pairs so you can ssh to your instances on Amazon EC2.

Also, the Scalr service looks interesting. I have the impression they are putting a much better interface over AWS.

Dan Bezdek posted on LinkedIn, suggesting that it was impossible to launch an online business nowadays due to the hyper-competition on the web. He wrote:

I still think there is a huge difference between an online business and a traditional business. We have this grocery store in the neighborhood which is right next to Safeway. Now, you’d imagine there is no hope for this store; however, not only it is surviving, but in fact has so many customers.

He also wrote:

The main problem is that the culture of internet is free service, and this problem will remain with Internet forever. You might spend a couple of years in development, and provide a great service; but you can’t charge users even a $1 except in very few niche sections.

I wrote a response, but LinkedIn limited me to 4,000 characters, so I was not able to post my whole response. I post it here, instead.

- – - – - – - – - – - – - – -

Dan Bezdek, I’m afraid I’m unable to understand your reasoning. I’m going to try to paraphrase what I think you are saying.

You write “It is so easy for anyone in the world to compete with you even in places that $100 is a week pay.”

I think you are trying to say that because computer programmers can be hired cheaply, it is more difficult for you to build a business? But the opposite is true: cheap computer programmers make it easier to launch a new business. I can not think of a single case in economic history where the falling price of a supply factor made it more difficult to launch a new business. A few examples:

1.) During the late 1600s, in Europe, the dramatic fall in the price of paper allowed for the creation of the first modern newspapers. The Spectator launched on March 4, 1712.

2.) During the mid 1700s, in Europe, the dramatic fall in the price of coffee beans allowed for the first coffee houses to open. Paris had hundreds of small coffee stands at the outbreak of the Revolution.

3.) During the late the 1800s, in America, the dramatic fall in the price of communications (the telegraph and the telephone) allowed for the organization of businesses over a distance, and at a scale, never before seen.

4.) During the late 1900s, all over the world, the dramatic fall in the price of computing power brought several new technologies into the mass market, including cell phones, personal computers, and personal (non-business) software.

As a general rule, the cheaper supplies get, the easier it is to start a business.

You also wrote this:

“In any case, the point of all this is to say that developing even a barely successful online business is probably 10 times more difficult than setting up a grocery shop.”

You didn’t mention which country you are in. If you are in the United States, then your statement is incorrect. The retail sector in the US is overbuilt and is expected to consolidate over the next 10 years. Many grocery chains are expected to go bankrupt. Meanwhile, the Commerce Department projects that the software industry will continue to grow, and much of that growth will be happening on the web.

If you do not live in the US, then I’d have to know what country you are in, to know if there is any truth to what you are saying. Some countries, such as France and Italy, offer strong legal protections to small firms. In those countries, small groceries have some hope of surviving, but that is because of government protection, not economic fundamentals.

I think the concept that you are trying to get at is what economists would call “barriers to entry”. That is, what barriers keep competitors from entering your market and competing with you? And I think what you are thinking is that a local grocery store, because it is grounded in a specific geographical space, has some immunity from competition, whereas a web site has to compete with every other web site on the web.

Again, if you are speaking about the US, you are plainly wrong. The history of post-war economic development in the US is the history of retail consolidation. Mom-n-Pop stores have been relentlessly replaced by big chains such as Wal-Mart. Massive numbers of bankruptcies have happened in every state. The small-scale grocery was once common, and now is nearly extinct. They’ve been hunted to extinction through the relentless competitive pressures of market consolidation.

Consider the graph that the US government has posted here:

In the late 1990s, a number of leading grocery retailers went on a buying spree. Between 1997 and 2000, more than 4,100 stores were acquired, amounting to almost a fifth of all U.S. supermarkets. Mergers and acquisitions by large grocery retailers, including Kroger, Albertson’s, Ahold USA, and Safeway, produced a significant increase in the share of grocery store sales by the largest firms. By 2005, the 20 largest retailers accounted for 61.6 percent of total U.S. grocery store sales, up from 40.6 percent in 1995.

If you look at the facts, you’ll admit that small-scale groceries have mostly been wiped out. And this happened during some of the same decades that saw the explosive growth of the US software market.

Clearly, it is possible to make a lot of money on the web. 37 Signals makes millions of dollars in sales, and Amazon makes billions of dollars in sales. AOL is making a shockingly large gamble on weblogs.

You include this surprising comparison:

Now, compare this store to many 2-3 men operations online trying to make it online offerring some service; there are thousands of such operations which wish they were making as much as that grocery store.

I assume you meant “people” where you wrote “men”. A 3 person web start-up will cost less than running a grocery store. Even a small grocery store is going to have more than $50,000 worth of inventory in it, then it will have labor costs, rent or real estate taxes plus land cost, licenses for health, fire, safety, etc. If you are going to compare businesses that have different capital requirements, then you might as well write “I tried to run a lemonade stand, but it never made as much money as Toyota.” The comparison is absurd.

Of course, as segments of the web mature, the capital requirements for starting a web site go up for that particular segment. Once upon a time, a long time ago, you could start a successful weblog for free. Nowadays, if you are starting a weblog which you hope will develop a mass audience, then you should probably have $100,000 for marketing and writers. Likewise, if for some crazy reason you decide to launch a competitor to YouTube, you should start with $100 million in the bank. In the future, you may need millions of dollars to get into any established segment. But some segments will remain open to the sudden hit that comes form nowhere. Weblogs, for instance, began to consolidate some time ago, yet new weblogs still occasionally burst forth and become large-scale hits. And one of the great things about the web (so far) has been the speed with which new segments emerge.

I’ve already written extensively about the costs of building a web site. And I’ve helped launched a number of successful sites that cost less than $100,000 to get going. And I continue to work with start-ups that have budgets under $100,000, and I’ve great faith that a number of these will become successful.

If you are thinking of starting a new business, you’d be wise to start it online.

I’ve been gathering data. I’m about to start doing a lot of work on EC2. I am curious about RightScale’s suggested upgrade technique, and also the criticism that appeared in the comments:

Once we’re confident in the new test version, we simply reassign the EIP 172.168.5.6 to the www_rel3 instance and shortly thereafter all users accessing the site are now receiving data packets from the new release. Remember, as long as the www_rel2 is available, you can easily swap back and forth between www_rel2 and www_rel3 until you are completely satisfied with the new site. And when you’re ready, you can terminate the old www_rel2 instance. See diagram below.

In the comments, Ajay says:

I’ve tried this technique in practice for upgrading a server on a site with live traffic and I’ve found that it does not work as you describe. Using your terminology, this is what I’ve experienced:

1) I assign the live EIP to www_rel3

2) For 75 seconds, live traffic is still directed to www_rel2

3) Then for 75 seconds after that, requests to http://www.rightscale.com go NOWHERE. Users get an error saying the browser cannot be found

4) Finally, traffic started getting routed to www_rel3, as expected.

The problem is part 3. I had always assumed that the switch was clean–either traffic would go to the old server or the new server, but it wouldn’t get lost.

Have you experienced that? How do you handle upgrades?

I am curious how long such switching really takes, on average.

We are about to launch a new project on Amazon’s EC2, so I’m looking for information about it. Jay Kuri has posted an in-depth review of the two, much worth reading:

One place where Amazon’s offering shines is in additional storage. Amazon offers EBS (Elastic Block Service) which lets you create large blocks of filesystem-level storage which can be mounted onto any instance. When configured this way, you pay per Gigabyte allocated. This is very important to those sites which require a significant amount of storage. Mosso does not offer mountable additional storage.

…Both Amazon’s EC2 and Mosso’s cloud-servers require you to pay for the bandwidth you use. In both cases, their costs are reasonable. Amazon is a bit cheaper here at $0.17 per gigabyte, compared to Mosso’s $0.22. Amazon’s prices drop as your bandwidth usage increases, but with the first drop at 10 Terabytes, not a lot of people are likely to notice that difference.

Interestingly, for inbound traffic, Mosso beats out Amazon with a rate of $0.08 per gigabyte to Amazon’s $0.10. This can make a difference if your service takes a lot of uploads or you do a lot of content aggregation. Generally speaking, however, unless your application has a somewhat inverted traffic pattern to most sites, this won’t make a huge difference for you.

…There is one difference between cloud servers and EC2 that can be extremely important. That is that Amazon enforces a strict 5-public-ip rule. This means that your company can only have 5 dedicated public IPs, regardless of how many servers you have. Note that I did not say application, this is a per-EC2-account rule. This can be a problem if your company hosts multiple sites, or if you have other public-IP requirements (such as SSL certificates for multiple sites for example.) Mosso does not have such a restriction. At Mosso, you have one public IP for every instance you have.

Mosso also provides reverse-DNS to your own domain, which can be very important for things such as email delivery. EC2 does not provide any method to edit reverse DNS. Among other things, this makes EC2 not feasible for anything requiring outbound email delivery. You can get around this by having a non-EC2 relay host for outbound email, but in many ways requiring an external host defeats the purpose of working within the cloud in the first place.

…On the whole Amazon EC2 and Mosso are very close competitors. Amazon loses out on CPU power per dollar spent, but Mosso costs a bit more per gigabyte of RAM. Mosso loses out on ‘extra storage’ but wins on general IO speed. Mosso has the advantage on individual server upgrades with it’s instant-upgrade features, but Amazon wins on large-scale deployment because of it’s API and solid auto-scaling options.

So which do you choose? If your application is particularly memory bound, requires a huge amount of disk space, or if you are at the higher end in terms of clustering requirements, Amazon’s EC2 is a good solution.

If that is not the case, however, EC2 just doesn’t hold up, cent for cent, against Mosso’s offerings. In my opinion, the combination of lower cost, better base CPU/RAM options and a smoother upgrade path make Mosso’s cloud-servers the clear winner.

A very interesting post by I Am El Gringo:

For the time-constrained, I submit to you the results of my highly scientific research:

• Yahoo: Minimal Use of tables. I found a picture of Hugh Downs horizontally aligned with it’s caption in a table
• Google Home Page: Not only does Google use tables for it’s iconic home page, it embeds styling in the <td> tags. The horror.
• YouTube: Uses tables for of layout of videos
• Windows Live: Uses tables for footer layout
• MSN: There is one table, but it’s only for stockquotes which is tabular data
• MySpace Semantically pure. MySpace. Whoda thunk it
• Facebook: Does form layout with tables
• Blogger: No tables anywhere on the front page
• Orkut All tables all the time
• Rapidshare: A table with a single <td> for header placement. And again a single <td> table for the central “browse” section. Tsk tsk
• Microsoft: Navigation bar is a table. What did you expect? Unicorns and rainbows?
• Google India: It’s the same Google layout. I wonder if they used copy and paste for the template?
• Ebay: Tables, tables every where
• Hi5: Tables for every thing, pretty much. BTW, I didn’t even know this site existed until last week. Alexa rank 14!?
• Photobucket: Tables for photo gallery layout
• AOL: AOL’s layout is semantically pure! Friggin AOL?
• Google UK: Same GOOG layout. I’m now sure the copied an pasted their html
• Amazon: Now that’s just silly
• IMDB: They used tables for their 3 column layout. What! No CSS framework?
• Imageshack: Semantically pure as the driven snow.
• Finally, even though it’s not on Alexa’s top 20, log in to your Gmail account and look at
  the use of tables

My Hypothesis: Pure CSS design == overcompensation

So, the five companies that use CSS are the web powerhouses–MSN, MySpace, Blogger, AOL and Imageshack. MSN, MySpace and AOL have been maligned for years throughout the web savvy community. My hypothesis is that these companies are overcompensating for the crap that they’ve taken thoughtout the years by designing their site in pure CSS.

Other companies that have more web street-cred like Google and Facebook don’t really have to worry about how the web design community sees them. This leads to things like Google making extensive use of inline styling on their homepage instead of putting it in their stylesheet. I’ve never heard anyone claim that the Google folks are slouches at the web design/development thing. Why is that?

In late 2005 I was working at Bluewall and the owner of the company became convinced that some of the sites we were building were about to go super nova, so he got a second server from Rackspace. This was a stupid decision, but it is a a common one. I find clients often get over-excited about their projects. I’m sure some of this excitement is a healthy part of being emotionally invested in the project that you’re trying to bring to life, but that same excitement can lead to needless expense. Dedicated servers from RackSpace are expensive and, as it turned out, Bluewall’s need for extra servers was years in the future. The purchase decision in 2005 represents a lot of wasted money. So why didn’t the second server get shut down? Because once you have even one (low traffic) site on a server, it becomes a pain to shut down that server.

The Second Road had a similar experience. They started off with 3 dedicated servers, plus a firewall and load balancer, from Rackspace. The cost was $1,800 a month. When we got involved with the project, we moved the site to a server from Hostway, which costs us $150 a month (and on which we have several other websites, all sharing one server). So far, this has met all of the Second Road’s needs on the web.

Dedicated servers represent a lot of potentially wasted resources, especially for a small startup which may grown quickly, but which also may NOT grow quickly.  And what is growth, when you’re talking about traffic? If you get mentioned on BoingBoing, and suddenly you are getting 200 requests a minute, and your server collapses under the load, is it time to get another server? What if the spike in traffic lasts 2 days and then dies away and never comes back? If you get another server based on that one spike, you’ll regret it. Unless the spike is permanent. In which case you’ll regret not getting that extra server sooner.

These experiences have me interested in the new cloud services, which promise just the right amount of computing power that your site needs, no more no less. The idea of the “cloud” is that you can scale your resource needs up or down in increments finer than a whole server. So for the next big client we get we will try a cloud service. These are 3 we are looking at right now:

Mosso (this was just bought by Rackspace)

Amazon

IronServers

There has been a lot of investment, by a lot of companies, in cloud services and, really, it is hard to keep up. The fact that as good a company as RackSpace was willing to buy Mosso speaks well about Mosso. Since there is no way to evaluate all the contenders now competing in this field, we are forced to look at the best known, and then one of the somewhat unknowns, just as a test.

For the Amazon services, one of the best known users is SmugMug. They’ve mostly written about the S3 service, but also some about the cloud service.

Amazon is also now moving into the Content Distribution Network business. This is an aspect of building websites that we haven’t yet researched much. My feeling is that neither us nor our clients will ever be interested in the nitty gritty details of getting content to every corner of the world. We don’t want to be up late at night wondering “Is the site running fast enough in Malaysia?” So we’re pleased to see Amazon in this space and we hope they do a good job, because certainly we will give them a close look, if we ever work with a client who really needs that kind of scale.

This is from the article on Ajaxian:

With nearly 2.5 million requests per day to the jQuery website, the jQuery project team is constantly on the look out for ways to decrease hosting costs while still managing the growing number of requests for the site’s resources. Originally leveraging Amazon S3 for many of their static pages, the project has now turned to Amazon’s new CloudFront CDN. The change has allowed for jQuery pages to be globally hosted as opposed to being centrally located in Amazon’s Seattle-based S3 hosting center.

In tests, John Resig, team lead for the jQuery project, noticed substantial performance gains based on the switch:

“I ran a similar test here in Boston and even managed to see a large improvement. I was seeing latency of anywhere from 50-200ms on Amazon S3, but only a latency of 17-19ms with CloudFront.”

What does all of this mean? It means that the jQuery site is going to load even faster than it does now. We already receive some excellent hosting from Media Temple but being able to off-load these static files to the fast-loading servers will only make for a better browsing experience.

In less than 24 hours the project had received almost 2.5 million requests for over 50GB of data. The only drawback is an increase in bandwidth costs but still substantially less than that of a traditional hosting plan. The jQuery project makes use of the Google AJAX API as well and recommends it as choice for linking to the jQuery and jQuery UI libraries.

I find this quote very interesting:

Business realities force apologies. To cope with these difficult realities, we need code and, frequently, we need human beings to apologize. It is essential that businesses have both code and people to manage these apologies.

…

We try too hard as an industry. Frequently, we build big and expensive datacenters and deploy big and expensive computers.

In many cases, comparable behavior can be achieved with a lot of crappy machines which cost less than the big expensive one.

That’s from Pat Helland an ex-Amazon architect. His point is that businesses can get way with building imperfect systems that will occassionally fail, so long as the business realizes that sometimes it will have to apolgize. The simpler systems will be much cheaper than a complicated system that will offer a higher rate of reliability.