What The Amazon Outage Means For Startups

My blog, which is hosted on Amazon EC2, was down for about 18 hours due to an infrastructure issue at Amazon. I wasn’t alone. Quora, Reddit, and others were down as well.

When I started my blog, it was hosted on a server at Godaddy.com where I had registered my domain name. When I decided I was going to switch, I could have moved to a hosted platform. But I wanted to try out EC2 for a variety of reasons – not because I needed to run on EC2 but because I wanted the first-hand experience with what I consider a fundamental building block of today’s startups – Amazon’s cloud infrastructure. Plus, it was fast, inexpensive, and easy to get started.

There is, however, no button for “make this reliable,” aka, take this application (database, pages, IP address, logins, etc.) that I put on this instance, and make it work no matter what, regardless of whether my instance, storage, or network goes down. And if something bad happens, like a virus, database corruption, or security attack, get me back to where I was before things went bad, and without the bad stuff.

The real issue is that building scalable applications is difficult. People go to EBS because it is supposed to be more reliable, but one thing the Amazon outage showed is that it isn’t. And that means that applications themselves have to be built in a reliable way, or infrastructure vendors have to make it easier and cheaper to obtain reliability through failover and redundancy. The problem is, that’s a hard problem.

The problem is, when did things go bad and what was the cause? Was what looked like a little spike in database and CPU usage actually a bad bug infecting the system? Was it someone who got ahold of the admin password and hacked their way in? Or was it something lower level, like a disk error?

As with the Amazon failure, what should I do? Should I try rebooting? Should I restore a snapshot (whoops… the snapshot is in the same zone as my instance and primary storage and until some changes were made, I couldn’t restore across zones). If I take any of those actions will it improve things or just make them worse?

The Ideal Scenario
What I’d really like is to put my application (in this case, WordPress) out on my instance and know that somehow, magically, it is being deployed reliably, and if anything does go seriously wrong at the application level, I can get back to the point a little while before things went bad, perhaps even changing usernames and passwords before it comes online. In theory, that’s what the cloud is supposed to deliver: highly reliable, on-demand compute and storage capacity. That’s easier said than done. The ideal scenario simply isn’t possible today.

In fact, I really don’t want to have to deal with instances, storage, and networks at all. On Amazon, I have to choose between small, medium, and large instances. But what I really want is to be able to say, give me another unit of compute, another unit of storage. If CPU usage is high, give me another few units on demand and also notify me that something’s going on so I can take a look. And I want all that with the security that separate instances provide: one instance doesn’t corrupt another, apps can be separated from each other, databases can be separated from each other; and my points of failure are reduced.

That in fact, is what EBS is supposed to provide, when it comes to storage. The problem is when the underlying infrastructure has a problem it affects everything running on top of it. What is supposed to be more reliable becomes less so – a single point of failure.

What I need is a hybrid architecture that delivers the reliability of isolation and geographic, network, and hardware diversification, but with the ease of use of true scalability; when I want more units, I get them. But if a piece of underlying infrastructure goes down, it doesn’t impact me. All at the same price I currently pay (or perhaps at a very slight premium).

Throw in easy backup and restore, and a button I can push that will put up a “this site is temporarily down for maintenance” page (hosted elsewhere but without me having to pay for a whole other instance), plus the ability to bring up my new stuff while leaving the old stuff intact until I’ve verified that the new stuff works, and a built-in “scalability” simulator that takes a copy of my entire site, back-end, and infrastructure, right down to the hardware, and tests it at all the different points of failure, and you have a winner.

Sure you can pull a number of pieces together to do all this, but a one-click solution available to web sites large and small? Like I said, easier said than done.

What the Amazon Outage Taught Us
What the Amazon outage really taught us is that being down for more than few hours of late night scheduled maintenance is simply unacceptable for today’s real-time startups. Users don’t care that your infrastructure provider had an issue: they blame you. With users and customers spread all around the world, there is no window longer than a couple hours when it’s acceptable to be down. And even that is pushing the limit. Frustrated, users write permanent posts and tweets, while your numbers suffer. The impact lasts far longer than the time your site was down.

Would those startups that were impacted by the outage be better off running their own hardware and data centers? Not necessarily, for all the reasons they didn’t do that to begin with. It’s expensive, it’s a lot of fixed cost, and it requires significant up front capital expenditure. It also requires not only software but hardware expertise as well – actual data centers and servers, a near constant turnover of old hardware for new, and long-term data center contracts.

No doubt startups that were impacted will be revisiting their architectures, changing their redundancy approaches, and accelerating those long-planned yet oft put-off re-architecting projects that will make them truly scalable and redundant. They may even put in more simulation tools to help them test usage spikes, infrastructure failures, security attacks, and data corruption.

But what they’ll really be hoping for is a solution that gives them the flexibility to scale with the reliability that comes from being isolated from other users of the same infrastructure, down to the physical level, at a price they can afford.

Perhaps after the finger-pointing is over and the black-eyes are done being iced, the greatest outcome will be innovative new technologies, which will eventually trickle all the way down to today’s single instance, single application user, whether it be one of millions of small business owners hosting web sites, or a blogger like me.

Apr 23, 2011

doxo’s Steve Shivers on Fox News


doxo, which provides consumers a secure digital file cabinet for receiving and organizing bills and other important documents, announced in February that it had closed its $10 million Series B financing to support accelerated growth of the company.

Apr 18, 2011

Entrepreneurship: An Art Not A Science

Steve Blank and the Pivot

I was fortunate enough to attend one of the final Lean LaunchPad classes taught by Steve Blank, Ann Miura-Ko, and Jon Feiber at Stanford. One big message was on the pivot. It was incredible. In 10 weeks, students formulated an idea, “got out of the building” to get feedback on it, and then in many cases, pivoted to a completely new idea (or a distantly related one). All in 10 weeks.

Steve sat down with a company founder to brainstorm about his business. He pushed hard on the pivot.

“How long do you wait before you pivot?”

“Only you know the answer to that. That’s what makes entrepreneurship an art, not a science,” said Steve.

Well put.

Apr 6, 2011

5+1 Rules For Building Great Products

At the Pacific Crest Tech Summit earlier this month, Joe Perez, EVP Product at Demand Media gave an excellent talk on building great (consumer) products. Given the controversy around Demand, I wasn’t sure what to expect. But the points Joe made are well worth sharing and I finally found some time to blog about his talk.

Rules for Great (Consumer) Products

  • Universal
  • Simple, easy (one-click benefit)
  • Viral (markets itself)
  • Network effect
  • Collects data (content, behavior, personalization)

The Secret Sauce: Chemistry
And then there’s the last rule, the secret sauce: the chemistry of all these things working well together. The best products have a set of features that work well together. It takes a lot of trial and error to get there. This seems obvious, yet it’s often over-looked. Far too many products are comprised of interesting features that fail to deliver a great experience.

Chemistry defines success.

One example Joe showed was the “food map” on Livestrong.com, which shows “what the world is eating today.” When users describe what they’re eating, that generates more and more digital signals, producing more data, and more insight into users, resulting in…

The FlyWheel Effect
Getting the flywheel effect may not happen overnight. First you have to get the right set of features, which takes a lot of trial and error; then you get to momentum.

Sheer Simplicity
Joe gave a few examples of sheer simplicity:


  • 140 character limit
  • Don’t have to think too much
  • Link is a very simple way to share something you’re interested in
  • So easy to use
  • Users see other users using it, which produces a network effect

Thumbs up, thumbs down – simple.

What else has changed? Mobile.
You can get the Internet everywhere. Users are no longer tethered to their desks. You’re always connected.

Advice for Startups

  • Early on you’re naïve, then you create a business.
  • It’s all about chemistry before you run out of time and money.
  • Focus on one thing. Then pivot if it doesn’t work.
  • Do the “self test”- will I use this product? You can do this cheaply nowadays
  • Launch fast and iterative. Don’t commit to massive releases. Make things smaller, build in bite-size chunks.

One thing that’s hard to capture in text is that Joe talks much the way he builds products. He delivers his messages in clear, bite-size chunks; he’s ultra-focused; and he speaks quickly. His passion for building great products is obvious.

In a world where product innovation is your acquisition strategy, it’s helpful to be reminded of the importance of focusing on chemistry, speed, and simplicity.

Mar 28, 2011

Daily Deal Companies Desperately Need Targeting

Remarkably, both Groupon and LivingSocial sent me offers for Spider-Vein treatments today. This made me wonder:

1) With all the data and big data-crunching resources now available, why aren’t these companies doing more advanced targeting to optimize offers? As far as I can tell, I’m not a likely buyer for Spider-Vein treatment offers.

2) Are they running out of good deals? This seems unlikely given the number of local merchants there are in San Francisco, but perhaps existing merchants are reaching burnout.

3) Is there customer scarcity? Given that one offer came in at 5:43am and the other at 8:31am, perhaps one company was worried that there would be a limited supply of customers for a very similar offer.

4) Was there a commitment that required them to send out the offer to a minimum number of people within a certain time-frame?

Whatever the case, it seems far too unlikely to be coincidence. Here’s hoping for a double burrito or sushi deal tomorrow.

Feb 23, 2011

Product Innovation Is Your Acquisition Strategy

A wave of Web 1.0 offerings are now being rebuilt as truly social, mobile Web 3.0 applications. That doesn’t mean just creating an iPhone app version of an existing site, or adding Facebook Connect as a way to login. It means designing social and mobile in from the beginning.  As a result, many of these new offerings won’t come from existing players, but from innovative entrepreneurs designing social and mobile Web 3.0 versions from the ground up.

Facebook now reaches 600M consumers worldwide and receives some 25% of US pageviews. Apple last month announced the 10 billionth app download. Some 17 million tablets (most of which were iPads) shipped in 2010. This unprecedented growth makes these powerful distribution platforms. Companies have to pay a tax to leverage them, but relative to the large percentage of invested venture dollars that traditionally go into acquisition cost, that tax may not be so bad.

This does not mean traditional online sales and marketing (including SEO, SEM, etc.) are out of the picture by any means. But it does mean that in categories where social, mobile or both can be core, those who leverage these platforms have a significant distribution advantage (and associated lower cost of acquisition) over those who don’t.

In Web 1.0, marketing and sales were responsible for acquisition and monetization. In Web 3.0, product innovation is the acquisition and monetization strategy. Affiliate and referral programs pre-date the web by many years. But the social web means that rather than user and customer acquisition being transactional, it is now an on-going, core part of the product experience.

The hybrid acquisition model, where acquisition cost can be exchanged for valuable digital goods, is at the core of Web 3.0. Consumer IT company Dropbox provides one great example of this: want more storage? Pay for it or just invite a few friends. Where inviting those friends would once have meant importing email addresses and disrupting social norms, those friends are now available at the click of a button; meanwhile, friends and colleagues are used to receiving updates and invites.

The recent success of many social games has shown that people are always searching for a sense of fulfillment and new ways to be entertained. But it’s also demonstrated that putting social at the heart of your application, and using product innovation as the primary way to acquire users, will cause that product to spread like wildfire.

For several years, we were in a period of platform innovation. Now we’re in a period of platform consolidation and application innovation. To call out one interesting example, the iPad, equipped with the right applications, is fast becoming a replacement device for a range of traditional physical items, from books to flash cards.

At the same time, remaining categories of the offline/traditional world are finally making the move online. This is because high speed broadband is widespread, smartphones are pervasive, and, online or off, the web is now a competitive requirement as much as it is a technology advantage.

Opentable moved restaurant reservations from the phone to the web while operationalizing restaurants. Group buying companies such as Groupon, LivingSocial, and others are bringing the web to local, brick and mortar merchants. Zocdoc is web-enabling doctors and making online reservations possible in the process; and companies can hire labor worldwide via marketplaces like Odesk and Rent-A-Coder.

The history of Silicon Valley is the cycle of king of the hill. New kings have accelerated adoption faster than their peers, by putting high leverage acquisition mechanisms at their core. Two high leverage acquisition mechanisms – social and mobile – are now widely available and accessible and can be core to a wide range of online offerings. Marketing and sales now take place through product innovation. Significant capital is required for significant scale, but with the social, mobile web, in many categories, more adoption risk can now be taken out faster and at lower cost. That’s good news for entrepreneurs, investors, and consumers alike.

Feb 22, 2011

The Tech Bubble

“It may be only a few days into 2011, but tech investors seem to be partying like it’s 1999,” wrote the Wall Street Journal yesterday, talking about investors getting lathered up over Tech.

To read the news, the world is facing a tough economic environment. Governments are having trouble servicing their debt. Good jobs are hard to come by. Investors are continuing to hoard gold and cash. Many say that we’re still deep in a recession and the market is going to correct. And for every indicator pointing to a tech bubble, there are a dozen more that indicate there isn’t one.

Some articles are saying it’s 1999 all over again. They’re close. It’s actually more like 1996. After some recent brainstorming with digital media expert Bill Gossman, here are five reasons why:

1.       Tech is sexy again. Apple and Facebook have transformed the mobile and social web, respectively. Apple has made mobile cool and accessible; it has changed the nature of the relationship with wireless carriers, and even though it levies a tax on those who distribute via its platform, it has spawned incredible creativity in the form of hundreds of thousands of applications. And it has transformed the music industry in the process. The company’s market cap is now north of $300B.

If the old web was connecting clients to servers, and people with content, Facebook has made people the new content. What you do, what you post, what you share, who you are connected to is all content on the Facebook platform.

2.       Platform disruption. We’ve just gone through a period of incredible platform invention and innovation. The browser stole the show from Windows, and now Facebook and Apple are stealing the show from the browser. But these platforms are different in one big way that is going to continue to spawn incredible new amounts of innovation: they give you distribution. Sure, they tax you for it, but they deliver incredible distribution. As a result, we’re going to see more innovation, but it’s going to leverage the platform disruption. And did I mention the cloud?

3.       Investors reducing allocations to venture capital. In the early to mid ‘90’s, leading up to the Internet bubble, investors did the same thing. Meanwhile, entry prices on private market investments started to go up, because active investors were more eager than ever to get into deals. Whether Facebook is an anomaly or not, once it’s public, more investors are going to want to get back in on the excitement. That will likely result in more capital going into the private markets, further driving the tech bubble.

4.       Public market tech value opportunities are few and far between. If you subscribe to the Warren Buffet model of public market investing, and you’re looking for value stocks in tech with great growth potential, they’re hard to come by. On the other end of the spectrum, the seed investment market has become similarly highly valued.

5.       The 15 year cycle. Although there is no way to predict the market, there is evidence that IPO cycles tend to peak every 15 years. Enough people have to forget the last bubble for there to be a new bubble.

Now, it’s no sure thing that we’re going into a tech bubble. In fact, a black swan event could create major disruption and shake the market again. But if history is any predictor, a lot of indicators are lining up to pave the way for another tech bubble.

Mobile, social, and cloud here we come.

Jan 10, 2011

Leading Edge Design Principles from Social Games

With social games seeing some incredible growth over the holidays, it seemed appropriate to start the New Year off with a post on the topic.

Today’s social games are transforming not only online gaming itself but consumer and enterprise services as well. Love ‘em or hate ‘em, there’s a lot to be learned from their powerful yet subtle principles.

Old: A progress bar that does nothing more than tell you the game is loading.
New: Graphics illustrating the growth of a city, which send the player a powerful aspirational message.

Old: Your score increases as you earn points.
New: Coins appear on screen and reinforce a reward dynamic by requiring you to click on them.

That’s just the beginning. Here are 28 design principles from social games:

Goals and work – Engagement:
1.       The story is the setup for the game. It tells you who you are and sets the stage for…
2.       The major goal is what players are trying to accomplish and creates a competitive dynamic.
3.       Minor goals + levels provide direction, a way to move up, and a sense of accomplishment.
4.       Work. Only 45% of Americans like their jobs, but in games work is fun and enables earning.
5.       Time. Work takes time, forcing the player to do other activities and return.
6.       Objects. Players prepare, build, or assemble, a form of work, in which nothing comes complete.
7.       Collections. People love to collect not just badges but stamps, art, photos, music, postcards…
8.       Simple animations represent complex actions yet are easy to understand and inexpensive to create.

Viral GrowthAdoption:
9.       Staffing requires other people.
10.   Socially locked items only unlock if you invite friends.
11.   Social goals require the involvement of friends or other players.
12.   Points can in some cases only be earned by involving others.
13.   Gifting items that you must give away or get points for giving away promotes reciprocity.
14.   Sharing photos (screenshots) of scenes/people/pets/ items provides a sense of pride.
15.   Wall posting is free marketing that leverages existing users.
16.   Suggestions to invite friends produce some number of conversions.

Game mechanics – User behavior and monetization:
17.   Sound effects lend authenticity to virtual objects and actions.
18.   Clickable coins, which users must click to get, create a subtle click/reward dynamic.
19.   On-screen goals + locked items constantly remind the user to invite friends + complete objectives.
20.   Success step by step, illustrated during game loading, sends a subtle aspirational message.
21.   Virtual guide makes the game personable and trusted and gives the player someone to impress.
22.   Visiting (virtual guides and real players) reminds users what they are striving for by example.
23.   Limited availability and expiring items create scarcity and a sense of urgency.
24.   Badges are a form of status.
25.   Repeat points reward players for returning multiple days in a row.
26.   Notifications with incentives such as free gifts give users a reason to open emails and click.
27.   Land and expand (not the enterprise sales kind!) causes players to buy items to build and fill space.
28.   In-game purchasing means players don’t need to leave to buy.

These are just a few of the design principles today’s social games are applying. Many have been used by more traditional games for years, but their leverage of widespread social graphs makes them much more widespread and impactful on a much larger user base than ever before. They’re no longer restricted to hard-core gamers. Soon enough, they’ll be everywhere, from the games themselves to just about every web site, service, and mobile offering available. Apply these principles to your service today!

Jan 4, 2011

Five Laws of Successful Digital Media Companies

Venture capital’s newfound love for content and distribution businesses: What’s changed and why.

It used to be that venture disliked investing in content and distribution businesses, especially those that had to reach millions of consumers. That was because content and distribution were incredibly expensive. One had to spend a lot of money on content and then spend two or three times that on distribution, before even knowing whether a hit was possible. That is no longer the case.

Today’s digital media startups are taking advantage of the five laws below:

1. Use low cost, high leverage distribution. Facebook, iPhone, Android, or all three. Reaching the consumer used to be prohibitively expensive. During the dot-com boom, a company would raise tens if not hundreds of millions of dollars to build a brand and reach consumers. Today’s startups leverage platforms that are already reaching millions of consumers.

Companies that don’t start on these platforms leverage them to let their users share and to interact while mobile. Others use these platforms as jumping-off points. Their biggest challenge is crossing the chasm to build their own sustainable brands.

For those that monetize their users, a host of digital advertising, data, and optimization companies provide the tools for highly efficient and measurable acquisition of even more consumers.

2. Deliver low cost content. VC’s used to dislike investing in content companies. Content, from games to news, was prohibitively expensive to produce, and determining whether one had a hit cost huge development and distribution dollars. Not so today.

Today’s user generated content (UGC) comes in many forms: files, links, payments, photos, posts, reviews, texts, tweets, videos, and virtual gifts, to name just a few. UGC content isn’t the only

3. Use game dynamics to promote adoption. On top of low cost distribution and low-cost content, today’s startups use a new form of marketing: gamification to reach and engage users. Virtually every startup is applying game mechanics to reach and engage users.

Startups are using digital incentives from badges to leaderboards to storage to encourage sharing and viral product adoption. Companies get their users to drive more adoption. Network-effects are easier because sharing is supported by the underlying platforms.

4. Substitute for existing time or money consumption. Leveraging low cost content and distribution, today’s startups are replacing more traditional forms of entertainment, information, and services. Zynga, and others are stealing disposable entertainment time from television. While the paperless office is still not reality, startups like doxo, Dropbox, Evernote, and others are transforming Consumer IT at home.

5. Monetize via advertising or direct payments. Paypal has existed now for over a decade, and Google Adsense for more than seven years. But what has really transformed user monetization is widespread consumer acceptance of payments for digital content, from music to video to virtual coins and goods. Today’s startups are also offering hybrid models that let the consumer choose how they want to pay: with their wallet, time, or identity. Startups now allow consumers to buy items for a fee, watch advertisements, or provide identity information to receive offers or enable more targeted advertising. All are viable forms of payment in today’s digital economy.

Conclusion. Startups are leveraging the five laws above to reach consumers faster and more efficiently than ever before. As a result, investing in content and distribution businesses and the tools companies associated with them has become one of the most appealing areas for venture investment, and will likely produce some of the most appealing returns.

Nov 22, 2010

Top 10 Ways to Apply Game Mechanics To Non-Game Services

One of the great benefits of investing in the social gaming space is the opportunity to be immersed in game mechanics. Game mechanics don’t just apply to gaming companies: they apply to virtually every web site on the net.

Here are my top 10 ways to apply game mechanics to non-game services:

1.       Your service: The game. Start thinking of your service as a game and it’s easy to envision all the subtle and not so subtle ways to take advantage of game mechanics in your service — whether it’s a consumer offering or an enterprise one.

2.       Status / reputation. People want status. It’s human nature. But the thing about status is, it has to be visible: both to the person who has it and to everyone else. The driver of the hot yellow Ferrari doesn’t drive it 20 miles per hour down the street so that he can see it: he drives it so that he can enjoy everyone else seeing him driving it. The easiest way to make status visible is through badges. Badges are simple and easy. Start with silver, gold, and platinum. Then add special badges – badges that are only available for completing a certain task, or in the case of a shopping site, a special, rare shopping cart for elite shoppers.

3.       Gifting and reciprocation. Surprise: People love to receive gifts! In the gaming world, that means giving high status users free coins with which they can buy goods – not for themselves, but to send to other people. What makes gifts so powerful is they cause a very personal feeling of reciprocation. In the gaming world, the gift recipients will reciprocate – the key is that they have to pay in order to do so. So by letting your highest status users give away something to others (for free), you get revenue in return.

4.       Hybrid monetization. Let your users choose how they want to pay you: with their time or with their wallet. As long as you get the money, you don’t care whether it comes from your user or from a third party who wants to pay you for access to that user. This doesn’t work for big purchases, but for small purchases, such as upgrades, it can be used to great effect. Let your user choose: they can pay you for the upgrade or they can watch an advertisement. It’s up to them. (Of course, you get to set the price on what the ad is worth.)

5.       Leaderboards and points. People are competitive by nature. Associate points with actions so that people can earn points. Those points don’t necessarily have to convert into anything – simply making them visible (in the form of a leaderboard) gives them value.

6.       Free stuff. Just like gifts, people also like to get good old free stuff. In the gaming world this comes in the form of “150 coins for logging in.” It feels good to get 150 coins for doing… well, almost nothing! Give people free stuff to get them to show up. Give away a little bit of what you offer (storage, for example) to users for completing certain tasks.

7.       Make the virtual real. How do you make a virtual good feel real? Through look and sound. If you’re going to use coins, for example, give them the look and sound of real money. Animate them. Add accurate sound. That’s why Vegas casinos have all that sound of money – because even though you can’t touch it, you can hear it.

8.       Social proof. One other thing that people want: social proof. When users see that their friends are doing something, they want to do it too.  If 80% of people have purchased a certain upgrade, let the rest of your users know that: they’ll wonder why they haven’t.

9.       Create scarcity. In games, this means things have a limited time – the special potion or weapon only has a life of a few minutes or a few hours. The same phenomenon can be implemented in non-game sites: coupons that quickly expire, countdown timers on items so that once they reach zero, the item is no longer available.

10.       A/B test and track the metrics. Game designers are constantly running side by side experiments on everything from the look and feel of their virtual coins to the optimum path for a user to make a purchase in an online store. You can do the same even down to the animated versus non-animated buy button. The key is to measure the results like there’s no tomorrow and track the resulting metrics.

Of course, there are many more ways to apply game mechanics to your service. Hopefully, this list will get you off to a great start.

Sep 14, 2010