The Storage Games

Most of the computerized data you interact with is stored in a corporate data center or the cloud, on a class of device known as enterprise storage. Their capacity is measured in petabytes, or millions of gigabytes. The number of input/output operations per second (IOPS) generated by applications from Excel to Facebook would boggle your mind.

In response, the once-lethargic $20+ billion enterprise storage industry is exhibiting unprecedented innovation. Giants like EMC and Dell are vying with, partnering with and acquiring start-ups for supremacy in a morphing landscape.

It’s a serious game. More than $3.5 billion was pumped into VC-backed storage start-ups between 2007 and 2011, with more than $1 billion in 2011 alone. And $10 billion more has been poured into M&A, with the most recent example being EMC’s $430 million purchase of a company that hasn’t finished developing an initial product. Tectonic shifts come from collisions of forces. There are three major force vectors here.

First is the demand for more scalable, instantly provisionable, faster and higher-capacity storage. Even in an anemic economy, demand for data storage grows more than 50 percent per year.

Storage in a Flash

The second driver is the widespread proliferation of flash memory. Remember when you could feel the hard drive spinning inside your iPod? Today, practically every consumer carries flash memory in his or her pocket or purse.

Flash has been around for years, but was too expensive for broad adoption. Thanks to companies like Apple, which consume enormous amounts of flash, the cost is dropping like an apple from a tree. It’s still much more expensive than rotating hard drives, but its notable physics are compelling. Solid-state flash is faster than mechanical drives, and doesn’t forget everything when the power is turned off. Perfect, right?

Flash in the Pan?

Actually, not so perfect for corporate, governmental or cloud environments. In addition to high cost, flash has some unfortunate features. For example, it wears out in the same way your favorite pair of jeans will become threadbare with use. You’ll never wear out your phone from too much texting. But in an enterprise application, the number of IOPS can be so staggering that flash has to be treated almost like a printer cartridge, a consumable.

Many companies are developing techniques to deal with flash’s inherent limitations to make it suitable for data centers. It’s a gold rush, with vast sums of capital chasing big markets. That $430 million acquisition by EMC? Yup, flash.

Why pay so much for a pre-product company? There were multiple bidders. NetApp made a rich offer, which Dell topped by a lot, which EMC topped by an equally wide margin. There will be more M&A.

Interestingly, this activity is driving breakneck commoditization. This is great for customers, but not for vendors, who will not long enjoy rapacious (oops, I meant healthy) margins on proprietary technology. Ironically, the value in flash-based systems is really in the software that wrests the value from the hardware. Everyone in the industry knows that the days of differentiated flash hardware are numbered.

When Is Storage Not Real? When It’s Virtual.

As if hot, high demand and cool flash aren’t enough, the storage games are impacted by a third force called virtualization. Virtualization has transformed computing. The leading vendor, VMware (not coincidentally, owned by storage company EMC), has built a market capitalization of roughly $40 billion. All by making fake computers.

We call them virtual machines. Your iPhone may be talking to one right now over the Internet. Their magic allows the creation of what looks like a physical computer server. A virtual machine, or VM, appears to embody central processing units, memory and communication networks like physical computers. But it’s a software abstraction. Through this prestidigitation, data centers run scores of VMs on a single server box.

Enterprises can deploy vastly more applications because virtualization from Microsoft, Citrix, Red Hat and VMware saves enormously on capital and operating expenses. And provisioning is so much faster. Just a few clicks and, voila, you have a new server. By the way, you can buy server hardware from anyone. That freedom makes vendors compete harder, which you like if you run an information technology department.

What does this have to do with data storage?

Enterprise and cloud storage still live in the physical age. That is, storage system features are embedded in proprietary hardware. Want a cool software feature? You have to buy hardware to get it. Before virtualization, this was how the server industry worked. But virtualization is stressing traditional modes of delivering storage to applications. Performance problems, high costs and inflexibility cause VM users great pain on a daily basis.

Hello, Storage Hypervisor

The key enabling technology in compute virtualization is called a hypervisor. This core magic remade the server industry for the benefit of all. Until recently, there was no storage equivalent.

Now the storage industry is beginning to buzz about the concept of a storage hypervisor — the analog of the server hypervisor, but for storage. Storage hypervisors promise to increase the effective performance of hardware by an order of magnitude. By virtualizing resources to provide the administrative paradigm needed in virtualized environments — VM-centric management — they provide unprecedented flexibility and efficiency. Naturally, the ideal storage hypervisor leverages flash, just as server hypervisors unleash the power of Intel-based silicon.

Giant publicly-traded storage vendors and ambitious start-ups alike are talking up their offerings. All have differing approaches, but share the goal of giving data-center storage buyers the benefits already bestowed on server customers.

A Serious Game

Today we observe a convergence of forces transforming a multi-billion dollar market. The unending pressure for more data increases demand for high-performance flash-based storage hardware. This in turn is driving the essential requirement for virtualization of storage hardware resources. This confluence will enable vastly larger amounts of storage to be applied to every imaginable use case, all while making the economics not only affordable, but also compelling.

The winners in this game? Corporate and cloud data centers and their users. In other words, you.

Before co-founding Virsto, Mark Davis was CEO of storage resource management vendor Creekpath, where he engineered its acquisition by Opsware (now HP).

Must-Reads from other Websites

Panos Mourdoukoutas

Why Apple Should Buy China’s Xiaomi

Paul Graham

What I Didn’t Say

Benjamin Bratton

We Need to Talk About TED

Mat Honan

I, Glasshole: My Year With Google Glass

Chris Ware

All Together Now

Corey S. Powell and Laurie Gwen Shapiro

The Sculpture on the Moon

About Voices

Along with original content and posts from across the Dow Jones network, this section of AllThingsD includes Must-Reads From Other Websites — pieces we’ve read, discussions we’ve followed, stuff we like. Six posts from external sites are included here each weekday, but we only run the headlines. We link to the original sites for the rest. These posts are explicitly labeled, so it’s clear that the content comes from other websites, and for clarity’s sake, all outside posts run against a pink background.

We also solicit original full-length posts and accept some unsolicited submissions.

Read more »