It’s as though this week was tailor-made for Bartkiewicz (pictured), who argues that companies in the cloud business–and their customers, too–are in denial about risk. And by risk I mean not the technological possibility that a service may fail to work as advertised, but in the financial liability sense.

In Amazon’s case, there’s not been any real discussion of financial liability. Even though several companies effectively had to pause operations during the period of its outage last week, the only compensation they seem to be getting, at least for the moment, is a credit on their bill for the time that affected systems were offline and an apology. Apologies and billing credits won’t work for large companies. In a case like that, someone, somewhere has to be on the hook financially in the case of failure.

Handing your data over to someone is in a way comparable to handing goods over to a shipping company who promises to get it safely from one place to the other. Something bad can happen along the way, and often does. Trains derail, ships sink or get attacked by pirates. This is why the insurance industry exists. Yes, data is slightly different because it can be copied, but you get the idea.

Anyway, as if on cue, I found in my in-box today a report from Bartkiewicz’s company, CyberFactors, which specializes in risk analysis related to cloud services. It made for very interesting reading: It has estimated the financial costs associated with the Epsilon breach, and the findings should get your attention. The security breach and release of customer data at the email marketing provider has exposed the company to liabilities that could be as high as $225 million. According to CyberFactor’s research, as many as 75 other companies were involved and the total number of affected email addresses may be as high as 60 million.

Dealing with the repercussions of the breach–informing customers about it, making changes to marketing strategies, and so on–could eventually cost those at the affected companies, which included household names like Best Buy, J.P. Morgan Chase, Citibank, Walgreen’s and the Walt Disney Company, as much as $412 million, pushing the aggregate cost of the incident to $637 million. Think about that. The exposure of an email database could wind up costing more than half a billion dollars.

Yet even that isn’t the worst of it. Once you take into account down-the-line costs, such as fines, forensic audits, litigation and loss of business, the total cost could exceed $3 billion. Roughly half of the total costs to the affected companies will occur in the first year after the breach, and the rest will come in the second and third years. Security breaches have a way of costing long after the incident itself fades from the headlines. Cloud companies, CyberFactors argues, are going to have to start thinking more like banks, insurance companies and hedge funds. The cloud is going to have to grow up.

It all started at 12:47 am PT on April 21 in Amazon’s Elastic Block Storage operation, which is essentially the storage used by Amazon’s EC2 cloud compute service, so EBS and EC2 go hand in hand. During normal scaling activities, a network change was underway. It was performed incorrectly. Not by a machine, but by a human. As Amazon puts it:

The configuration change was to upgrade the capacity of the primary network. During the change, one of the standard steps is to shift traffic off of one of the redundant routers in the primary EBS network to allow the upgrade to happen. The traffic shift was executed incorrectly and rather than routing the traffic to the other router on the primary network, the traffic was routed onto the lower capacity redundant EBS network.

EBS works using a peer-to-peer technology that keeps data in sync across several nodes, and using two networks–one fast, used for normal operation, and one slower, used as a backup when the primary one fails. Each node uses the network to create multiple copies of the data being used as needed, and when one node stops talking to another mid-stream, the first one assumes the second one failed and looks for another to replicate to. This normally happens so fast that humans aren’t even involved.

When the network change didn’t work properly, one group of EBS nodes lost contact with their replicas. When their connection was restored, so many had gone down that when they started up replicating again, the available space ran out. That left several nodes in a bad loop looking over and over again for space on other nodes when there was none. New requests to create new EBS nodes piled up, overwhelming everything else. At 2:40 am Amazon disabled the ability of customers to create new volumes. Once new requests stopped piling up, it seemed Amazon had turned a corner, but those hopes were short-lived.

The EBS volumes kept looking for new nodes to replicate to, putting continued strain on the system. By 11:30 am, technicians had figured out a way to quiet things down without affecting other communications between nodes. Once this was done, 13 percent of EBS volumes were still “stuck.” By noon, attention shifted to finding new capacity for the stuck volumes to replicate to. Not easy. That meant undergoing a time-consuming process of physically moving servers and installing them into the degraded EBS cluster. Naturally, it took a lot longer than expected. The process didn’t start in earnest until 2 am the following morning. By 12:30 pm April 22, only 2.2 percent of affected volumes were still “stuck.”

With the new capacity installed, Amazon started work on letting each node communicate normally again. This had to be done gradually, and work to dial it up just right went on well until the early morning hours of April 23. By 6:15 pm that day, operations were almost back to normal–except for the 2.2 percent of volumes that had remained “stuck.” It turned out they would have to be recovered manually. Their data was backed up to Amazon S3, its general storage service. By noon the next day, all but 1.04 percent of that data was recovered.

A more intensive recovery process was tried, and in the end 0.07 percent of the data involved in the crash could not be recovered, Amazon says.

The company says it is auditing its process for carrying out changes in its network, which is where the problem started, and that it will “increase automation” to prevent a similar mistake from happening again. From that statement I gather that it was a human-caused mistake that was then exacerbated by the way the cloud system was designed to work. Customers who used the affected services, whether or not their services were interrupted, are getting a 10-day credit. The list of other changes it is promising is long and detailed, ranging from having more capacity on hand for use in a recovery to making it easier for customers to access more than one availability zone (those who had done prior to the outage fared better than those who hadn’t) to improvements to its status dashboard.

Finally, Amazon apologized:

Last, but certainly not least, we want to apologize. We know how critical our services are to our customers’ businesses and we will do everything we can to learn from this event and use it to drive improvement across our services. As with any significant operational issue, we will spend many hours over the coming days and weeks improving our understanding of the details of the various parts of this event and determining how to make changes to improve our services and processes.

What’s not clear is the effect not only on Amazon and its reputation, but on the planning of customers who rely on its cloud services or are thinking about using them. A failure as widespread and as widely publicized as this may be forgiven but it won’t be forgotten. Mike Rowan, CTO of RatePoint, a reputation management service for small businesses, indicated on Twitter that he was happy with the billing credit. But time will tell if Amazon loses customers over this.

Last week I talked with David Young, co-founder and CEO of Joyent, an Amazon rival with 30,000 customers around the world. His criticism of Amazon in this instance is rather harsh. The way the Amazon cloud is built, he said, virtually guaranteed that a service outage such as this would happen. While on one hand he gives Amazon high praise for “evangelizing the cloud computing model,” with the other he disparages it as “the Atari of the cloud.”

“Amazon does not represent the cloud,” Young says. “These guys are booksellers who got in the cloud business. That’s like Nordstrom’s getting into the cash register business,” he told me. “They may be the market leader, but there’s a bunch of us who are building things in such a way that we don’t see these downtimes.”

It’s hard to miss Amazon’s outsize influence. Though Amazon Web Services amounts to just a fraction of the company’s overall revenue, it is the market leader, controlling about 60 percent of the market with Rackspace, IBM, Joyent and Terremark, recently acquired by Verizon.

Joyent, which is privately held and backed by investments from Greycroft Partners and Intel Capital, would never have suffered an outage, Young claimed, because of the way it is built. Amazon, on the other hand, was never designed for what he calls persistent computing that customers need to be available all the time. The problem, he said, started in Amazon’s Elastic Block Storage, which is vulnerable to being overwhelmed by demand, something he likened to a run on a bank, where depositors panic and rush to withdraw the cash from their accounts.

Essentially, he said, Amazon became a victim of its own popularity, unable to meet the demands placed upon the EBS storage infrastructure by the network, causing in the end a cascading failure. The company has blogged about its theory in greater technical detail here.

Meanwhile, there were Amazon customers who managed to keep their services up despite the crash, and they were comparing notes today. Don MacAskill, CEO of the photo-sharing site SmugMug, blogged about how designing for failure allowed its service to remain live during the crash. Others tweeted, like Mathias Meyer of Basho Technologies, who noted that having their service running in more than one Amazon region helped avert failure.

There were others. Donnie Flood, VP of engineering of Bizo, an advertising service aimed at business executives, said that the company used all of Amazon’s regions except its most recently launched one in Japan, and combined that with a global Domain Name Service system that would direct traffic to the nearest Amazon region. When Amazon’s U.S. East region went down, all traffic in the U.S. was routed to its Amazon instances running in Amazon’s Western U.S. data center. “We were able to stay up fully the entire time,” Flood said.

Oren Michels, CEO of Mashery, a cloud-based manager of software APIs, said that his company had additional cloud infrastructure in place from Internap that took over when Amazon failed. Neither Michels nor Flood said he was likely to move the services currently hosted with Amazon to another provider.

Amazon hasn’t made any public statements about what happened, beyond those made in its status dashboard, and it hasn’t responded to any of my messages seeking a comment on any aspect of this. The company reports earnings tomorrow, so expect some questions on the conference call about Amazon’s terrible, horrible day that stretched into nearly a week.

A failure in the cloud is of course one of the fundamental problems that its critics always point to. Yes, you can save money and time and effort by farming your IT services and infrastructure out to someone else. But when those services crash unexpectedly, you–and scores of others that rely on the same infrastructure–are left to wonder what’s going on and when it’s going to be fixed.

As of now, it seems like Amazon is getting the situation under control, it seems to be getting worse, as other parts of the Amazon service that are tied to EC2 are reporting various failures via Amazon’s status dashboard. Failures are showing up Elastic Beanstalk, and the relational database, and Cloudwatch among others.

Amazon’s status messages are below.

1:41 AM PT We are currently investigating latency and error rates with EBS volumes and connectivity issues reaching EC2 instances in the US-EAST-1 region.

2:18 AM PT We can confirm connectivity errors impacting EC2 instances and increased latencies impacting EBS volumes in multiple availability zones in the US-EAST-1 region. Increased error rates are affecting EBS CreateVolume API calls. We continue to work towards resolution.

2:49 AM PT We are continuing to see connectivity errors impacting EC2 instances, increased latencies impacting EBS volumes in multiple availability zones in the US-EAST-1 region, and increased error rates affecting EBS CreateVolume API calls. We are also experiencing delayed launches for EBS backed EC2 instances in affected availability zones in the US-EAST-1 region. We continue to work towards resolution.

3:20 AM PT Delayed EC2 instance launches and EBS API error rates are recovering. We’re continuing to work towards full resolution.

4:09 AM PT EBS volume latency and API errors have recovered in one of the two impacted Availability Zones in US-EAST-1. We are continuing to work to resolve the issues in the second impacted Availability Zone. The errors, which started at 12:55AM PDT, began recovering at 2:55am PDT

5:02 AM PT Latency has recovered for a portion of the impacted EBS volumes. We are continuing to work to resolve the remaining issues with EBS volume latency and error rates in a single Availability Zone.

Update: Here’s more companies that are affected by the outage, according to status updates on Twitter.
Hootsuite, the cloud-based Twitter client, is down because of the outage too. Here’s what the site looks like right now. Also down is the Hootsuite URL shortener ow.ly.

SCVNGR is reporting that it is down too as a result of the outage.

Discovr, an iPad music app, reported that it went down, but shortly reported its service was restored.

Wildfire, a social media app, reports that it is down.

Livefyre is down.

Here’s an interesting one. CampgroundManager.com, apparently a software-as-service application used to manage campgrounds, say it is down.

A service called Totango, which appears to do something with managing customer relations and subscriptions, had some issues, but moved some things around, and got things mostly working again.

ESchedule, a Canada-based employee scheduling service, reports its service is down.

ZeHosting, a Web host, says it is experiencing slowdowns.

Recorded Future, which bills itself as a “temporal analytics engine” is reporting an outage.

PercentMobile, a mobile analytics firm, say its service is down.

The Cydia Store, which hosts applications available for jailbroken iPhones, reports it is down.

Here’s the latest update from Amazon:

6:09 AM PT EBS API errors and volume latencies in the affected availability zone remain. We are continuing to work towards resolution.

Here’s another update from Amazon

6:59 AM PDT There has been a moderate increase in error rates for CreateVolume. This may impact the launch of new EBS-backed EC2 instances in multiple availability zones in the US-EAST-1 region. Launches of instance store AMIs are currently unaffected. We are continuing to work on resolving this issue.

Another Amazon update:

7:40 AM PDT In addition to the EBS volume latencies, EBS-backed instances in the US-EAST-1 region are failing at a high rate. This is due to a high error rate for creating new volumes in this region.

Updating again at 9:10 AM PDT

Amazon continues to post regular updates on its multi-faceted cloud services outage this morning. The latest update message came in about 15 minutes ago, and is reprinted below.

I’ve heard about three other sites that are affected by the outage. Radarsync, a cloud-based service that updates drivers for Microsoft Windows users tells me via Twitter that its service is down. I’ve also seen that Thrillist is having some troubles sending emails. Venmo, an iPhone-based payment service is also down.

This is an update on Amazon’s Relational Database service.

8:12 AM PDT Despite the continued effort from the team to resolve the issue we have not made any meaningful progress for the affected database instances since the last update. Create and Restore requests for RDS database instances are not succeeding in US-EAST-1 region.

And this is one from the EC2 team.

8:54 AM PDT We’d like to provide additional color on what were working on right now (please note that we always know more and understand issues better after we fully recover and dive deep into the post mortem). A networking event early this morning triggered a large amount of re-mirroring of EBS volumes in US-EAST-1. This re-mirroring created a shortage of capacity in one of the US-EAST-1 Availability Zones, which impacted new EBS volume creation as well as the pace with which we could re-mirror and recover affected EBS volumes. Additionally, one of our internal control planes for EBS has become inundated such that it’s difficult to create new EBS volumes and EBS backed instances. We are working as quickly as possible to add capacity to that one Availability Zone to speed up the re-mirroring, and working to restore the control plane issue. We’re starting to see progress on these efforts, but are not there yet. We will continue to provide updates when we have them.

Update at 10:17 AM PDT: Here’s more companies affected by this outage, as offered by contributors to the comments below: ELog.com, a sort of all purpose-notepad in the cloud is down (link goes to a Google cache).

About.me, which AOL acquired last last year is down, and is currently displaying a message saying “We are currently experiencing an outage.”

ECairn, a social media marketing app says it’s down.

Travelmuse, a vacation-planning site is down, though I can’t find an official update on Twitter confirming that it’s connected to Amazon’s troubles.

Web host and design firm Drupal Gardens has an blog entry on its partial outage.

PeekYou, a search company that specializes information about people, tells me it has experienced some trouble, but has shifted its hosting to compensate.

Gamechanger.io, a service that tracks live baseball scoring stats, is down.

I’m beginning to understand what it feels like to be a radio announcer on a snow day reciting school closures after another! The only thing is, there’s no kids cheering.

Update at 10:39 AM: Yet more communication from Amazon, who says it is making progress.

10:26 AM PDT We have made significant progress in stabilizing the affected EBS control plane service. EC2 API calls that do not involve EBS resources in the affected Availability Zone are now seeing significantly reduced failures and latency and are continuing to recover. We have also brought additional capacity online in the affected Availability Zone and stuck EBS volumes (those that were being remirrored) are beginning to recover. We cannot yet estimate when these volumes will be completely recovered, but we will provide an estimate as soon as we have sufficient data to estimate the recovery. We have all available resources working to restore full service functionality as soon as possible. We will continue to provide updates when we have them.

And another concerning the relational database.

10:35 AM PDT We are making progress on restoring access and IO latencies for affected RDS instances. We recommend that you do not attempt to recover using Reboot or Restore database instance APIs or try to create a new user snapshot for your RDS instance – currently those requests are not being processed.

There have been numerous complaints on Twitter like this one from people saying they are having access issues. At least one company, Yola.com, is unable to respond to customer support requests as a result. You can see the action unfolding on Twitter here. Let me know in the comments below if you’re among those affected.

Salesforce said first it was having trouble with its NA1 instance, fixed it, and then started having troubles with its NA7 instance. Other instances seem unaffected.

It’s not clear at the moment how many people are affected, as the other seven instances running in North America are showing green.

The timing isn’t good because Salesforce has just started telling customers about its Spring 11 update, the first of its three-times-a-year upgrades, which is due next month.

I’ve got calls into Salesforce for a further explanation and will update as I learn more.

Update: It’s fixed as of 11:14 AM Pacific Time. And here’s a statement from Salesforce corporate VP Jane Hynes:

“At salesforce.com, nothing is more important than the continued success of our customers. Salesforce.com has set the standard in the industry for transparency with our customers about the availability and performance of our systems with trust.salesforce.com.”

That doesn’t say much about what happened, or what the root cause was. I’ve asked. More as I get it.

CIO Lars Rabbe says in a blog post that a set of support servers responsible for Skype instant messaging became overloaded, and as a result sent delayed responses. A bug in the latest Windows version of the Skype desktop software failed to process these delayed responses, causing them to crash. About half of the world’s Skype users who were signed on at the time the problem began were using that version of the software, and of those, about 40 percent crashed. Among them were users whose machines were serving as supernodes. Rabbe says as many as 30 percent of the Skype network’s supernodes were among the crashed machines.

Losing those supernodes increased the load on other still-functioning supernodes, which was compounded by all the crashed Windows users trying to restart their software and get back on the network. He says traffic to these supernodes surged to 100 times normal volume for that time of day.

What he doesn’t go into great detail about was why the instant messaging servers became overloaded in the first place. Was this another bug in the server software? It’s a little unclear from this explanation.

Rabbe says Skype is trying to learn from the incident and has instituted new procedures to try to prevent this sort of thing from happening again. But this can’t help but hurt its reputation as it looks for ways to diversify its base beyond the millions of free users it has and make some actual money.

The whole reason Skype is supposed to work as well as it usually does is the strength and resilience of the network, and the fact that the network gets stronger as more people are signed on to it. To say that two bugs in a strange confluence of events could bring that entire network down raises a lot of fundamental questions about Skype.

Rabbe says an investment program to increase capacity to support paid consumers and enterprise customers is underway and will continue into 2011. I’m betting Skype will speed it up.

Also still outstanding from CEO Tony Bates and his team is a full explanation about what caused Skype to fail for the better part of two days. He promises that will be coming early next week.

Offline instant messages and group video chat services remain offline, he said.

Bates said Skype now knows what caused the crash, but he didn’t disclose it. He ruled out the possibility of some kind of malicious attack, and said it’s conducting a detailed postmortem.

This would probably be the worst time for Skype to experience a high-profile outage. Though the Skype service is working today, lots of people who might have used it to call family members heading into Christmas may have made alternate plans.

However, the failure, whatever its cause, is also a reminder that Skype isn’t always in charge of its own ability to stay online. In 2007 an otherwise routine Windows security update issued by Microsoft forced an abnormally high number of PCs running Skype around the world to restart at roughly the same time. A software flaw prevented the Skype peer-to-peer network from compensating properly and the service crashed for two days.

This incident will also hurt its reputation with two key constituencies: Prospective business customers and potential investors. Business customers will rethink plans to deploy Skype. And potential investors will question whether this company has its act together, hurting the potential benefit from its forthcoming IPO.

To its credit, Skype did manage to restore service much faster than it did in 2007, as SkypeJournal notes here.

System failure is one of the risks that Skype admits to in its S-1 filing with the U.S. Securities and Exchange Commission. Of the 2007 failure, Skype says in its filing:

“We experienced significant adverse publicity and lost net revenues as a result of this outage, and any similar outage in the future would likely harm our business. As we increasingly introduce products particularly targeted at business customers, any system failures could have a significant impact on our ability to attract or maintain our relationships with business customers.”

Bates’s video message to customers is below.

BoomTown has always enjoyed–although I have not always agreed with–the ruminations of Fred Wilson in his must-read blog, A VC.

Today, the New York venture investor–heard of Foursquare, Twitter?–penned one that was flatly on point, simply titled “Chasing Returns” about a potential crisis in start-up funding.

It was all due to a new class of investors too focused on chasing returns, noted Wilson, “and some of them do not understand what they are investing in.”

Rut-roh.

Perhaps dumb moneybags are not the greatest worry of our time–but dumber justifications of questionable success certainly should be, which Wilson also called out recently in an article in the New York Observer about the bubbly start-up scene there.

Said Wilson:

“It’s not all going great. You know, companies are failing. A couple of high-flying entrepreneurs came crashing to the ground recently. Justin Shaffer of Hot Potato and Sam Lessin of Drop.io–both of those companies essentially failed. Both of them ended up ‘selling’ their businesses to Facebook, but those were really just–Facebook wanted to hire those people, and they wrapped it up in a ‘sale.’ But those companies were unsuccessful. They failed. So there is failure out there–like, right in front of us. We can see it.”

Many quickly cried foul after Wilson’s remarks, but many more secretly shook their heads in agreement at the reality distortion field around entrepreneurs.

This has happened a lot lately in Silicon Valley, of course–as evidenced by a series of just-in-time saves of once high-flying start-ups (Slide being bought by Google comes to mind) sold as wins.

Indeed, painting failure as another form of success is one of the favorite and endearing canards of tech, showing an ability to bounce back from any negative and pivot into a new direction.

But most pivoting, as Loren Feldman of 1938 Media recently pointed out, is just another word for covering up misdirection.

Perhaps misdirection–as it should be in any world that thrives on innovation and entrepreneurial instincts–is all well and good, but it is only that way as long as it is identified as such when it occurs.

So, good for Wilson for truth-in-labeling when it comes to start-ups, investing and chasing returns these days.

Yesterday marked the end for Xmarks, the Mitch Kapor-backed social bookmarking start-up that was founded in 2006.

What was most remarkable to BoomTown was the classiness and honesty of the goodbye, especially in Silicon Valley, which is loath to call a failure just that.

That was certainly clear in a terrific blog post about its history, titled “End of the Road for Xmarks,” written by its CTO and co-founder Todd Agulnick.

After noting Xmarks’ substantive growth as a browser synchronization service, he wrote:

“Tomorrow, however, will hardly be anything but typical, for tomorrow one of our engineers will start a script that will email each of our users to notify them that we’ll be ceasing operations in around 90 days.”

In the fascinating post, Agulnick did note that the company came close to selling recently. Actually, I heard it had gotten close twice and to no avail.

The San Francisco-based company–which had been called Foxmarks initially–had been seed-funded by Kapor, the well-known tech entrepreneur, and also got an additional investment from First Round Capital.

Xmarks garnered another $5 million in funding from Redpoint Ventures in 2008,

That year, it also hired Silicon Valley entrepreneur James Joaquin as CEO, whose job it was to carve out a business with Xmarks’ assets, including using its mass of data.

Xmarks had certainly been growing its user base and bookmarked Web addresses strongly, via a browser widget that recorded bookmarking information.

Earlier this year, it tried out an advertising product called SearchBoost, which gave advertisers additional analytics about their ads, as well as organic search results.

But all that ultimately did not translate into a viable business model for Xmarks.

At the time of launching this money-making effort in April, Kapor said that after growing its user base of actives, this was the next logical step for Xmarks.

“This is the effort to move from that category to the category of sustainable enterprises,” he said. “And that is certainly a good thing.”

But, as Agulnick concluded:

“We built it and it put it front of potential advertisers. Many were interested, but ultimately the feedback was negative: our user base was too small to be worth their time and attention.”

After thanking investors, employees, users and others, Agulnick ended:

“In the words of Douglas Adams, so long and thanks for all the fish.”

Indeed.

Looking back to happier times, here is a video interview I did with Joaquin in late 2008 about Xmarks’ prospects: