The carrier noted its overall 99 percent uptime, and characterized the outages as the growing pains that come with the territory as it pioneers a new 4G LTE network.

Each of the outages, Verizon said, had distinct causes.

Here’s the full release:

Statement From Verizon Wireless on 4GLTE Network

12/29/2011

In light of recent events, Verizon Wireless shared the following statement about its 4GLTE Network:

The Verizon Wireless 4GLTE Network is BY FAR the largest and the most advanced 4GLTE wireless network in the world. It is available in 190 US markets and covers more than 200 million people, providing the fastest 4G Network in the US.

Being a pioneer comes with growing pains. The recent issues that affected our customers’ 4GLTE service were unforeseen despite careful, diligent planning, deployment and ongoing upgrade programs.

Problems customers experienced affected connectivity to the 4GLTE Network and data service. Several times, we have proactively “moved” 4GLTE customers onto our 3G Network to ensure all would have a data connection. For brief periods, such as on Wednesday (12/28), 4GLTE customers could not connect to the 3G Network as quickly as we would have liked.

Nonetheless, we estimate that 4GLTE connectivity has been available approximately 99 percent of the time this year.

Why have these issues occurred with our 4GLTE Network? Each incident has been different from a technical standpoint. Our engineers have successfully diagnosed those past triggering events, and they have not re-occurred. We also work diligently to rectify technical problems in the Network before they affect any customers.

Our 3G and 1X Networks continue to reliably process calls, texts and data for customers with 3G devices and, when necessary, 4GLTE devices. It continues to perform at the high level that established it as the nation’s largest and most reliable 3G Network.

We are taking a number of steps, working closely with our network suppliers, to ensure the integrity of our 4GLTE Network. We continue to fortify and improve its performance, and our goal is that our 4GLTE Network meets the same high standards that our 3G Network has set for performance and reliability.

Among the numerous measures we have taken or will take are: geographic segmentation, which enables us to isolate, contain and rectify network performance issues, and maintain service to the majority of customers when an issue does develop; and software fixes that we have developed, tested and applied regularly – and will continue to do so. Both will improve performance and reliability.

And finally, we are learning from these issues and applying the same gold standard to our 4GLTE Network that make our 3G Network the nation’s largest and most reliable.

Verizon Wireless is a leader and pioneer in this cutting edge technology that provides very fast wireless data speeds, enabling customers to enjoy the best experience in video and other wireless data usage. Clear unbuffered streaming video, super fast file downloads and wide availability are among the advantages we offer to customers. The capabilities of 4GLTE have exceeded many expectations.

We will not rest until our 4GLTE network performs at the very highest levels that our customers have come to expect from us.

Users in Alabama, California and Indiana, among other places, are taking to the company’s support Web site to document their connection woes.

The company had an outage last week and another earlier this month.

With the past outages, Verizon has confirmed the issues but declined to discuss the specific causes. A Verizon representative was not immediately available for comment on the latest outage.

Update: Verizon has provided the following comment:

We are investigating reports of some customers experiencing trouble accessing the 4GLTE network. The network itself continues to operate and all customers continue to be able to make calls, send text messages and utilize data services. 3G devices are operating normally.

Kleiner Perkins has nabbed former Twitter engineering head Mike Abbott, who left the social communications company less than two months ago to be an entrepreneur in residence at Benchmark Capital.

(Well, that didn’t last long, Mike, but maybe the food was better at 2750 Sand Hill Road than at 2480 Sand Hill Road.)

In an interview this morning, Abbott said that he hopes to stay a VC for 20 years (yipes!), since it allows him to work closely with a wide range of entrepreneurs and also get a broad view across a spectrum of businesses.

“I am really energized about what’s been happening in a lot of places like software,” he said. “From my experience, I think I bring a lot of differentiation for the companies Kleiner is invested in.”

And tech cred too. “We think engineers will be thrilled to have access to Mike and he’s a magnet for talent,” said Kleiner partner Ted Schlein, who compared him to all the comic-book heroes, The Avengers, in one person. “Mike is multi-faceted.”

Abbott was indeed a high-profile hire for Twitter a little over a year ago from Palm, where he served as head of its software and services, in charge of its webOS mobile platform.

He was brought in to provide a level of discipline and reliability to the Twitter communications platform and service, which had been plagued by persistent outages that made the Fail Whale infamous.

Abbott will focus on social, mobile and cloud investments at the well-known Silicon Valley venture firm while working on a team that includes high-profile players Mary Meeker and Bing Gordon.

Here’s the official press release:

Michael Abbott Joins Kleiner Perkins Caufield & Byers as Partner

Engineering Leader to Help Social, Mobile and Cloud Entrepreneurs Build Teams and Ventures

MENLO PARK, Calif., December 1, 2011 – Kleiner Perkins Caufield & Byers (KPCB) today announced that Mike Abbott, former vice president of engineering at Twitter, has joined the firm as a partner on its digital team. Abbott led the building of innovative, high-performance applications and services at Twitter, Palm and Microsoft. With a deep background in social and mobile applications and infrastructure, Mike is also an expert in enterprise infrastructure and cloud computing and “big data” businesses, having founded Composite Software, and advised Cloudera and Jawbone.

“I’m excited to join KPCB’s partners to build new ventures faster,” said Abbott. “The partner mix of founders, operators and investors is ideal for entrepreneurs racing to scale at this disruptive time.”

“Mike is an exceptional and well-respected leader with an outstanding track record shipping great products,” said Ted Schlein, partner, KPCB. “Mike’s deep expertise from Palm and Twitter will help social, mobile and cloud entrepreneurs win.”

Dick Costolo, CEO of Twitter, said, “Mike is a huge engineering talent and will be a terrific asset to Kleiner’s technology companies. He was instrumental in helping us scale Twitter’s architecture to support incredible growth ̶ from 100 million daily Tweets in January 2011 to about 250 million daily tweets today.”

In less than a year and a half, Abbott grew the Twitter engineering team from 80 to more than 350 engineers in an intensely competitive recruiting market. Abbott’s team rebuilt and solidified Twitter’s infrastructure. Prior to joining Twitter in 2010, Abbott led the software development team at Palm that created HP/Palm’s next-generation webOS platform. Abbott was previously the general manager at Microsoft for .NET online services, which became Azure. Prior to that, he co-founded Passenger Inc. and founded Composite Software. Abbott has advised and invested in numerous software companies such as Cloudera, Hearsay Labs, Saynow and Jawbone.

Mike Abbott is just the third senior KPCB partner added in three years, joining Bing Gordon and Mary Meeker, each with exceptional records serving mobile, social and cloud entrepreneurs. KPCB’s digital team also bolstered its infrastructure expertise with the recent addition of Ray Bradford from Amazon Web Services, where he helped grow the company’s cloud database business.

The company posted a statement on its Twitter account saying it is investigating the issue and promising a speedy update.

However, the Waterloo, Ontario, company sought to downplay concerns it could be experiencing a repeat of last month’s massive outage that crippled the company’s service globally, led to a huge stock plunge and put a huge stain on the company’s reputation for delivering rock-solid service.

“There is no system-wide outage,” RIM said in a statement to AllThingsD. “However we are investigating reports that some users in (Europe, the Middle East, India and Africa) have experienced delays.”

Update, 3:45 p.m.: RIM says service is now operating normally in that region.

Reports started cropping up around 11 am PT from users around the U.S. saying that Siri — which, to be fair, is still in beta — was unable to connect to its servers to complete searches. The problem appeared unrelated to geography or carrier, but seemed to be affecting significant numbers of users intermittently.

While I was able to compose and send a few text messages through Siri, it was unable to complete any tasks that required searching the Web for information, like answering the queries “What’s the weather like in Paris?” or “What’s Apple’s stock trading at?”

Apple has not responded to requests for comment.

One final thought: Siri’s error message in cases like these should always be “I’m sorry, I can’t do that right now, Dave” — regardless of user. What a missed opportunity for more “2001” high jinks.

“We have an unprecedented track record of reliability,” Bates told Mobile News, evidently without a hint of irony. “Our focus is on providing the best communications experience and the feedback we get from our customers is we normally deliver on that. BlackBerry is about the ultimate communications platform. There is a market for that, people see the value in it and will choose that over other platforms.”

That’s the hope, anyway. And as Bates notes, RIM’s network uptime does average about 99 percent. Not too shabby.

That said, it’s certainly possible that the duration of the latest outage will inspire businesses and consumers alike to rethink their use of the BlackBerry. The quality and reliability of the device’s email and messaging system has long been its biggest selling point. But after this outage, you’ve got to imagine it’s a bit tarnished.

Early on Monday, the company announced it would give BlackBerry users a package of free apps in an attempt to mend fences with angry customers.

In a statement, RIM said it was offering more than $100 worth of apps “as an expression of appreciation for their patience during the recent service disruptions.”

“Our global network supports the communications needs of more than 70 million customers,” RIM co-CEO Mike Lazaridis said in a statement. “We truly appreciate and value our relationship with our customers. We’ve worked hard to earn their trust over the past 12 years, and we’re committed to providing the high standard of reliability they expect, today and in the future.”

The apps will be made available for download beginning on Wednesday, with more rolling out over the coming four weeks. Titles include the games Sims 3 and Bejeweled, utilities such as translation and photo editing apps, and entertainment titles like Shazam Encore and Nobex Radio Premium.

Business customers will also get one free month of technical support, RIM said.

“We are grateful to our loyal BlackBerry customers for their patience,” added Lazaridis. “We have apologized to our customers and we will work tirelessly to restore their confidence. We are taking immediate and aggressive steps to help prevent something like this from happening again.”

Whether these steps will be enough to hang on to the customers, though, is an open question.

According to sources close to the situation, Twitter VP of Engineering Mike Abbott has left the company.

He is apparently interested in doing more investing.

[UPDATE: Twitter confirmed the departure in a statement, noting that Abbott was joining Benchmark Capital as an entrepreneur in residence. Benchmark’s Peter Fenton is on the Twitter board.

“We can confirm that Mike Abbott is no longer with Twitter. Mike will be available to assist the company in his new role as an Entrepreneur in Residence at Benchmark Capital. We thank Mike for his leadership in growing an outstanding team of engineers from 100 to 400 that have successfully taken on the unique challenge of making Twitter reliable and stable. While Mike led the engineering team, Twitter grew from handling 55 million tweets a day to 230 million. Twitter is focused on aggressively moving forward to fulfill the profound opportunities that are in front of us.”

Said Fenton:

“What Mike Abbott accomplished at Twitter is nothing short of heroic, when Benchmark introduced him to the company we felt he had the courage and entrepreneurial drive to build a strong engineering foundation. Today we applaud him for that extraordinary accomplishment. As we continue to work closely with the team at Twitter, we’re delighted to welcome Mike to Benchmark as an entrepreneur in residence. Mike will not only continue to lend his considerable talents to supporting the team at Twitter, but as a true entrepreneur at heart, will undoubtedly be an incredible asset to our entrepreneurs and our portfolio overall.”]

The move feels sudden, but sources said it has been brewing for a while and happened several days ago.

Abbott was a high-profile hire for Twitter a little over a year ago from Palm, where he served as head of its software and services, in charge of its webOS platform.

He was brought in to bring a level of discipline and reliability to the Twitter communications platform and service, which was plagued by persistent outages that made the Fail Whale infamous.

In large part, Abbott has done that, as Twitter has grown significantly. It now has 230 million tweets per day, he said in a recent interview.

Twitter does not have a replacement as yet, sources said.

“BlackBerry services are operating well globally,” the company said in a statement. “Some customers in Canada and Latin America who are sending messages to other regions may see intermittent message delays. Support teams are actively addressing this.”

The outage began earlier this week after what RIM said was a core switch failure followed by a problem that caused backup systems not to kick in as anticipated.

The company held a conference call, which AllThingsD covered.

Earlier on Thursday, RIM posted a video apology from co-CEO Mike Lazaridis, as well as an update noting that it is starting to see service levels improve.

Here’s the liveblog of the call:
7:06 am: Call kicking off now.

7:07 am: Both RIM co-CEOs are on the call, along with CTO for Software David Yach.

Lazaridis: I want to personally update you on what is going on.

“Before I do, I want to personally apologize to all of the BlackBerry customers we have let down.”

As he noted in his video, “You expect better of us. I expect better of us.”

“We’ve now restored full services.”

7:09 am: Going through what happened. Also says taking immediate steps to make sure the same thing doesn’t happen again, and to restore trust.

We’re working with vendors to prevent the failure mode that happened, conducting analysis of why system took longer than expected to restart.

7:10 am: On to Q&A.

7:11 am: First question on whether recent layoffs affected response.

Lazaridis: No. “That would not have affected this.”

7:11 am: Second question: Are you willing to talk about reparations, and do you have arrangements with customers guaranteeing service?

Co-CEO Jim Balsillie: Focus has been on restoring service, though he did say the company does have service-level agreements.

7:12 am: Was this incident preceded by any changes to the network, and who is the vendor of the switch that failed?

RIM: No recent changes were made. The system is provided by multiple vendors, and until the company has the root cause, it is not identifying those vendors.

7:14 am: Are there any customers still experiencing issues?

Lazaridis: All the services are back up globally.

7:15 am: With the outage starting on Monday, why was the video apology not posted until Thursday?

Lazaridis: “I was very much involved in the management of the procedures to bring (the services) up.”

Another RIM executive notes that Lazaridis was directing the team responding to the issue, and notes that no one at RIM has been going home.

7:17 am: As I am sure you are aware, you have a lot of angry customers asking, “How are you going to make this up to us?”

Lazaridis: “That is something we are now turning our attention to.” Our priority right up until this moment was making sure our system was up and running.

In response to another question, Lazaridis notes this was the largest outage in the company’s history and the first major disruption in 18 months.

7:20 am: Another compensation question. This time asked about how RIM will compensate carriers.

Balsillie: “I’ve talked to global CEOs of carriers.” From the heads, their issue was one of support, not about compensation.

“This kind of discussion was not the priority.” In this sort of situation, everyone pulls together. “Everyone was focused on how do we work together.”

Of course, with the service back up, I’d imagine those same carriers will be ready to talk about those issues any time now.

7:23 am: Are you concerned you are alienating potential customers on the eve of the launch of the new iPhone?

RIM: “We’ve worked 12 years since the launch of the BlackBerry … We’re going to fully commit to win that trust back.”

7:28 am: All right. Well, there ya go.

Kik — which, like Research In Motion, is based in Waterloo, Canada — got its start on the BlackBerry but had been missing for months after RIM pulled its service. A month after pulling Kik’s app from its store, RIM sued the company.

The 20-person company has put its focus on other smartphone platforms, and now has about five million registered users sending more than 300 million messages a month.

Although Kik has seen plenty of growth from Android and iPhone users, the company wanted to restore service for BlackBerry customers and has been working for the past several months to make that happen. At around noon on Wednesday, Kik relaunched for the BlackBerry — just as RIM’s outage spread to North America. But because it was redesigned to not use any of the BlackBerry infrastructure, Kik remained functional even as many services stayed down.

“It’s so ironic,” said Kik CEO Ted Livingston.

Even though Kik remains embroiled in litigation, Livingston said he isn’t rooting for RIM to fail.

“I’m not enjoying it,” Livingston said of RIM’s troubles. “We’re in Waterloo, and RIM is a pretty important company to Waterloo and Canada.”

Indeed, Livingston himself is a BlackBerry user.

“It’s personally affecting me,” he said. “I can’t get work done.”

Livingston spent much of Wednesday shuttling back to his desk to check email, since he wasn’t getting any on his mobile device. By Wednesday evening, though, he had email up and running on an iPhone.

And that, of course, is the big danger for RIM — that longtime customers might use the outage as an opportunity to switch.

For his part, Livingston said he isn’t sure whether the BlackBerry will remain an important part of his business.

“We are not sure how important it is going to be, long-term, to support BlackBerry, but we had a lot of people request it and so we thought it was important to get back on the platform.”

It’s been heaven. My staff can’t find me, nobody can reach me — only my wife, who I call all the time … I’m enjoying the peace and quiet.

Vermont Sen. Patrick Leahy compared the BlackBerry outage (favorably) to when he goes scuba diving and leaves his BlackBerry behind

“You’ve depended on us for reliable, real-time communications, and right now we’re letting you down,” RIM Chief Information Officer Robin Bienfait said in a posting on RIM’s Web site. “We are taking this very seriously and have people around the world working around the clock to address this situation. We believe we understand why this happened and we are working to restore normal service levels in all markets as quickly as we can.”

The outage comes at a terrible time for Research In Motion, which has been losing ground for months as its aging smartphone lineup has struggled to compete with the iPhone and Android smartphones. Making matters worse, the company is scheduled to host a key developer conference next week.

The BlackBerry maker has said that a failure of core switch led to the initial outage earlier this week, which was exacerbated by the fact that redundant backup systems did not kick in as anticipated. The outage began to affect North America in the last 24 hours, something RIM said is due to its efforts to manage the backlog of email in its systems.

Although the outage is now global in nature, RIM has said different customers are seeing different impacts, with some customers severely affected and others seeing normal service.

“We are doing everything in our power to restore regular service everywhere and to restore your trust in us,” Bienfait said.

The company has not said what, if anything, it will do to compensate or appease unhappy customers. It has pledged that no emails will be lost.

“All of the email will be delivered,” David Yach, RIM’s CTO for software, said on a conference call. “We will not be dropping any email messages.”

On the call, RIM CTO for software David Yach said that the outage appears to be the result of a core switch failure in Europe, followed by the failure of redundancy systems to kick in.

The outage has since spread as the company has had to throttle its service amid a backup of undelivered messages, particurlarly in Europe.

Yach said there was no evidence that hacking or a security breach was involved.

“We’ve seen no evidence to believe this is the case,” he said.

As for what the company will do to appease customers, Yach said his focus is on restoring service, as that is the main thing that customers want. No time frame was given for when service would be fully restored.

Yach did promise that all of the backlogged email messages will eventually find their way to their intended recipients.

“All of the email will be delivered,” he said. “We will not be dropping any email messages.”

Here’s a recap of our live coverage.

12:09 pm: Call just getting started with usual caveats and warnings.

12:10 pm: Begins with spokeswoman: “As you can imagine, we have been pretty busy here and we have limited time for the call.”

12:10 pm: RIM CTO: Global teams focused on the issue.

“It is our top priority to return service to our customers.”

A separate team is trying to discover all of the root causes for the failure as well as the fact that redundant systems didn’t kick in.

12:12 pm: Speaker is David Yach, CTO for software.

The reason the outages have spread globally, Yach said, is because the company now has to throttle service everywhere because of its backlog of undelivered messages.

RIM doesn’t believe the issue is a result of hacking or other security breach.

“We’ve seen no evidence to believe this is the case,” Yach said.

12:13 pm: On to Q and A.

Reporter clarifies: there was an initial core switch issue, followed by a failure of the backup systems, and then the spread to North America and beyond was the result of so many messages piling up.

Yach: Yes.

12:18 pm: Q: Will RIM have to purge backlogged email to restore service?

Yach: No, all of the email will be delivered. We will not be dropping any email messages.

12:19 pm: Q: What about all the angry customers?

Spokeswoman: We have been posting some regular updates on our Twitter handles and our Web sites.

Q: Yes, but that doesn’t seem to be enough. Is there anything else you are doing to communicate with customers?

Yach: Our priority is to get the service up and running because that is what is going to make our customers happy.

Q: Is there any thought of a “make good” for affected customers?

Yach: I don’t have an answer right now. Focus is on restoring service.

12:22 pm: And, with that settled, the call wraps up.

Despite RIM’s repeated reassurances and apologies, ill-will toward the company is building, with public outcry mounting and growing calls for compensation from users all over the world.

And it couldn’t happen at a worse time — just two days before the launch of Apple’s iPhone 4S, which racked up pre-order sales of 1 million in its first day alone, and a week before BlackBerry Devcon, RIM’s big developer event.

A three-day outage like this would be a PR challenge for anyone, but for a company working hard to shore up its declining position in the smartphone market, it’s a daunting one indeed. These outages may well drive some BlackBerry users to consider alternative smartphones, as Jeff Fidacaro of Susquehanna observed in a note to clients this morning.

“Our view is that the longer this issue remains in the headlines and the longer users are without service, there is clearly an increasing risk of building negative goodwill and increasing the churn rate to other platforms such as iPhone and Android,” Fidacaro wrote.

As I write this, RIM shares are trading under $25. In February they were at about $70.50.

Below, a particularly apt skit from comic legend Ronnie Corbett …

The breadth of this particular outage isn’t clear, though reports suggest it again extends to Europe, the Middle East and Africa. RIM hasn’t yet responded to a request for comment, but T-Mobile has confirmed the disruption and blamed it on a European outage on the BlackBerry network. “RIM has apologized for the interruption to services and said it’s working to restore normal operations,” the carrier said in a statement.

So another black eye for RIM, which continues to lose smartphone market share in North America and the rest of the world.

UPDATE: RIM has issued the following statement: “Some users in Europe, the Middle East and Africa, India, Brazil, Chile, and Argentina are experiencing messaging and browsing delays. We are working to restore normal service as quickly as possible. We apologize for any inconvenience this has caused.”

UPDATE, 2:45 pm: Research In Motion says it has found the problem: “The messaging and browsing delays being experienced by BlackBerry users in Europe, the Middle East, Africa, India, Brazil, Chile and Argentina were caused by a core switch failure within RIM’s infrastructure. Although the system is designed to failover to a back-up switch, the failover did not function as previously tested. As a result, a large backlog of data was generated and we are now working to clear that backlog and restore normal service as quickly as possible. We apologize for any inconvenience and we will continue to keep you informed.”

The disruption, which occurred at about 6 am ET, was still affecting users nearly three hours later, with no time given for a resolution.

“We are working to resolve an issue currently impacting some BlackBerry subscribers in Europe, Middle East and Africa,” Research In Motion said in a statement. “We’re investigating, and we apologise to our customers for any inconvenience caused while this is resolved.”

Another tough break for RIM, which endured a service disruption in Canada last month, and at least two fairly major outages in the U.K. and U.S. last year.

A recent spate of technology glitches at U.S. airlines has inconvenienced thousands of travelers, spawned long airport lines, delayed or canceled flights and led to a wave of negative publicity.

United Continental Holdings Inc.’s United Airlines unit suffered a meltdown June 17 that forced it to cancel 36 flights and delay another 100. US Airways Group Inc. was hit by separate glitches on June 10, 18 and 19.

Read the rest of this post on the original site »

“This predominantly affects people using Skype for Windows,” Skype said on its Web site. “We have identified the problem and will issue a fix in the next few hours.”

In the meantime, it offered steps that Linux, Mac and Windows users can take to sign in if they encounter problems.

Skype, which is in the process of being acquired by Microsoft for some $8.5 billion, has had periodic issues, including a major outage last December.

UPDATE: In a blog post, Skype suggested that Windows users update to the latest version. A Mac for Skype update should be available tomorrow.

The Blogger team is now in the process of restoring posts that had been removed during the outage. As part of addressing whatever went wrong, it had made whatever new content that bloggers had actually been able to create in the last 48 hours unavailable.

Google hasn’t explained the outage, except to say it originated from “routine maintenance” on Wednesday night. We’ve asked Google for further detail but have not heard back yet.

Blogger, which is a free service, is primarily a personal blogging platform, though some people do use it professionally. Users published more than half a billion blog posts in 2010, consumed by more than 400 million active readers, with 75 percent of traffic from outside the United States.

In a post on the Blogger help forum, the product team said that it had rolled back a scheduled maintenance release from last night and that its “engineers are working hard to return Blogger to normal and restore your posts and comments.”

Google’s reply for a request for comment was, “The team is working on this.” The company has posted some short updates to the Blogger Twitter account and Status blog, but hasn’t yet explained what’s happening, how widespread it is, and what will happen to users’ content.

Blogger had started rolling out its first major redesign in years in April. The team said in a tweet that the outage was not related to launching the new design.

Update: At 6 am Friday Google said Blogger would be “back to normal soon.”