That was a point that IBM CEO Ginni Rometty made today in remarks at the company’s investors day in San Jose, Calif. IBM is known for making numerous acquisitions over the years, a recent example being StoredIQ.

What does IBM look for in an acquisition? Rometty boiled it down to three questions the company asks before every deal: “Does it extend a capability we have? Does it have scalable intellectual property? Can we extend it to 173 countries around the world?”

In a perhaps not-so-veiled shot at rival Hewlett-Packard, given its combined $16 billion in write-downs on the acquisitions of EDS and Autonomy last year, Rometty said IBM feels strongly that “companies get in trouble when they acquire something that takes them into a new space.”

But just as important to Big Blue’s success in recent years has been its decisions to get out of certain businesses. Examples include the PC business, which it sold to Lenovo in 2004, and, more recently, its retail point-of-sale business, which it sold to Japan’s Toshiba last year.

“Over the last decade, we have divested $15 billion worth of revenue,” Rometty said. “If we had not, we would be a larger company, but we would also be a lesser-margin company, and we would have capabilities that our clients would be less interested in.”

Rometty also said that IBM expects to continue its big bets on technologies like Big Data and analytics. “Data will be the basis of competitive advantage for every company, for every industry in the coming decade.”

To that end, she said that IBM now expects revenue from business analytics to account for as much as $20 billion in annual revenue by fiscal 2015. The prior target was $16 billion. And if Big Blue hits that goal it would amount to a doubling of analytics revenue from 2010.

Image copyright Gregory James Van Raalte

Financial institutions are embracing digital offers out of necessity because revenue from current sources is in decline. The future of the payments industry is shifting quickly to monetizing consumer relationships — particularly through mobile devices — and away from extracting new value from the payments value chain.

What many do not know is that credit card companies have experienced significant pain as a result of the capping of their lucrative interchange fees by the Durbin Amendment, which became effective in 2011. Faced with declines in what had previously been a significant form of revenue, credit card companies know that they need to open new sources of revenue. This need is becoming ever more urgent with companies like Google and Paypal joining the mix and leveraging their unique assets — novelty, retailer relationships, and extensive technology infrastructure.

The opportunity here is massive — total real-world (not online) retail commerce is $4 trillion a year in the U.S., and eMarketer just reported that mobile advertising is expected to increase by 180 percent in the coming year to reach $4 billion. Credit card companies looking to take their fair share of this revenue need to figure out how to leverage their great relationships with consumers to open up new sources of revenue.

Here are my suggestions for strategies that will help credit card companies come out ahead in 2013:

Stop chasing Chase. Instead, start acting like a media company.
To open new sources of revenue through media and technology, banks need to add new DNA that is not focused on traditional banking concerns such as regulatory compliance, security and fraud. Of course, I am not suggesting that they stop complying with regulations, or that they abandon security. But banks must become more nimble at attacking new revenue opportunities — particularly in consumer offers. At Google and other companies in Silicon Valley, teams still pull all-nighters to release alpha versions of products for consumers and partners in a matter of days to test an experience, an approach that is antithetical to the banking world. Yet if banks want to succeed in the media world, they have to figure out how at least part of their organization can play in an API-driven ecosystem that encourages collaboration and rapid product releases. First test for the C-suite at banks: Have you developed and released anything new to consumers in a single quarter?

Play ball with the competitors.
Consider the value of the ecosystem in growing your mobile offers initiative. In order to succeed in delivering a great mobile rewards program, the key is having a rich supply of merchant offers upon which to layer targeting. No one company is going to assemble enough, so think instead about collaborating with all the other players and getting the network effects of a larger audience and a bigger pool of offers. An interesting model for this kind of collaboration is WEVE in the UK, in which three mobile operators — EE, Telefonica UK (O2) and Vodafone UK — have banded together to form a large consortium. The individual companies behind the consortium still compete to acquire subscribers, but they now collaborate with a common platform for monetizing those subscribers through marketing.

Turn that marketing focus inside out.
Now that you have a great offers program, the biggest challenge is getting enough customers into it to move the needle. Each card provider today has marketing teams that focus solely on signing up new customers for their credit cards. Yet somehow, these are not the folks developing new digital offerings. These customer acquisition teams are digital media experts. Imagine focusing that valuable marketing experience on getting consumers to opt into new and sophisticated customer reward programs, and to download their new rewards apps. It is customer acquisition, pure and simple.

Don’t wait for big data.
My last and most controversial suggestion for 2013 is to let go of the obsession with big data! Many banks are building huge databases for micro-targeting customers with niche offers. Microtargeting is cool, but believe me, media is a war of attrition and all of that big data-based targeting will yield a customer segment of just a few people. Brands do not want to target only a select handful of people with their offers; they need to reach a million people in order for their investment in creating and distributing that offer to make economic sense. Silicon Valley has managed to convince enterprises that massive amounts of data will build a program that really performs in terms of offers and sales; but ultimately, microtargeting is not monetizable at scale until there are an equally large set of available offers. The real way to success is to provide a great experience with lots of offers for your entire customer base. Yes, there are a handful of important variables that should be considered, including location, time of day, gender, interests and even retargeting; but other data on top is gravy and too much is at odds with reaching a large enough number of people with relevant offers. The “big picture” is not always based on big data. (Hint: You might derive as much value from customers by asking them what they want instead.)

The Long View
As Bob Dylan said, “you don’t need a weatherman to know which way the wind blows.” Credit card companies can no longer hesitate while other companies race forward to make a profit on this kind of consumer spending. If credit card companies employ the strategies I propose, they will be real contenders against the likes of Paypal, Google, Apple and the mobile operators. The weather forecast is clear for mobile offers: Sunny, with billions of dollars blowing in the wind. It’s up to each financial institution to figure out how to get to market faster and catch some of it.

Alistair Goodman is the CEO of Placecast, where he leads a team of mobile, technology and marketing experts who have created the most scalable, proven, location-based marketing system currently available. Alistair has more than 20 years of experience working in marketing and product development efforts for media and technology companies.

A company called DataGravity, based in Nashua, N.H., is aiming to reduce the complexity of getting useful information from stored data. It announced today that it has taken $30 million in a Series B round of venture capital financing. The round was led by Andreessen Horowitz, and Peter Levine, an AH partner, will join DataGravity’s board. Existing partners Charles River Ventures and General Catalyst Partners also particpated in the round.

The company is being a little cagey about what exactly it is doing, but it was started by Paula Long and John Joseph, two execs from EqualLogic, the storage company acquired by Dell in 2008. Long was a founder, and Joseph was vice president of marketing and product management.

Where EqualLogic was all about making the complex business of managing data storage in the enterprise easy enough for a general IT person to handle, DataGravity is aiming to do something similar in the area of extracting useful intelligence from data repositories.

“You’ll hear a lot about companies going after data repositories, but it’s usually done with a big professional services component to it,” Long said. “Usually a third-party company will arrive at your site with three or four people who will mine your data for you for $150 an hour. The companies we want to sell to won’t be able to afford that. Nor will they have the people with the necessary skills internally.”

“These are companies looking for business intelligence, but who are looking for it in a more consumable fashion,” Joseph said.

In a statement, Levine called DataGravity one of the “pioneering companies doing the hard work of solving intractable enterprise challenges.”

DataGravity expects to be more specific about its plans next year. In the meantime, it will use the new round of funding to help build the product and set a go-to-market plan for 2014. It is also actively recruiting people in the fields of data visualization and user experience.

Last year was a year when the phrase “Big Data” was all over the place. Dig through the troves of data your business generates, the thinking goes, and some useful business intelligence falls out. That, at least, is the idea, and there are numerous companies — some startups, some big and established players — trying to build business plans on different aspects of that idea.

But in the course of being reduced to a simple buzzy phrase, Big Data as a concept implies some expectations, some realistic, some undoubtedly not. The research house Gartner has a phrase for this tendency as well: The Hype Cycle.

The Hype Cycle goes like this: A new technology that promises to fundamentally “change everything” gets talked up incessantly in the press and at industry events and often also in research reports. At some point the chatter peaks, and expectations reach a fever pitch. Soon, maybe a year or two after it all started to build and some money has been spent and everything that was supposed to have changed for the better actually hasn’t, the narrative focus turns negative. What seemed so brilliant and earthshaking 18 months ago, seems in restrospect to have been an ill-advised waste of time, money and attention.

This is what Gartner calls the “Trough of Disillusionment” phase of the Hype Cycle.

Gartner analyst Svetlana Sicular argues in a blog post that Big Data may have reached that point. She has been “hearing from people in the center of the Hadoop movement,” the open-source technology central to companies like Cloudera, Hortonworks and MapR. She also presents a video of a Hadoop gathering called Elephant Riders where reps from these three companies are debating its current state. (It’s about 90 minutes and if you’re so inclined, you can see it here.)

By Gartner’s standards, the trough of disillusionment may indeed have arrived, though you certainly wouldn’t be able to tell from the level of investment interest in companies like Cloudera, which late last year raised a massive $65 million round of funding.

One source of that disillusionment, she writes, is that companies are struggling with a basic problem: What questions do you attempt to answer with your data in the first place? “Several days ago, a financial industry client told me that framing a right question to express a game-changing idea is extremely challenging,” Sicular wrote. “First, selecting a question from multiple candidates; second, breaking it down to many sub-questions; and, third, answering even one of them reliably. It is hard.”

Hadoop doesn’t exactly make that process any easier. Once you’ve decided to use it, getting anything useful out of it requires some pretty specialized knowledge and training, and finding the right people to do that isn’t easy. But the industry is beginning to respond to that need: Startups like Mortar Data have sought to make Hadoop more readily accessible to mainstream programmers, while another called Trifacta makes the resulting data easier to manipulate.

And versions of Hadoop itself are getting incrementally easier to work with. Hortonworks, for example, recently released HDP 1.2, a new version of the open-source platform, but also Sandbox, a set of training tools that lets developers play around with Hadoop and get a feel for its use.

I talked with Hortonworks CEO Rob Bearden recently, and he said that, in 2011, companies had no idea what Hadoop could be used for, then spent 2012 experimenting with it, and now want to get some real-world value out of it in 2013. “This year, all the technology is coming together in a way that is consumable,” he said. “In the last quarter of last year we saw a lot of interesting production environments. Now the objectives are becoming clear for getting useful in 2013.”

The next milestone in the Hype Cycle, Sicular writes, is negative press. Eventually it’s followed by a period called the “Slope of Enlightenment,” and finally the “Plateau of Productivity.” It’s nice to know there could be a positive conclusion to all this somewhere down the road.

The scientists encoded in DNA an audio clip of Martin Luther King Jr.’s “I Have a Dream” speech, a photograph, a copy of Crick and Watson’s famous “double helix” scientific paper from 1953 and Shakespeare’s 154 sonnets. They were then able to retrieve them with 99.99% accuracy. The experiment was reported Wednesday in the journal Nature.

Read the rest of this post on the original site »

Image copyright Petr Vaclavek

Armed with mountains of data, most experts put in charge of predicting outcomes still can’t come up with accurate answers. Why? Because most people have a human bias to manipulate data to deliver the outcomes they really want to see.

Philip Tetlock, a psychologist at the University of California at Berkeley, published a book in 2005 titled “Expert Political Judgment — How Good Is It?” For the book, a team of researchers interviewed 284 political experts and extracted 80,000 predictions from them over 15 years, rating the probabilities that each event would occur (they gave each prediction three possible outcomes). The result was surprising: the experts’ predictions were worse than those made simply by assigning equal probabilities to all three outcomes. Dart-throwing monkeys could have performed just as well. It’s easy to make fun of talking heads and political pundits, but what about experts in other fields?

Another psychology professor and a past President of the American Psychological Association, Paul Meehl, has conducted similar research in other fields. In his work, he studied whether a simple algorithm could predict freshmen grades at the end of a school year better than expert college counselors could. The algorithms were only given SAT scores and grades. The expert counselors, on the other hand, were given the same data as well as a 45-minute interview and a personal essay. Again the simple algorithm won. In yet another example of experts versus algorithms, Paul’s work pitted algorithms predicting the future price of a bottle of wine against connoisseurs. In this case, the wine predicting algorithm had access only to three weather variables while the wine connoisseur, on the other hand, had all of the data and the ability to taste the wine. By now it should be obvious the algorithms beat the connoisseurs when it came to predicting the wine’s price. Paul’s analysis didn’t stop there. It extended to medical experts, flight trainers, parole officers, bankers and many others. In all the cases, the algorithms won. So where did all these experts go wrong? There were several reasons — but we will focus on just one here.

The biggest bias experts have is they tend to value perceived causes or stories over the underlying data. Storytelling is a powerful way to inject order into chaos, but reality is often far more complex than a simplified story. Whenever data supports their stories, they promote the data, but when data contradicts their stories, they question or ignore the data. Experts become really good at “finding the data” to promote their case.

What does all this have to do with marketers? A marketer’s job is to tell a story. Any expert will have a tendency to manipulate data to get it to conform to his or her story. Marketers are susceptible to the same sins, except they’re even more prone to storytelling. There is also a terrible feedback loop at work here. There may be good marketers who don’t want to torture data to fit into a clean story, but these marketers couldn’t make the bold claims that end up as headlines. So in short, marketers are some of the worst “experts” at predicting specific outcomes because their job biases them to storytelling and rewards them for making predictions that end up in the news.

To better understand why marketers tend to ignore data, let’s look at one possible scenario.

Imagine a Father’s Day ad that’s unfortunately misplaced next to a news article with the headline “One in Four Women Victims of Domestic Violence.” As a marketer, you are presented with the following two formulations about the poorly placed ad and you want to find out which publisher you should blame:

Problem Description A
85 percent of the impressions in the Father’s Day ad campaign are served in publisher A, and 15 percent are served in publisher B. Ad verification software says the ad was found in a bad context on publisher B, and the verification software is right 80 percent of the time.

When asked, most marketers choose publisher B as the most likely cause of the problem. But let’s try an alternate formulation of the problem:

Problem Description B
Publisher A and publisher B served the same number of ad impressions.
We learn that 85 percent of ads served in publisher A are served into inappropriate content. Ad verification software says the ad was found in a bad context on publisher B. The verification software is right 80 percent of the time.

Now which publisher do you think caused the problem? Most marketers choose publisher A as the culprit.

It turns out both formulations of the problem result in the same probabilistic answer — that is, 61 percent probability that publisher A caused the problem, according to Bayes rule in odds form. Most marketers get the first version wrong and the second version right, even though it’s the same math problem! Why do most marketers get the second version right? In the first story, how you use the fact that 85 percent of impressions are served in publisher A is not obvious. In the second story, you have a cause. Publisher A is, for all practical purposes, almost always inappropriate. Marketers prefer stories that point to an understandable cause over statistics. Marketers will hone in on a story and ignore the rest of the data, no matter how big the data set.

Think of the many problems marketers have that involve some form of prediction. Who is the target audience for this product or service? Why do they need, like or want the product or service? Where can we find this audience? What creative will succeed with this audience? We even have a whole industry built around the RFP (request for proposal), which itself is a form of a story. The RFP mechanism guides billions of dollars of ad spending, and ad allocation decisions are forms of predictions.

So can marketers learn to overcome their biases and use data to make more accurate predictions? It may well be possible.

In his book, “The Signal and the Noise, ” Nate Silver tells a great story about how the PECOTA system he built to predict pitcher performance in baseball was beat by expert scouts. In 2006, Nate’s program PECOTA produced a list of 100 prospects from the minor leagues. He then pitted that list against the Baseball America Top 100 prospects list, which was constructed by scouts. After watching six years of actual performance, and worrying their jobs would disappear due to PECOTA, the scouts performed better than the PECOTA list by a good margin. It turns out the scouts had learned to tone down their own biases and instead “listen” to the data. They learned they could combine their expertise with the reams of public data available.

Marketers can learn to separate the signal from the noise. Marketers are dedicated storytellers who believe their own stories, downplay luck and impose cause on randomness. In the world of Big Data, marketers have to learn to to let the data speak for itself. Otherwise, all the big data in the world won’t help marketers understand their customers better.

Omar is the co-founder and CEO of BlueKai, the data activation system that supplies both Fortune 100 companies and leading publishers with solutions for managing and activating first- and third-party data for creating highly effective customer and marketing campaigns. Omar’s previous roles include Chief Advertising Officer for mobile search and advertising solution Medio and Chief Marketing Officer for early behavioral data leader Revenue Science.

To understand how graphs address data complexity, we need first to understand the nature of the complexity itself. In practical terms, data gets more complex as it gets bigger, more semi-structured, and more densely connected.

We all know about big data. The volume of net new data being created each year is growing exponentially — a trend that is set to continue for the foreseeable future. But increased volume isn’t the only force we have to contend with today: On top of this staggering growth in the volume of data, we are also seeing an increase in both the amount of semi-structure and the degree of connectedness present in that data.

Semi-Structure

Semi-structured data is messy data: data that doesn’t fit into a uniform, one-size-fits-all, rigid relational schema. It is characterized by the presence of sparse tables and lots of null checking logic — all of it necessary to produce a solution that is fast enough and flexible enough to deal with the vagaries of real world data.

Increased semi-structure, then, is another force with which we have to contend, besides increased data volume. As data volumes grow, we trade insight for uniformity; the more data we gather about a group of entities, the more that data is likely to be semi-structured.

Connectedness

But insight and end-user value do not simply result from ramping up volume and variation in our data. Many of the more important questions we want to ask of our data require us to understand how things are connected. Insight depends on us understanding the relationships between entities — and often, the quality of those relationships.

Here are some examples, taken from different domains, of the kinds of important questions we ask of our data:

• Which friends and colleagues do we have in common?
• What’s the quickest route between two stations on the metro?
• What do you recommend I buy based on my previous purchases?
• Which products, services and subscriptions do I have permission to access and modify? Conversely, given this particular subscription, who can modify or cancel it?
• What’s the most efficient means of delivering a parcel from A to B?
• Who has been fraudulently claiming benefits?
• Who owns all the debt? Who is most at risk of poisoning the financial markets?

To answer each of these questions, we need to understand how the entities in our domain are connected. In other words, these are graph problems.

Why are these graph problems? Because graphs are the best abstraction we have for modeling and querying connectedness. Moreover, the malleability of the graph structure makes it ideal for creating high-fidelity representations of a semi-structured domain. Traditionally relegated to the more obscure applications of computer science, graph data models are today proving to be a powerful way of modeling and interrogating a wide range of common use cases. Put simply, graphs are everywhere.

Graph Databases

Today, if you’ve got a graph data problem, you can tackle it using a graph database — an online transactional system that allows you to store, manage and query your data in the form of a graph. A graph database enables you to represent any kind of data in a highly accessible, elegant way using nodes and relationships, both of which may host properties:

• Nodes are containers for properties, which are key-value pairs that capture an entity’s attributes. In a graph model of a domain, nodes tend to be used to represent the things in the domain. The connections between these things are expressed using relationships.
• A relationship has a name and a direction, which together lend semantic clarity and context to the nodes connected by the relationship. Like nodes, relationships can also contain properties: Attaching one or more properties to a relationship allows us to weight that relationship, or describe its quality, or otherwise qualify its applicability for a particular query.

The key thing about such a model is that it makes relations first-class citizens of the data, rather than treating them as metadata. As real data points, they can be queried and understood in their variety, weight and quality: Important capabilities in a world of increasing connectedness.

Graph Databases in Practice

Today, the most innovative organizations are leveraging graph databases as a way to solve the challenges around their connected data. These include major names such as Google, Facebook, Twitter, Adobe and American Express. Graph databases are also being used by organizations in a range of fields including finance, education, web, ISV and telecom and data communications.

The following examples offer use case scenarios of graph databases in practice.

• Adobe Systems currently leverages a graph database to provide social capabilities to its Creative Cloud — a new array of services to media enthusiasts and professionals. A graph offers clear advantages in capturing Adobe’s rich data model fully, while still allowing for high performance queries that range from simple reads to advanced analytics. It also enables Adobe to store large amounts of connected data across three continents, all while maintaining high query performance.
• Europe’s No. 1 professional network, Viadeo, has integrated a graph database to store all of its users and relationships. Viadeo currently has 40 million professionals in its network and requires a solution that is easy to use and capable of handling major expansion. Upon integrating a graph model, Viadeo has accelerated its system performance by more than 200 percent.
• Telenor Group is one of the top ten wireless Telco companies in the world, and uses a graph database to manage its customer organizational structures. The ability to model and query complex data such as customer and account structures with high performance has proven to be critical to Telenor’s ongoing success.

An access control graph. Telenor uses a similar data model to manage products and subscriptions.

• Deutsche Telekom leverages a graph database for its highly scalable social soccer fan website attracting tens of thousands of visitors during each soccer match, where it provides painless data modeling, seamless data model extendibility, and high performance and reliability.
• Squidoo is the popular social publishing platform where users share their passions. They recently created a product called Postcards, which are single-page, beautifully designed recommendations of books, movies, music albums, quotes and other products and media types. A graph database ensures that users have an awesome experience as it provides a primary data store for the Postcards taxonomy and the recommendation engine for what people should be doing next.

Such examples prove the pervasiveness of connections within data and the power of a graph model to optimally map relationships. A graph database allows you to further query and analyze such connections to provide greater insight and end-user value. In short, graphs are poised to deliver true competitive advantage by offering deeper perspective into data as well as a new framework to power today’s revolutionary applications.

A New Way of Thinking

Graphs are a new way of thinking for explicitly modeling the factors that make today’s big data so complex: Semi-structure and connectedness. As more and more organizations recognize the value of modeling data with a graph, they are turning to the use of graph databases to extend this powerful modeling capability to the storage and querying of complex, densely connected structures. The result is the opening up of new opportunities for generating critical insight and end-user value, which can make all the difference in keeping up with today’s competitive business environment.

Emil is the founder of the Neo4j open source graph database project, which is the most widely deployed graph database in the world. As a life-long compulsive programmer who started his first free software project in 1994, Emil has with horror witnessed his recent degradation into a VC-backed powerpoint engineer. As the CEO of Neo4j’s commercial sponsor Neo Technology, Emil is now mainly focused on spreading the word about the powers of graphs and preaching the demise of tabular solutions everywhere. Emil presents regularly at conferences such as JAOO, JavaOne, QCon and OSCON.

A classic problem in Big Data circles is the reflexive urge to move data before you manage it and do anything useful with it. StoredIQ’s approach calls for managing data in the place it is originally stored. It works with a lot of different kinds of data sources — desktop machines, file shares, even old tapes. The point is to analyze them all, find the important bits, and delete the bits that aren’t important.

IBM will add StoredIQ to its Information LifeCycle Governance Suite, which aims to help companies spend less on the data they accumulate over the years. Data has a bad way of multiplying. If you’ve ever seen an email chain with attachments get sent to lots of people — then re-sent, forwarded and so on — you can pretty easily understand how unnecessary copies of the same data make storing it all so difficult. And if it’s happening on your email servers, it’s happening with other kinds of data, as well. Copies get made, then copies of copies. In time, it becomes unrealistic to sort through it all and just clean it up.

There is value in that older data as it ages. There are business patterns that repeat themselves that aren’t apparent years later: Pricing data, seasonal patterns and other information that’s worth keeping on for years and years. That’s where IBM’s analytics chops come into play. The key is figuring out which bits are important and which aren’t.

It’s not a trivial problem. Some companies are required by law — there have been a lot of new regulations around data retention in recent years — to store everything in case of a lawsuit or an enforcement action by a regulatory agency. As data multiplies, so does the cost to store and manage it, creating a headache for both the CIO and general counsel.

The deal builds on prior IBM acquisitions, like PSS Systems in 2010 and Vivisimo earlier this year.

That includes the students from Professor Marti Hearst’s class at the University of California at Berkeley, all of whom just finished off a semester entrenched in data sets sliced straight from the Twitter fire hose, after taking a course called I290: “Analyzing Big Data with Twitter.”

It was essentially a crash course in Twitter’s big-data practices. The engineering and math students enrolled spent half the semester in a lecture hall listening to a host of guest product managers and engineers from Twitter, who explained the company’s approaches to analyzing and dealing with the massive amount of data that flows through Twitter’s pipes every single moment.

For data nerds (and wonks like me who follow the company), the course was a treat, as Twitter top brass like VP of Growth Othman Laraki stopped in to deliver lectures on some of the company’s best practices and philosophies when, for example, it comes to growth strategy. Other visitors included data systems engineers, search and relevance team members, and a former Twitter anti-spam/security expert.

When the course neared its end last week, some 40 students from Professor Hearst’s course visited Twitter’s headquarters in San Francisco and presented their small group projects created using Twitter data.

Bummed you missed out on the course? Fret not. Professor Hearst posted video lectures from the semester on her blog at the UC Berkeley School of Information. It’s all super-geeky stuff, but fascinating to watch if you’re at all interested in how Twitter makes sense of all the data that comes its way on a regular basis. (I recommend Laraki’s talk in particular — Twitter’s approach to growth is key to watch over the next few years, as the company continues to scale.)

Or, you know, you could just skip it and party like a true undergrad for all of winter break.

Since DataRaker is relatively small and privately held, financial terms aren’t being disclosed. By my count, and it may be wrong, this is Oracle’s ninth acquisition of 2012.

DataRaker supplies cloud-based analytics software to electrical and gas utilities that are using smart meters. It combines data gathered from those meters with data gathered from older legacy systems and provides analysis that helps control costs, saving both customers and the utility money.

Here’s Oracle’s announcement.

Oracle Buys DataRaker

Adds Cloud-based Analytics Platform to Oracle Utilities Solutions to Transform Big Data into Actionable Intelligence

Redwood Shores, CA – December 13, 2012

News Facts
Oracle today announced that it has entered into an agreement to acquire DataRaker, a provider of a cloud-based analytics platform that enables electric, gas and water utilities to leverage vast amounts of data to optimize operational efficiency and improve the customer experience.

Leading-edge utilities are investing in infrastructure to collect massive amounts of data from millions of distributed smart meters and sensors, and they require modern technologies to analyze and understand the insights provided by this data.

DataRaker’s proven solutions enable customers to gain immediate benefits from smart devices, by transforming meter, customer and network data into insights that can have a dramatic impact on organizational performance.

The combination of Oracle and DataRaker’s cloud-based solutions is expected to provide utilities with the most complete solution to harness the benefits of utility Big Data to improve operational performance and enhance customer experience.

More information on this announcement can be found at http://www.oracle.com/DataRaker
Supporting Quotes

“Big Data created by smart meters and sensors has presented utilities with an enormous opportunity to improve operations and deliver better customer service by acting on the unique insights that can only be found by understanding the massive amounts of data coming from their customers and networks,” said Rodger Smith, Senior Vice President and General Manager, Oracle Utilities. “With DataRaker, Oracle can provide customers a complete and integrated set of products to further unlock efficiencies and create data insights that maximize business value.”

“Oracle’s proposed acquisition of DataRaker represents a strong endorsement of DataRaker’s proven, scalable and high-performance platform. We are excited to be a part of Oracle, and look forward to combining our resources to drive new innovations that will continue to deliver value to customers,” said Rick Brakken, CEO and Co-Founder, DataRaker.

Image by timquo

The BIG deal is the disruptive value caused by the advent of high performance digital data capture. This technology has made data collection almost free. Internet-based marketing systems auto-magically capture masses of information about prospective customer preferences. Flash-enabled digital movie cameras can be emptied every night and re-used, a far cry from the film model in which every frame was burned on expensive media — which then had to be manually processed and edited. Digital capture systems also enable capture of exponentially more data per event. Social media sites and businesses increasingly create, store and analyze HD video instead of text, at ten to one hundred times more granularity of data per customer or product. The availability of compute power to manipulate this data to create business advantage enables companies like WalMart to maintain and frequently utilize multi-petabyte databases to analyze customer patterns and speed decision making. For the first time, solutions are also becoming available which allow for productive analysis of video as well. Finally, the resulting content is being stored forever: after all, that terabyte per day per oilfield of 3D seismic data being collected today might turn into the next decade’s oil find; or today’s genomic profile might be tomorrow’s cure for cancer.

As a result of this “free” data capture, increasing information granularity, more frequent usage and extended data value, businesses, research institutions and governments are growing enormous stores of large unstructured data (increasingly video) that needs to be stored and managed. This result presents a number of challenging data storage problems: extreme scalability; affordability (in general); managing the balance between cost and easy online access; maximizing application and user access; and assuring data durability.

Like the cavalry, a new set of “wide area storage” solutions based on second generation object storage are arriving just in time to help enterprises manage these issues. Object storage is a unique storage architecture utilizing a kind of valet parking ticket system to store and retrieve data. The creator of a piece of data (an object) hands the object to the storage in exchange for an object identifier (the digital equivalent of a valet parking ticket). When the data is later needed, the user hands the system the object identifier, and data is returned. The power of this model is that it is highly scalable — many objects can be stored and retrieved in parallel — and retrieval can be independent of the original application or physical location. Any application or user who gains authorized access to the electronic key can use the data. Historically, this technology has been used to store data in large, containerized archives, but these were limited by the size and performance of the object storage “box.”

What makes the latest generation architecture different is its ability to copy and disperse objects very efficiently across large numbers of independent processing and storage elements, which can even be distributed widely — geographically providing disaster recovery protection without the need for traditional replication. This wide area storage essentially frees object storage from the constraints of a box or even a site. Wide area storage is similar to storing data in the internet or cloud — data is freed from the traditional boundaries of expensive storage hardware and processes. This results in capabilities which are perfectly suited to the challenges of Big Data: almost infinite scale; low cost per petabyte; the ability to afford 100 percent online data storage; global multi-application access; and a content store which can live forever without ever needing to endure a disruptive data migration.

How does it achieve this? Well first, as previously described, the underlying object storage is natively extremely scalable; unlike systems with centralized indices, if you want to add more data to a wide area storage system, you simply add more objects, which are then dispersed across more scaled out components to add access, performance or storage. And the hardware architecture is scale-out, which means if you need more storage, more performance or more communications bandwidth, you just plug in more pre-packaged storage, processors or network capacity, and the system takes care of the rest. Growth is simple.

Second, much like the systems we’ve all read about at Google, wide area storage systems are built under the assumption that individual hardware components will fail. Because objects are copied and dispersed across many storage and geographic resources, a multitude of components can fail while data is continuously available. This “failure-assumed” model allows for wide area storage to operate on lower cost, off-the-shelf disk and processor technologies, which translates to much lower capital cost than traditional disk storage. The ability to defer replacement of failed components also results in lower support and operating costs. And because multiple users and sites can share the system in common, the overhead of storage and disaster protection is shared across all users. No archive silos.

Third, the native cloud-like access model of these new offerings embeds the capability for “wide” geographic and application access, supporting a wide range of uses – from streaming data (like video, sensor information, and genomic sequences) to parallel processing (like Hadoop for analytics). Some solution vendors are also providing easier access for existing applications, including policy-based tiering from traditional disk or the ability to appear like a NAS archive.

Finally, and perhaps most intriguing, these new solutions offer the capability for content, once stored, to never need to be migrated again. This may be the most critical element of all. For any user or technology manager who has endured the pain of unloading a broken, filled or obsolescent NAS or traditional block storage array in order to migrate to the next big thing, with next generation object storage, you never need do this again. This is going to be critical when your Big Data store is holding hundreds of petabytes of high value data. Forever.

Janae Stow Lee is senior vice president for Quantum’s file system and archive products.

Hadoop has become a big thing, and in fact there are numerous flavors of it and numerous companies offering their own version of it, while others are building things to work with it.

The biggest and best known company in the Hadoop world is Cloudera. Started in 2008 by a trio of engineers from Facebook, Google and Yahoo (Jeff Hammerbacher, Christophe Bisciglia and Amr Awadallah) plus CEO Mike Olson, a former Oracle exec, it has in four years gone from the start-up that few really understood to the company you have to talk to if you want to stand a chance wrestling your data challenges to the ground.

Today, the company will take a significant step forward. AllThingsD has learned that Cloudera has landed a $65 million Series E round of venture capital funding. The funding round is being led by Accel Partners. Accel is not a new investor in Cloudera — it led its $5 million Series A round in 2009 — and it is leading a round that includes all repeat investors, including Greylock Partners, Ignition Partners, In-Q-Tel and Meritech Capital Partners.

The round brings Cloudera’s total capital raised to about $140 million. And according to sources familiar with the terms, this deal values Cloudera at about $700 million.

The round was to be announced after a closed meeting of Cloudera employees at its Palo Alto headquarters led by CEO Mike Olson and other senior executives. The meeting was scheduled to take place at noon PT.

The company also said it plans to expand its European operations with the opening of new offices in the United Kingdom in the first quarter of 2013.

Coming as it does on the heels of Cloudera’s hiring of a new CFO — Jim Frankola, the former CFO of Yodlee and Ariba who started his career at IBM, joined on Oct. 3 — the questions about a possible public offering of Cloudera shares will begin to increase.

I asked Olson and Frankola about it in an interview Monday. They both said they’re interested in the long-term future for Cloudera, and if that includes an IPO, then great. “From the very beginning we’ve seen the opportunity here that is big enough to create a long-term business,” Olson said. “You don’t often get a chance to step up to the plate in a ballpark that is as exciting as this one,” Olson told me.

Frankola hearkened back to his days in corporate finance at Big Blue. “At IBM we got to make really big, multibillion dollar bets, and then you go to smaller companies and you sometimes have to play it safe. But the space that we’re playing in is one of the biggest bets there is in technology.”

I also talked with Accel Partner Ping Li, who has long been a Cloudera fan and created Accel’s Big Data Fund in part to fund companies doing interesting things with Hadoop. “These guys pretty much created the big data wave when they decided to take Hadoop out of the bowels of the data center and into the enterprise,” he said. “They really created the idea that there’s all this data and Hadoop is a really critical layer in the stack.”

How’s the business doing? No one would say exactly, though Frankola said Cloudera has been seeing its revenue and number of customers roughly doubling every year. Li put it this way: “You don’t see this type of growth in software and infrastructure companies that often.”

When you consider that four years ago people sort of scratched their heads and wondered what they would actually use Hadoop for, the list of companies that are Cloudera customers is impressive. It includes AOL, CBS, eBay, Expedia, J.P. Morgan Chase, Monsanto, Nokia, Research In Motion and the Walt Disney Company.

Cloudera is sometimes compared — not inaccurately, but perhaps imprecisely — to Red Hat, which built a business on helping companies deploy and manage the free Linux operating system. Like Linux, Hadoop is technically free, and any company can go out and download it for free. The trick comes in knowing what to do with it, and managing it, and getting value out of it. And that’s where Cloudera’s money-making value comes into play. It sells its own software that runs on top of Hadoop, plus it also distributes its own flavor of Hadoop that it supports and helps its customers manage on a subscription basis.

“Everything that can be invented has been invented.”
—Charles H. Duell, Commissioner, U.S. patent office, 1899

Last month, we gathered 75 of the top CIOs from around the country to discuss the new generation of enterprise software and the redefined role of the CIO. These CIOs are dealing with an unprecedented level of experimentation and innovative new approaches focused on unsolved problems in enterprise software. The end result will be a complete remaking of the entire enterprise software stack at the intersection of cloud, mobile and SaaS.

All of the CIOs are also facing a changed environment, one where every department within an organization makes its own software buying decisions, outside the purview of the CIO. This “departmentalization of applications” — from Box for collaboration to GitHub for software development to Tidemark for Enterprise Performance Management — means the CIO not only needs to figure out how to enable the department and employee to leverage these software products, but also meet the security and compliance requirements of the larger corporate environment — which, by the way, Bromium and Okta allow you to do. These CIOs know that they can adapt or organizations will adapt without them.

Their jobs weren’t always so difficult. For those of you old enough to remember, there was a time when enterprise computing was almost exclusively dominated by Microsoft, Oracle and Cisco. It was a time when on-premise, Windows-based applications were the de facto standard and there was no alternative. The enterprise was so entrenched that challenging the status quo was viewed as suicidal and very stupid. So hardened was the thinking that most innovation in the enterprise was relegated to mere feature extensions of existing solutions.

Fast-forward to today and the world of enterprise computing has done a 180. Traditional IT is being blown to bits as cloud infrastructure, Software-as-a-Service and mobile computing become the new standards. We are experiencing innovation and usage as never seen before. It is truly a renaissance of massive scale. Hundreds of billions of dollars are up for grabs as buyers shift to new architectures and away from old, as new users and new markets embrace the availability and ease by which they can consume technology.

On the Road to a Revolution

VMware and Salesforce catalyzed this movement from unlikely origins. Both were little known and under-funded, but against all conventional wisdom each visualized a new world order — a world in which the data center was virtual and where applications would run off-premise, eliminating op-ex and painful software upgrades. The world watched but there were few believers. “Suicidal,” people said. “Why would I ever permit my precious customer data to reside outside my firewall?”

But momentum grew. VMware figured out how to effectively break apart the functionality of software from the hardware it resides on, driving a new set of economics into data centers. Salesforce began expanding beyond CRM, demonstrating the wider viability of subscription-based payments and the customer benefits of constant iteration. Customers began to believe that this new vision might actually come true. From a single virtual server and a single customer relationship app, both companies paved the way for a new world order.

Every part of the business software stack is now being remade — from infrastructure to applications to mobile to analytics — with every incumbent in danger of having its core business eroded. And, sure, incumbents will try to buy innovative products and will try to develop their own competing technologies, but the reality is that this new paradigm disrupts the entirety of these businesses. Overcoming a foundational shift cannot be met by a simple product buy or even a strategy change — the new breed of enterprise software start-ups has different revenue recognition policies, different sales models and different go-to-market models, and engineering processes than incumbents. We are talking about transformations occurring here simultaneously in technology and business models! It’s an entirely new approach to IT.

The Departmentalization of Applications

Buyers are clamoring for this new approach. None of our portfolio companies use Oracle. Some use Microsoft, but the majority opt for Google or an open source package. In our own Executive Briefing Center, where we connect and facilitate exchange amongst global brands and the rising stars in tech, we’re finding that even enterprise CIOs are looking beyond mature players to new and emerging technology companies, especially in areas like cloud computing, mobile, big data and SaaS. These are the early indicators of a more permanent shift in IT consumption habits. This shift is resulting in software applications that are targeted for specific business functions. Apptio, for example, has built a world-class application that specifically targets the CIO as a customer. Mixpanel has an application that is in the big data space, but specifically targets analytics for mobile applications. This shift is what I am calling the “departmentalization of applications”.

And entrepreneurs know that incumbents are vulnerable. We see a tremendous number of entrepreneurs bringing a new approach to this crusty old enterprise software market. We see entrepreneurs like Ben Werther of Platfora, who is passionate about up-ending the Business Intelligence market, and Ash Ashutosh of Actifio, who is creating the next generation storage software.

These are entrepreneurs who choose to do the hard work of building software for companies to use, and the software they are creating is elegant, fast, does what it’s supposed to and priced fairly. This is an unbeatable value proposition. For everyone except perhaps the incumbents, this is a great time to be involved with enterprise software.

Peter Levine is a partner at Andreessen Horowitz and blogs at http://peter.a16z.com/. He has been a lecturer at both MIT and Stanford business schools and was the former CEO of XenSource, which was acquired by Citrix in 2007. Prior to XenSource, Peter was EVP of Strategic and Platform Operations at Veritas Software, where he helped grow the organization from no revenue to more than $1.5 billion, and from 20 employees to over 6,000.

Weighing in at 7.5 pounds, it is an ambitious, jaw-dropping effort helmed by former Time, Life and National Geographic photographer Rick Smolan — he of the “Day in the Life” series of photography books, as well as “America at Home,” “America 24/7” and “24 Hours in Cyberspace.” Smolan’s new book attempts to demystify — largely successfully — the nebulous concept: What is big data?

It was exactly the question that Smolan was asking when he first hit upon the idea for the book while attending the D9 conference in 2011. Hearing the phrase “big data” uttered in so many conversations, he had no idea what it meant. Asking at first yielded unclear answers, yet he persisted, eventually landing on the idea.

Today, the book is landing on the desks of world leaders, dignitaries and other notable people around the world: Among those on the list: President Obama, the Dalai Lama, Pope Benedict XVI and Chinese Premier Wen Jiabao, and also Twitter CEO Jack Dorsey; Daniel Tunkelang, chief data scientist at LinkedIn; and actor Robin Williams. Among the images they’ll see upon opening it is the blended image of 1,400 different shots of New York’s Times Square taken across 15 hours. Big data is about people: What they do, where they go, who they know and so on. The stories about how data, once harnessed, solves problems and in some ways creates new ones, is its overarching theme.

There’s also a smartphone app for iPhone and Android that is launching today. It’s an interactive viewer app from Aurasma that aims to bring the book’s content to life, accessing videos and animations by pointing the camera at images on certain pages flagged within the book. On top of that, there’s a $2.99 iPad app that enables readers to take a deeper dive with some of the stories, using videos, charts and animated infographics.

I talked with Smolan about the book yesterday by phone, after spending more than a few hours perusing an advance copy over the weekend. Here’s a little of what we talked about:

AllThingsD: So where did you get the idea for a book on big data? It’s a phrase that doesn’t necessarily jump out at me as part of the title of a bestseller.

Smolan: I was at D: All Things Digital in 2011, and I kept hearing the phrase big data, and I kept asking people what it meant, because I felt stupid and because it sounded like one of those marketing phrases. The first person I talked to said, “It’s so much information it won’t sit on your personal computer.” Well, that wasn’t very interesting. The next one said, “It’s taking information from one place and overlapping it with information from another, and finding these patterns.” And that wasn’t interesting, either. The third person said, “It’s like watching the planet grow a nervous system.” And that sounded interesting. Basically, we’re seeding the world with low-cost sensors, and we’ve all become sensors with our cellphones. And instead of doing random samplings, we can almost survey every single person on the planet in real time — where they are, what they’re doing, how fast they’re going, what they’re spending money on. The ability to gather that information, process it, visualize it and then respond to it while it’s still happening is something we’ve never had the ability to do before.

Some of the material in the book I’m familiar with. The first image I saw when I opened it was one I recognized from MIT’s Sensable City Lab, and I also recognize big data anecdotes from IBM, like the one where they harnessed medical data to detect infections in premature infants. In this way, it seems it’s a little different from your previous books.

It’s sort of a combination of original photography and curation. I think that putting all the information in one place and weaving it together, with these wonderful essays that I think are just as strong as the pictures. I’ve been getting notes from people like Marissa Mayer and Jack Dorsey saying that this is the first time they’ve had something that helps them explain how important this is. Amazon called last week to say they sold out of copies of the book on the first day and people were ordering 50 or 60 copies at a time, which has never happened ever to any book I’ve done in 25 years. They were dumbfounded. The hard thing about the book world is that you never know whether 10 people or a million people will find it interesting. A lot of people have never heard about big data and the ones who have, have a lot of trouble explaining it to other people. So I’m hoping that this will become the thing the people who know give to their parents or their family as a way of saying “this is why what I do is important.”

Obviously you’ve spent a lot of time thinking about all this during the last year, and you’ve probably been asked a million times if you think this is all creepy or intrusive in some way. Is it?

I’m an optimist. Every new tool can be used for good or evil. The whole point of doing this project is to start a conversation about it all. The people who are thinking most about big data right now are corporations and governments. I’d like to broaden the conversation and I hope the book makes some kind of contribution. I’m worried that the only ones profiting from it right now are corporations. As individuals we have very little say about how our data is being used. I’m not worried about the privacy implications of it so much. But it seems to me that as an individual, if I’m the one generating the data, I should have some kind of say in how it’s going to be used.

Did you have a particular favorite anecdote or photograph?

I just came back from Australia, and they have this expression down there: Gobsmacked. I think a lot of the pictures in the book convey that feeling. There are some that are funny, some that are just thought-provoking. There’s the case of the Environmental Systems Research Institute (ESRI) that creates these incredibly detailed satellite maps for governments. They found there were villages in Nigeria, which has the highest rate of polio resurgence in the world. There are villages there that have never shown up on any map, no one in the government knew they were there. ESRI can recognize the shape of huts and pathways. The Gates Foundation has been trying to eradicate polio in places like Nigeria, and they have a very big effort there. They took the satellite maps and handed out 10,000 GPS-enabled cellphones to polio workers. They could see where they were in real time, and make sure they got to each of the houses. We spent a week travelling with the polio workers watching them do their work. I think the idea of using satellites to help cure polio is a pretty interesting concept.

You have a lot of examples where understanding of big data is saving lives, which I think will surprise some people who don’t initially see it as having direct benefits for real people. What are some others?

There’s the case of the recent earthquake in Japan. I heard a fascinating story by Kai Ryssdal on Marketplace Radio about how 43 seconds before the shaking actually began, all the bullet trains and factories in Japan stopped running. It was all automated. That country spent 15 years and half a billion dollars to build the system that automated all of this. Obviously the devastation was horrible, but the system worked. Then I read about a group of engineers in Palo Alto that had created a program called Quake Catcher that uses the accelerometer in your laptop. Its the part in your laptop that detects when it’s been dropped and quickly moves the head on the drive drive before it smashes to the ground. It uses the same acceleromter to detect earthquakes. If it senses vibration and sees the same pattern over a 30-mile area, that’s an earthquake. On one side of the page, you have this huge half-billion dollar project, hardwired, dedicated parts that have to be replaced, lots of engineering time. And on the other you have this free ubiquitous crowdsourced mobile sensor system that has no profit motivation, and no cost. I love it. It’s a delightful story of people doing this to help each other. And the data just underpins it all.

Is there anything in the book that has some practical, everyday value?

Yes. There’s the example of Shwetak Patel, he’s a MacArthur Fellow and teaches at the University of Washington. He found a way to detect every device in the home and measure how much power it’s using. Every month we get a bill from the power company and we just pay it, we don’t even ask what it’s about. He’s created a sensor that can be plugged in anywhere in the house that detects the unique digital signature of everything that’s drawing power in the house — your computer, your toaster oven, whatever. I asked him if there was anything he had learned that would surprise the average American. He said it’s the DVR. The average American spends 11 percent of their monthly electrical bill on their DVR. It was designed in such a way that the hard drive never spins down, so even if you record only one show a week, it’s running the entire time and consuming power. So instead of drilling another oil well or burning more coal you could reduce America’s power bill by 5 percent just be redesigning the DVR. So many stories have this sense of delight: The data has been there all along, it’s just that no one was paying attention to it.

You’ve also done iPad and smartphone apps to enhance the book. What can you tell me about that?

I don’t know that anyone has ever done this with a book like this before, but there’s a free app you can download to your smartphone. Some of the pictures in the book have this little yellow key symbol in the corner. When you have the app, and you point the app at the page, it launches the page in the app. There are videos, there are Ted Talks. I think there are 22 or 23 videos. There’s an animated version of a story about pizza delivery guys in Midtown Manhattan. There’s also an iPad app, the profits from which go to charity: water, which is a nonprofit that’s working on bringing safe drinking water to people in developing nations. My goal here is to keep people turning the pages.

What do you want people to be left with in the end?

There’s an essay toward the back of the book called “Data Driven” by Jonathan Harris that has a really interesting thought. It’s that there is a relatively small group of people who are living in cities like San Francisco and New York, are mainly between the ages of 22 and 35, who are having an outsized effect on the rest of the human species. The kinds of societal changes that used to be the result of wars and famines are being brought about through software. …What I like about the essay is that EMC, which funded the book, had no right of review. I told them that this book wasn’t going to be all about cheerleading big data as the solution to all our problems. I said it was also going to sound a cautionary note because I think that right now governments and corporations are the ones having conversations about big data and that the average person isn’t. But it can have an effect on so many things in our lives, from our credit rating to our ability to get hired and our ability to do lots of things. I think it’s really important that we have this conversation now.

Managing systems logs is one of the more unglamorous tasks that come with managing IT infrastructure — not that there’s much glamour in the first place — but today is the sort of day when it takes on a cooler sheen.

Today, a fascinating and fast-growing start-up called Sumo Logic announced that it has taken a $30 million Series C round of venture capital funding led by Accel Partners. It is the latest investment by Accel’s Big Data fund, led by partner Ping Li.

The round brings Sumo Logic’s total capital raised to $50.5 million. Prior investors include Greylock Partners — Workday CEO Aneel Bhusri sits on the board — and Sutter Hill Ventures.

Sumo Logic’s approach to handling logs is to treat them like any other big-data problem. It uses cloud-based software to analyze, monitor and visualize data generated by machines in real time. System logs are just another one of those collections of data that used to be considered disposable or difficult to sort through. Sumo aims to make them a key source of insight into how well — or not — IT infrastructure is running.

Sumo often gets mentioned in the same breath as Splunk, the analytics company that specializes in machine-generated data and which had a successful IPO earlier this year. Splunk even spun off a company called Loggly that specializes in log management.

I talked with Sumo Logic’s CEO Vance Loiselle on Monday. He said that companies have been looking for years for a good way to easily make use of their log files. “In the last couple of years, we’ve seen more and more companies are generating tons of log data, but also all kinds of unstructured data,” Loiselle said. “That could be for servers that are or are not running well, or applications that you need to know specific analytic data about its performance, what your customers are using, and what kind of performance they’re seeing.”

Companies have been looking for a way to get ahold of that data and do a deep dive on the patterns that can be found in it, in order to decide if changes need to be made or to see how different parts of the business are performing. Doing that analytics work in the cloud, and thus saving on the purchase of extra storage hardware plus software, is sort of a no-brainer, but up to now that’s how this work has tended to be done.

The company launched only this February. More than 800 companies have tried the free service, and already more than 40 have traded up to the paid service. Among them are Netflix, Ooyala and Limelight Networks.

When you’re rolling out a new application, the last thing you need is some weird error cropping up somewhere in the chain of servers or on the network and getting in the way. Sumo Logic’s engine quickly crunches through the log data and can be used to pinpoint exactly where the problems are so they can be corrected, Loiselle said. It also works with log files generated by virtual machines.

Loiselle said the investment will help the company grow its sales and marketing operation, but also to do some of the heavy lifting needed to build up some new core features. “We need to bring in as many high-caliber inside and outside sales people as we can because the opportunity before us is so big. First we’ll scale up in North America, and then we’ll start thinking about what we’re going to do in Europe.” There’s also a huge need to hire more engineers, he said. “We’ve doubled the size of the engineering team over the last six months, and basically we have to double it again.”

Apparently, this is now a officially a thing, so here are some stats taken from the latest snapshot as of 3 pm ET, because we just know you’re not shopping on a tablet, you’re on the edge of your seat waiting to hear about how many others are:

• Online sales are up 20 percent for this same time period over Black Friday 2011.

• The number of consumers using a mobile device to visit a retailer’s site is at 28 percent, up from 18.1 percent in 2011.
• The number of consumers using their mobile device to make a purchase is 14.3 percent, up from 10.3 percent in 2011.
• Shoppers using the iPad led to more retail purchases more often per visit than other mobile devices, with conversion rates reaching 4.2 percent, higher than all other mobile devices.
• Shoppers referred from social networks like Facebook and Twitter generated 0.18 percent of all online sales on Black Friday.

So, you might be wondering how IBM gets all this info. It’s all part of its strategic play in the world of big data, essentially helping companies make more sense of the huge troves of data they’ve gathered that were previously being ignored. Smarter Commerce is the area of IBM devoted to helping retailers better understand that data so they can come up with improved ideas concerning how to sell more stuff.

Where they gather that data is the IBM Benchmark. It’s a cloud-based digital analytics platform that soaks up digital information about how consumers respond to different ways of selling things online, 24 hours a day, seven days a week, all year long, from 500 different online retailers. IBM won’t name them — they joined the network under condition of anonymity — but Big Blue says the companies that participate include about half of the companies named on the Internet Retailer Top 100 list. A lot of the technology comes from Coremetrics and Unica, acquisitions IBM made in 2010.

Last year, I talked about all this with Craig Hayman, IBM’s VP of the WebSphere, Application and Integration Middleware Software Division of the IBM Software Group. One quote from that conversation sticks out in my memory; it bears repeating here:

“If you think about consumers, and you think about the amount of technology that they have at their hands, to reach out to read reviews and talk to friends and families, they’re incredibly empowered. There’s not one purchase decision that they make that is not impacted by some element of social networks. What does that do to the companies that have to deal with that by offering the best products and services, and you see companies are struggling to do that: To make the right offer at the right time with the right price. When they do it well, we all talk about how it went well; and when they do it badly, we talk about how annoying it was.”

Now you know. Not only are retailers and your credit card companies watching you shop, so is IBM.