Liz Gannes

Recent Posts by Liz Gannes

Andreessen and Mixpanel Call for an End to “Bullshit Metrics”

bullshit3802

Creative Commons image via toolmantim

Page views, registered users, app downloads. A hundred thousand of these, a million of those. These are the numbers and milestones tech companies brag about, both to press and investors.

But honestly, Web sites that require you to repeatedly click through to see additional pages are often just crappy. Download counts can be easily inflated if an app developer is willing to pay. Registered user counts don’t show whether something is the next big hit or a passing fad.

Analytics company Mixpanel and its investor Andreessen Horowitz are trying to persuade the tech world to be more honest with itself by reporting numbers that are far more informative: Engagement and retention. (Yes, those happen to be the very things Mixpanel measures, but as I’ve written before, they make a lot of sense.)

Some people call page views and the like “vanity metrics,” but Marc Andreessen and Mixpanel founder Suhail Doshi have decided they want to raise the shame level by calling them “bullshit metrics.”

Andreessen told me in an interview last week, “People think they’re richer if they have Zimbabwean dollars than U.S. dollars.”

“We and other investors need to get more vocal,” Andreessen said. “Page views and uniques are a waste of time.”

Andreessen said his firm won’t throw start-ups out the door if their pitches include bullshit metrics — but it’s perhaps something they might consider.

And it’s not only start-ups that try to push big numbers over reality; Google famously obfuscated Google+ user counts as it was getting its social efforts off the ground.

The problem is, once one company reports a bullshit metric — like how many registered users it has — its competitors don’t want to appear smaller by reporting something real — like how many monthly active users it has. So the bullshit metric shaming needs to be a broader movement for it to work.

(To which I say, where are the bumper stickers?)

“We can do better as an industry. We should do better because collectively we’re not benefiting — we’re all just fooling each other,” Doshi wrote in a blog post he’s publishing today.

Bullshit metrics do a disservice to the companies reporting them, Doshi said. When companies focus on numbers that are not meaningful, they aren’t optimizing their efforts for what could really help their business.

As an alternative, Doshi proposes — and he is not the first to do so — that every business has a natural goal that correlates with its success. For instance, Yelp benefits most when it has more reviews, and Instagram when it has more photos uploaded. Measuring that “one key metric” can lead to insights that are particular to that business, and optimizing for it can give the company an edge versus competitors that are not so fine tuned.

Here’s Doshi’s blog post:

Bullshit metrics

Every day feels the same. A fledgling startup tries to appear like the up-and-coming market leader while the market incumbent aims to protect its dominance. It has become exhausting to keep up with how fast everyone seems to grow: 100,000 new users per week here, 20 billion monthly pageviews there, and let’s not pass up a watershed moment like 3 million members total. These are the industry’s most praised metrics.

Sadly, we haven’t moved forward over the past decade despite our whole industry becoming smarter about how it measures and analyzes data. Companies still pitch investors with a cumulative user sign up graph, sell advertisers on how many pageviews they get, and bamboozle reporters with the biggest numbers they can find regardless of whether they correlate to success. We can do better as an industry. We should do better because collectively we’re not benefiting–we’re all just fooling each other.

So, enough with the bullshit metrics.

New metrics for the new world

The internet has evolved in the past decade. Websites are no longer static; they are dynamic. Businesses strive to build products instead of websites because they are more rich, more engaging, and more sophisticated. The web 2.0 era was characterized by its use of AJAX which made websites feel like applications instead of web pages. The concept of pageviews is dying. This is far more pervasive within applications we use on our mobile smart phones. Companies are still pressured to tout pageviews as a metric for success to entice advertisers to spend their dollars with them. And yet, users are not generating more pageviews while flipping through photos on Facebook. Extrapolating this idea further makes you consider what signups really mean. Facebook has done a great job with this. They undoubtedly passed their first billion signups long ago, but they’ve made a point to only talk about getting to their goal of 1 billion users who have been active in the past 30 days.

With any change in an industry, we need a mental model of mapping the old way of measurement to the new way.

Actionable metrics

Companies need to start using a new set of metrics that don’t simply make them feel good. They should use actionable metrics that provide insight, provide guidance, and help businesses make better decisions. I fundamentally believe that actionable metrics give companies a competitive edge.

Today, engagement is the better page view. Engagement is a specific action a user takes and it’s something specific to each business. Twitter can measure engagement by number of tweets and YouTube by the number of views a video receives. For Yelp, an actionable metric would be the number of reviews posted because more reviews likely lead to more users that find more value out of Yelp. More users means more reviews–a network effect. For Instagram, a mobile photo sharing application, an actionable metric would be the number of photos uploaded daily rather than the number of users that signed up and never did anything. Engagement helps us understand how actively people are using our products, whereas a pageview could simply be someone showing up to the product and leaving confused or unimpressed.

The most important metric today is retention. It tells you how often a customer comes back and uses your product again. Retention is an objective way to measure how valuable customers find your product so you can improve it if it’s not valuable enough. The best venture capital firms in Silicon Valley are beginning to expect this metric. Companies are starting to include this metric in their pitch decks to provide transparency into whether they can actually get a high percentage of their 100,000 newly signed up users per day to come back weeks later. The truth is, if less than 10% of people come back, then people don’t find the product valuable. To make matters worse, the percentage of people that come back as time goes on usually declines. Gaming best characterizes this decline because consumers eventually get tired as the replay value of a game diminishes. Now, only a small percentage of the original people who signed up are staying engaged. This particular situation is a reason the number of sign ups a company has attained is not an actionable metric.

But the most important thing is that companies find a specific, actionable metric they can focus vigilantly on.

Your One Key Metric

My experience (I am the co-founder of Mixpanel, an analytics company that helps companies like Airbnb, Khan Academy, Dictionary.com, Match.com, and Path) has shown that companies should start by tracking a single actionable metric that they can literally bet the company on. I call this their One Key Metric (OKM). Companies choosing their OKM realize they must pick an actionable metric because pageviews or sign ups aren’t harsh enough and don’t correlate highly enough with the success of their business.

The best part of OKM is that companies can measure other things related to it, to understand how to improve it. In the case of Instagram, their OKM is likely to be photos uploaded. They can measure the percentage of people that download the app and uploaded a photo, the percentage of people who come back and upload a photo a week later, the growth rate of photos uploaded month after month.

Understanding your OKM often leads to deeper, more valuable questions. The answers often surprise businesses and guide them better than before.

Cutting through

Despite the internet’s evolution, bullshit metrics perpetuate a constant cycle of poor understanding. Let’s strive to understand how our businesses are doing and to pick better metrics–the harsher, the better. Let’s stop fooling ourselves with numbers that don’t represent reality. And let’s push the industry forward as a whole because collectively we’ll all benefit.


Latest Video

View all videos »

Search »

When AllThingsD began, we told readers we were aiming to present a fusion of new-media timeliness and energy with old-media standards for quality and ethics. And we hope you agree that we’ve done that.

— Kara Swisher and Walt Mossberg, in their farewell D post