Inside the Recommendation Engines of StumbleUpon, YouTube, Pandora and Hotpot

When recommendations are done well, they lead to more engaged and satisfied users. But that’s hard to quantify or make obvious. Pandora, for instance, can go for a seemingly long time without major feature changes, when behind the scenes “there’s an army of people making changes to playlist infrastructure,” according to Pandora CTO Tom Conrad.

In some ways, discovery is the opposite of search. About 50 percent of searches on YouTube are “broad,” according to YouTube Director of Product Management Hunter Walk, by which he said he means users are seeking an experience rather than a particular video. So YouTube’s intent with recommendations is less about one right answer and more about a cluster of answers, or eventually, a narrative.

One way to measure the success of recommendations is to ask users to rate them directly, which is a major component of Pandora’s and StumbleUpon’s systems. Pandora, for example, has eight billion thumbs-up and thumbs-down actions in its index. StumbleUpon has an 80 to 85 percent thumbs-up percentage, said StumbleUpon CEO Garrett Camp. Users look at a page they have stumbled upon for an average of about eight seconds, and a median of 20 seconds, he said.

The social graph is only a small part of good recommendation engines, Conrad, Walk and Camp agreed on a SXSW panel about content discovery. Rather, a recommendation engine is a mix of art and science hinged on refining small changes over time and understanding how users respond to them.

I moderated the panel, so I’m sure I missed writing down all sorts of interesting tidbits, but I wanted to share some of the insights and stories.

In addition to the plaid-shirted trio of Conrad, Walk and Camp–speaking on music, video and Web page recommendations, respectively–we added Google Hotpot product manager Lior Ron to the panel after bumping into him in the hallway. (Hotpot is Google’s newly launched personalized local recommendations site.)

Walk warned to be careful with analytics, saying that positive stats about recommendation performance don’t necessarily correlate with a good user experience. As YouTube has improved its recommendation algorithm, it has negatively impacted the number of video playbacks it gets.

That’s because better recommendations reduce the number of times people start watching and then skip a video, Walk said. Yet the company has almost tripled session length in the last few years as it has reduced these skips, and recommendations have helped improve the overall experience.

One of the most significant improvements YouTube has made to its recommendation technology, Walk said, was in messaging and presentation. When the company started explicitly stating why it was recommending a video–for example, you should watch this Britney Spears clip because you just finished Justin Bieber’s latest post–satisfaction improved.

As Walk explained, “If it was wrong they didn’t blame us; They blamed themselves.”

Ron said there’s plenty of room for companies to play with recommendations, and that he’d welcome more Netflix Prize-like approaches that might introduce significant improvements. “We’re not living in a world with millions of recommendations and we need to turn it down,” he explained.

But often discovery is about that one right thing at the one right time. Conrad said Pandora’s goal for recommendations is to take the smallest possible signal and provide the best possible personalized results. The Internet radio company A/B tests everything it rolls out to thoroughly measure the impact on user experience.

Pandora has experimented with social–for instance implementing Facebook’s instant personalization feature–but it’s just one signal of many. Conrad said the company quickly realized that it shouldn’t have assumed friends had the same taste in music.

(I can vouch for this awkwardness, having loaded up Pandora while logged into Facebook and gotten a recommendation for a curated Celine Dion channel.)

Camp said that for his purposes, the social graph is too much of a closed loop, taking away from the serendipitous recommendations that StumbleUpon works hard to deliver. The company reserves five percent of its stream for entirely new stuff, he said.

StumbleUpon also doesn’t try to hard to figure out what’s on a Web page in its index, because its point of the service is to deliver a variety, rather than a homogenous stream, of content.

YouTube would like to add features that show “what your friends haven’t watched,” said Walk, so users can have the satisfaction of discovering something rather new, rather than the disappointment of learning after sharing a viral video that they’re the last to know about it.

Ron disagreed to some extent, at least for his domain area of restaurants and other local establishments. Knowing where your friends have gone is one of the more important signals, he said, especially knowing where your friends have gone recently.

One more funny anecdote: Pandora users tend to create channels for Christmas music at that time of year. But the company ran into a problem because there was an indie band named “Christmas.” It was worried that users looking for cheery carols would get an unexpected and dissatisfying stream of random indie music.

But before Pandora could introduce a fix, users took care of the problem by thumbing down those selections, effectively filtering out the Pandora spam (accidental or otherwise) of the band named Christmas.

Images grabbed from the panelists’ profiles on their company sites.