John Paczkowski

Recent Posts by John Paczkowski

Google and the Evolution of Search III: What's Next in Search? Much, Much Better Search

For many years, Google (GOOG), on its Explanation of Our Search Results page, claimed that “a site’s ranking in Google’s search results is automatically determined by computer algorithms using thousands of factors to calculate a page’s relevance to a given query.” Then in May of 2007, that statement changed: “A site’s ranking in Google’s search results relies heavily on computer algorithms using thousands of factors to calculate a page’s relevance to a given query.” In this third and final interiew with Google’s search team, Google Fellow Amit Singhal helps us understand why. Previously, Google Engineering director Scott Huffman talked about the company’s human evaluators and software engineer Matt Cutts discussed search quality and spam.

Google and the Evolution of Search

  1. Human Evaluators — Google Engineering director Scott Huffman
  2. Cheating the System — Google software engineer Matt Cutts
  3. What’s Next in Search? Much, Much Better Search — Google Fellow Amit Singhal

Part III: Amit Singhal

John Paczkowski: Talk a bit about the history of search evaluation and your role in it.

Amit Singhal: Search evaluation was born in the late 50′s and the early 60′s in the U.K. In the beginning it was very basic because back then, search was Boolean. The first evaluation measure was recall. You take a query and 100 documents relevant to it. How many of those documents does your search on that query retrieve? We quickly found out that it was very easy to get 100 percent recall. But we also found that our searches often returned a lot of irrelevant documents along with the relevant ones. So we came up with a second measure: Precision. That tells us what percentage of our search returns is actually good. So if a search returns 100 out of 100 relevant documents for a query, but it returns 1,000 documents total, its recall is 100 percent, but its precision is only 10 percent.

And those two measures or some combination thereof have evolved over time, and even modern search engines like Google use them. So since search began, there have always been teams in the lab judging how relevant a search return is to a human query.

JP: But relevance is a subjective notion.

AS: Right. But these evaluation measures don’t directly affect the results returned to our users. They are only used to evaluate whether an algorithm is working well or whether a new algorithm is working better than an old one. They don’t directly impact user experience. They are simply just calibration tools.

JP:Matt and Scott spoke at length about human search evaluators. Just how broad is their role at Google?

AS: Well, our search evaluation is based on many components. And one of those components is human evaluation. We have automatic systems as well–things that tell us if, for example, users suddenly stop clicking on a number-one result and instead begin clicking on the number-five result. Together these techniques tell us how well our system is doing at any point. And we do this in over a hundred languages.

JP: How do you balance fresh results with more historical ones?

AS: When is a fresh result more relevant than a historical result? That is a question… very important for our users and thus for our algorithms. So we evaluate queries for freshness—this query deserves freshness today, but it did not three weeks back. We do the same thing with documents. We are always asking how fresh is the document? How relevant? How useful? And we put the answers to those questions together purely algorithmically and present them to users in our universal search results. All this is done automatically. No human being is sitting there and saying GM is important today or Mumbai is important today. Because at the end of the day, human beings are far too prone to subjectivity to do it. Algorithms are not. And they can make the same sorts of determinations in hundreds of languages.

JP: How far have we come in search?

AS: We are still barely at the beginning. We are nowhere close to being done. Search is a hard problem, and the hard part about it is that user expectations are deep, and they keep going higher and higher as you keep improving search. And so search by no means is a solved problem.

JP: So what’s next?

AS: What is next in search? Much, much better search… universality of search, and by that I mean search where the user doesn’t have to go to YouTube specifically to search for video or to Google for documents. Whatever type of content is relevant to you should just show up in your search results. So search becomes focused on who you are and where you are. So it would be local to you as a person and it would be local to your geography as well. And those two things combined will give you universally relevant results much more relevant to you and to your locality.

JP: Circling back a bit to the role of human evaluators in search, do you think they will always be necessary? Will they be more necessary in the future or less necessary? How will their role change?

AS: I believe that the role of the human evaluator in search will be there until we can understand language by computers, which is a far distance from where we are today. You know, we have made great advances but by no means is our language understanding technology close to saying this person really meant to get this document or not.

Google and the Evolution of Search

  1. Human Evaluators — Google Engineering director Scott Huffman
  2. Cheating the System — Google software engineer Matt Cutts
  3. What’s Next in Search? Much, Much Better Search — Google Fellow Amit Singhal

Twitter’s Tanking

December 30, 2013 at 6:49 am PT

2013 Was a Good Year for Chromebooks

December 29, 2013 at 2:12 pm PT

BlackBerry Pulls Latest Twitter for BB10 Update

December 29, 2013 at 5:58 am PT

Apple CEO Tim Cook Made $4.25 Million This Year

December 28, 2013 at 12:05 pm PT

Latest Video

View all videos »

Search »

The problem with the Billionaire Savior phase of the newspaper collapse has always been that billionaires don’t tend to like the kind of authority-questioning journalism that upsets the status quo.

— Ryan Chittum, writing in the Columbia Journalism Review about the promise of Pierre Omidyar’s new media venture with Glenn Greenwald