This is the second of three interviews with members of the Google (GOOG) team responsible for overseeing search algorithms at the company. The introduction and Part I, an interview with Scott Huffman, appeared yesterday. In today’s installment Google software engineer Matt Cutts talks about search quality and spam. In Part III tomorrow, Google Fellow Amit Singhal will wrap up the series.

Google and the Evolution of Search

1. Human Evaluators — Google Engineering director Scott Huffman
2. Cheating the System — Google software engineer Matt Cutts
3. What’s Next in Search? Much, Much Better Search — Google Fellow Amit Singhal

Part II: Matt Cutts

John Paczkowski: How do you maintain quality in search?

Matt Cutts: Well, broadly, we improve our algorithms and hopefully, every so often, develop some punctuated equilibrium where we create totally new ways to improve our relevance. My contribution… is ensuring that people who try to cheat the system don’t show up higher than they deserve to in our results. We want sites ranking high based on merit, not based on shortcuts.

JP: OK, so how do you do that?

MC: Essentially we look at a wide variety of input. We look at user complaints, for example. We also have a variety of internal metrics we use to track current trends. They help show us what people are using to spam right now. What’s getting past our defenses. And when we detect those things, we write some new algorithms or develop some tool that helps us detect and, hopefully, counteract them. So a large part of what we do is simply spotting trends in spam.

JP: Is there a human evaluation element here as well?

MC: Each team is responsible for general search-quality evaluations, but it’s not like they’re changing rankings or anything like that. That said, there are some policy violations that are pretty egregious. So, for example, if you type in your name and instead of getting All Things Digital, you got a porn site, you would get pretty angry about that. And you might complain to Google. And it would be frustrating if our reply was, “Yeah, well, we think we might have an algorithm that might fix that problem in five or six months, so we’re just going to leave that porn site as the top result for All Things D until we get an algorithm up to help you out.” Obviously, that’s a deeply dissatisfying answer.

So in spam, we are sometimes willing to take manual action on those sorts of policy violations. But Google’s philosophy is that wherever you can use machines and algorithms, it is much better, more robust, more scalable. And so, to the extent that we can, we always want to rely on the computers as our first line of defense.

JP: But you’re willing to remove spam manually until you can find an algorithm to counteract it. Do you think that will always be the case? Will we some day reach a point where human intervention of the sort you just described won’t be necessary or are we headed toward increasing human intervention?

MC: That’s a really fascinating question, but I don’t know the answer. What’s interesting to think about is that page rank, the raw page rank algorithm, actually improves as it ranks more pages. So the more pages you add to it, the easier it is to determine how reputable a particular page is without human intervention.

But as the Web grows in size we also encounter new and different policy violations–hidden text, cloaking. Those are the sorts of things that humans are very good at spotting. You can certainly identify some of them with a computer algorithm, but not all. And so our intent is always to try to make sure that we handle things efficiently with machines and algorithms. But I don’t know that we will ever get there completely.

Google and the Evolution of Search

1. Human Evaluators — Google Engineering director Scott Huffman
2. Cheating the System — Google software engineer Matt Cutts
3. What’s Next in Search? Much, Much Better Search — Google Fellow Amit Singhal

The goal is to enable Google users to be able to ask the question such as ‘What shall I do tomorrow?’ and ‘What job shall I take?’…We are very early in the total information we have within Google. The algorithms [software] will get better and we will get better at personalization.

– Google CEO Eric Schmidt

For many years, Google (GOOG), on its Explanation of Our Search Results page, claimed that “a site’s ranking in Google’s search results is automatically determined by computer algorithms using thousands of factors to calculate a page’s relevance to a given query.”

Then in May of 2007, that statement changed: “A site’s ranking in Google’s search results relies heavily on computer algorithms using thousands of factors to calculate a page’s relevance to a given query.”

A slight adjustment in wording, but an important comment on the supremacy of the algorithm that Google had touted for years. Google had finally acknowledged that its search results were no longer solely and automatically determined by the company’s vaunted algorithms. Now they simply “relied heavily” on them. Why the sudden change?

Google claims it was arbitrary, unrelated to any sudden philosophical shifts within the company. But it seems far too specific an adjustment to chalk up to a random brand-management edit. We are, after all, talking about the company’s official explanation of its search results. And indeed, sources say the language was changed to account for the continual calibration of the algorithm, which these days is done with a bit of human help.

Google, for example, employs a vast team of human search “Quality Raters” (You’ll find a copy of an old training manual here). Spread out around the world, these evaluators, mostly college students, review search returns against established criteria–testing different algorithms and see which works “best” in predicting the quality of a site (though not directly judging the quality of any individual site itself).

They’re aided by Google’s own registered users, who can now, when logged into their Google accounts, promote and delete sites from their own search returns according to their preferences. These data too are used to tweak and further optimize the algorithm. So Google’s objective evaluation and ranking of Web sites is to some extent defined by subjective reasoning of a collective human intelligence. And so it must be if Google is to continue returning search results that we perceive to be the “best” answers to our search queries.

In interviews serialized over the next three days, key Google engineers with central roles in managing the company’s search engine discuss resources and techniques they use to optimize the system for users world-wide. The series kicks off below with Engineering director Scott Huffman, who oversees the company’s search evaluation team. Senior Google software engineer Matt Cutts appears tomorrow. And Google Fellow Amit Singhal wraps up the series on Friday.

Google and the Evolution of Search

1. Human Evaluators — Google Engineering director Scott Huffman
2. Cheating the System — Google software engineer Matt Cutts
3. What’s Next in Search? Much, Much Better Search — Google Fellow Amit Singhal

Part I: Scott Huffman

John Paczkowski: How do you maintain quality in search ranking?

Scott Huffman: We are constantly evaluating the quality of our results in something like a hundred different locales and language tiers all around the world. So every day, we are looking at a random sample of grades that we think represent the queries we get from users. Evaluators look at the quality of each result relative to those queries. We are constantly tracking a pretty wide array of different kinds of quality signals that come through our text.

JP: Talk a bit more about the human element here. You’ve hired people to evaluate pages?

SH: Yes, we have folks around the world who are trained to evaluate the quality of results. We like them to be in-country so they understand the culture and that type of thing. And then we have a work flow system that feeds them different kinds of evaluation tasks. Things like “tell us how good you think this result is for this query.” And then out of the data, we produce a set of aggregate metrics that we look at and that we can track over time.

JP: So how many of these evaluators are there?

SH: How many? I don’t think we can talk about the exact number, unfortunately.

JP: Ballpark? I’ve heard 10,000.

SH: Well, the number actually is pretty large and that’s for a couple of reasons. One is that, like I mentioned, we try to do an evaluation pretty broadly across all of the locales Google is in, and there are a lot of them. So you’re already talking about a pretty large group of people. Secondly, we prefer a larger group to a narrow one because we want to use our evaluations to give us an independent picture of our quality. We get a lot of queries from all over the world so we need a broad base of people to help us understand how good our results are for them.

JP: So are these raters college students or random folks responding to a job post? What are the requirements?

SH: It’s a pretty wide range of folks. The job requirements are not super-specific. Essentially, we require a basic level of education, mainly because we need them to be able to communicate back and forth with us, give us comments and things like that in writing.

JP: And how are they trained?

SH: The training is pretty simple. There are manuals and video training and, ultimately, participation in the rating program. We help them understand what it means for search results to be highly relevant and useable for the viewer. Is there a dominant result for a particular query today? If so, it should be right there at the top. Take a broad-based query like…“Olympics.” If a user searches for “Olympics,” the results from the 1996 Olympics are not as interesting as the ones from the 2008 Olympics.

JP: So how do you vet data provided by the raters? Is there any quality control?

SH: Well, the raters work in-country, so we don’t see them everyday. And we don’t typically talk to them on the phone. We have some automated measures that account for things like, say, evaluators who consistently say two sites in a side-by-side comparison are about the same. We also have moderators. But ultimately, the real quality control is done by the folks who are working on ranking and search UI. They’re the ones who understand why we are better today in China than we were a week ago or a month ago. What changed? What are we are doing better? The evaluation program really just gives our engineers an aggregate measure of how good their algorithms are so they can improve them.

JP: So you’re describing a process in which these evaluators are going to specific Web pages and rating them according to a specific criteria. Do these data have any effect on those sites’ page ranks or pay-per-click and Ad Word bids?

SH: We don’t use any of the data we gather in that way. I mean, it is conceivable you could. But the evaluation site ratings that we gather never directly affect the search results that we return. We never go back and say, “Oh, we learned from a rater that this result isn’t as good as that one, so let’s put them in a different order.” Doing something like that would skew the whole evaluation by-and-large. So we never touch it.

JP: Let’s backtrack a little bit. How did this project begin? Who came up with it? What were its origins?

SH: Well, from the earlier days of Google, of course, we have always been interested in measuring how well our search algorithms are doing. I wasn’t here, but what I understand is that way back when there was a set of Sergy’s favorite 10 queries, people would run those and they would make sure that any change they made to the ranking algorithms would make those work. Obviously, as Google grew in traffic and reach, it needed a broader set of queries, and there was a realization that we really needed to have evaluators in the countries we service who understand the culture to do that well. We needed a team that could evaluate results from the users’ perspective.

Google and the Evolution of Search

1. Human Evaluators — Google Engineering director Scott Huffman
2. Cheating the System — Google software engineer Matt Cutts
3. What’s Next in Search? Much, Much Better Search — Google Fellow Amit Singhal

Gabriel Stricker, Google’s Director of Search Communications kicks things off by noting that the company will be sharing a number of new developments that cater to the growing demands of its users. With that, Udi Manber takes the stage to offer a big-picture overview of search.

Manber says what Google does is the new “rocket science.” Search has to be fast, relevant, and fresh, he explains. But even that’s not enough. The real goal is to solve users’ problems. If users can’t spell, it’s our problem. If the content is there but in a language the user doesn’t speak, that’s our problem. If the Web is too slow, it’s our problem. Manber offers a few examples of how Google works to address these challenges: real-time data, translation, etc. With these services nailed down, he says, Google can move on to the more important task of working on “understanding.”

Manber invites Pat Riley, senior search quality engineer, to the stage to talk a bit about Google’s “did you mean” link. Lots of people use the link, Riley says, and Google has been working to improve it. Called “spellmillion,” the project provides not only related results for a misspelled query but for alternate ones as well (think labor as in “work” and labor as in “pregnancy”). But it requires Google to process multiple searches for a single query and demands a lot of processing power.

Riley notes that the project has been somewhat contentious because it also potentially questions user intent. He offers the example of “Macy Ray.” Some users might be searching for “Macy Gray,” the singer, others for a person actually named “Macy Ray.” How do you address those two potential queries on a single search results page?

Riley is followed by Engineering Director Scott Huffman, whose subject is mobile search. Huffman starts things off with a few truisms. Mobile search is often local. It should be easy to use. Effortless. And it should provide all that Google has to offer. Huffman notes that this is quite a task since Google must optimize its search for different mobile experiences and different user interfaces: Google’s own Android, Apple’s (AAPL) iPhone, etc. Some of these platforms require gestures–touch, swipe–others use a keypad. All must provide access to the Web and the mobile Web–sites that have been optimized for mobile devices. On the screen behind him, Huffman displays an example of Google search that displays desktop Web results and mobile Web results, the latter denoted by a red square.

Mobile search must also be easy. Huffman demos a shared desktop-mobile search for a flight number. Since he’s logged into his Google account, his search for “ba 284? SF-London on the desktop is immediately shared with the Google app on his mobile device. An unreleased feature, but it’s on its way. A quick look at local listings automatically delivered to devices on the basis on GPS/cell tower location, and then Huffman brings Mayer on stage.

Mayer talks a bit about universal search before moving on to Google’s “bento box” of search results. She talks about Google’s focus on the importance of presentation and its efforts to make search results more usable for the user. An example of this SearchWiki, a tool that allows users to annotate their searches, to “keep their train of thought,” says Mayer. We need to help our users find more and do more with it, she says, noting that the company is still working to address some longstanding user problems:

• Finding recent information
• Expressing that you want just one type of result
• Assessing which results are best
• Knowing what you’re looking for
• Expressing your searches in keywords

Mayer introduces Google Search Options, a feature that appends a search option panel to results, allowing users to “slice and dice” the results as they choose. A demo of the feature, in a search for “Hubble Telescope,” allows for search calibration by time, pages that include images, etc. Another search for “solar oven” is filtered down to specific genres–videos, discussion forums, reviews. Click on those links and that new search context is immediately displayed on the page.

Interestingly, the reviews feature uses something called “sentiment analysis” to extract sentiments from a review and present them in displayed snippets.

Search Options also includes a timeline feature that allows users to visualize results over time. And there’s something called “Wonder Wheel,” which presents a visual representation of a query surrounded by potential refinements (hence “Wonder Wheel”). Click on a refinement and results update automatically. Search Options should be going live now, says Mayer.

A bit of geometry monomania here today at Google Searchology. First the Wonder Wheel and now “Google Squared,” a sort of spreadsheet visualization of search being cooked up in Google Labs. Unstructured data pulled directly from search and organized according to the whim of the user. A search for “small dogs” pulls up a lists of–wonder of wonders–small dogs organized by size, weight, breed, etc. Click on an individual cell and you can change its source. Pretty slick. Still a work in progress, though. It should be available later this month, Mayer says during the Q&A.

Another new feature: Rich Snippets. A search for “drooling dog BBQ” returns your standard Google results along with a list of metadata–average user reviews, for example. A search for a GPS system includes an additional pointer to a recent CNET review of the unit in question. Rich Snippets is open API, incidentally.

Last up, an Android star map app that uses GPS to create a star map “local to your place on earth” and to your position. Move the phone and the map adjusts to your view–essentially the app transforms the device into map overlay for the sky. And how does this tie into search? Search for “Gemini” and a sort of pointer appears onscreen directing you to its location in the sky. And with that, Mayer wraps things up.