L04 - Search Engines

Contents

• Introduction to Search Engines
• Indexing
• Content-Based Ranking
• Inbound-Link Ranking
• PageRank algorithm

Recommended reading

Programming Collective Intelligence

• Chapter 4

Mining of Massive Datasets

• Chapter 5

Lecture slides

Available for download here.

Recording

Caution

The Wikipedia dataset has been updated since the recording, so the results in the last part of the recording are inaccurate. The slides have been updated to the new dataset.

Video

2018-10-04 28:29

📕 Errata

The Wikipedia dataset

The recording describes the dataset as 240 articles about video games and 399 about programming, and the slides as 250 and 400. The dataset on the Datasets page has 245 articles about games and 396 about programming.

The weights in A3

The demo at the end of the recording weights the metrics as 1.0 × word frequency + 0.5 × document location + 1.0 × PageRank. A3 uses the weights on the slides: score = word_frequency + 0.8 * document_location + 0.5 * pagerank.

Two slips in the slide pseudocode

In the query algorithm, for (scores.content[i] > 0) should be if (scores.content[i] > 0): a page is included only if a query word appears on it. In the normalization algorithm, the last line divides by max but should divide by max_val.

• Introduction
• The Wikipedia dataset
• Building a search engine
• Indexing a document
• Searching
• Ranking
• The search engine structure
• Generating the word list
• Content-based ranking
• The basic query structure
• Normalization
• Word frequency metric
• Document location metric
• Word distance metric
• Which metric to use?
• Inbound-link ranking
• Simple count
• The PageRank algorithm
• PageRank example
• Calculating PageRank iteratively
• Performance considerations
• PageRank implementation
• Testing the search engine

Summary

Full-text search engines

The lecture is about full-text search engines like Google and Microsoft Bing: systems that let people search a large number of documents for a word or a set of words, and that rank the results by how relevant each document is to the query. Companies like Google, Microsoft and Yahoo have spent huge amounts of money on improving search, and Google itself started as an academic project describing the PageRank algorithm.

Searching and ranking is a large field, so the lecture covers some key concepts on a small dataset: the Wikipedia articles about video games and programming used in A3. Search engines for the whole web have many additional problems, such as storage, that are not covered.

Indexing

A search engine needs a set of documents to search in. A web search engine collects them with a web crawler; a company search engine might index the documents on its intranet.

Each document is then indexed: a list is made of all the words in the document and where each word appears. For HTML pages, all markup, scripts and style sheets are stripped first, leaving a bag-of-words representation, a plain sequence of words. Storing every word as a string takes a lot of space, so a common optimization is to give each unique word an integer id and store each page as a list of integers, plus a table that maps words to ids.

In the lecture’s design, a PageDB holds a list of Page objects and a hash map wordToId. Each Page stores its URL and its words as a list of word ids. A new word gets the current size of the hash map as its id, so ids are unique without a separate counter:

int getIdForWord(String word)
  if wordToId.containsKey(word)
    return wordToId.get(word)
  id = wordToId.size()
  wordToId.put(word, id)
  return id

The index is built when the system starts and kept in memory, by reading every file in the Words/Games and Words/Programming folders of the dataset and creating one Page per file.

Searching and ranking

Once there is an index, searching is straightforward: convert the query to word ids and return every page that contains any of them. Whether a multi-word query should match pages with any of the words (super OR mario) or only pages with all of them (super AND mario) depends on how inclusive the search engine should be.

The hard part is ranking: sorting the results so that the most relevant pages come first. Each page gets a numeric score, usually a weighted combination of several metrics that can be tuned for the particular search engine. The basic query works like this:

1. For every page, calculate the value of each metric.
2. Normalize each metric so that all values are between 0 and 1, where 1 is best.
3. Calculate the final score as a weighted sum of the normalized metrics, for pages that contain at least one query word.
4. Sort the results with the highest score first.

Normalization

Some metrics give high values for good matches and others low values, so a common normalization function converts both to a score between 0 and 1, where 1 is best:

• If higher is better, divide every value by the highest value.
• If lower is better, divide the lowest value by every value, so the best page gets 1.

Use a small value such as 0.00001 instead of 0 to avoid dividing by zero.

Content-based ranking

Content-based ranking sorts results based on the contents of each page. Early search engines often used only content-based ranking, with usable results. The lecture covers three metrics:

• Word frequency: how many times the query words appear on the page. Higher is better.
• Document location: where on the page the query words first appear, based on the idea that a relevant word appears near the top of the page, maybe even in the title. For each query word, add the index of its first occurrence + 1, or a high value such as 100,000 if the word isn’t on the page. Lower is better.
• Word distance: for multi-word queries, how close together the query words are on the page, using the distance between the first occurrences of each pair of consecutive query words. A missing word adds a high value. Lower is better. This metric is expensive to calculate and only makes sense with two or more words.

No single metric gives consistently good results, so they are combined and the weights tuned, for example:

score = 1.0 * WordFrequency + 0.8 * DocumentLocation + 0.5 * WordDistance

Inbound-link ranking

Inbound-link ranking doesn’t look at the contents of a page at all. It uses what other pages say about it: who links to it. The idea is that bad pages are less likely to be linked to, while good pages have many other pages linking to them. For this, each Page also stores its outgoing links, as a hash set of strings (the dataset has them in separate files under Links).

The simplest version, simple count, counts how many other pages link to a page, much like academic papers are ranked by how often they are cited. On its own it ignores the query completely, so it must be combined with content-based ranking. It also treats every link as equally valuable, while a link from a high-quality page should count more. That is what PageRank does.

PageRank

PageRank was invented by Google’s founders and is named after Larry Page. Every page gets a score for how important it is, calculated from the importance of the pages linking to it. In theory, PageRank is the probability that someone who clicks on links at random ends up on the page. Since people stop clicking after a while, a damping factor of 0.85 is used: an 85% chance that the user keeps clicking.

To calculate the PageRank of page A, take the PageRank of every page that links to A, divided by the number of outgoing links on that page; multiply the sum by 0.85 and add 0.15:

PR(A) = 0.15 + 0.85 * (PR(B)/links(B) + PR(C)/links(C) + PR(D)/links(D))

With PR(B) = 0.5 and 4 links, PR(C) = 0.7 and 5 links, and PR(D) = 0.2 and 1 link, PR(A) = 0.15 + 0.85 × (0.125 + 0.14 + 0.2) = 0.54525. D contributes the most even though it has the lowest score, because it links only to A and gives all its score to A.

There is a catch: the calculation needs the PageRank of the pages linking to A, which in turn need the scores of the pages linking to them. The solution is to give every page an initial score, for example 1.0, and recalculate all scores over several iterations; each iteration brings them closer to their true values. For a small dataset like this, 20 iterations is enough. Calculate all new scores in an iteration before you update any of them.

PageRank is slow to calculate, but it doesn’t depend on the query, so it can be pre-calculated once when the system starts and normalized once afterwards. Data structures matter here: storing each page’s links in a hash set instead of a list made the calculation about ten times faster, from around 50 seconds to around 4 seconds in the lecturer’s implementation.

Testing the search engine

The final ranking in the lecture combines word frequency, document location and PageRank. Queries such as nintendo or java programming should return relevant pages at the top. Exact scores depend on the weights and details of your implementation, so compare against the test cases on the A3 page rather than the recording.