L03 - Clustering

Contents

• Hierarchical Clustering
• K-means Clustering

Recommended reading

Programming Collective Intelligence

• Chapter 3

Mining of Massive Datasets

• Chapter 7

Lecture slides

Available for download here.

Recording

Note

There is a small error in the Pearson pseudocode in the recording. Pearson similarity should be calculated between two blogs, not a blog and a user as the function declaration incorrectly stated. The slides PDF is updated with correct function declaration.

Video

2019-11-07 20:05

📕 Errata

The size of the blog dataset

The recording says that the blog dataset has 120 blogs. The dataset used in A2 has 99 blogs and 706 words, which is also what the lecture’s own numbers later on (98 merges, 161,700 comparisons) are based on.

• Introduction
• The blog dataset
• Hierarchical clustering
• Dendrogram
• Closeness and Pearson distance
• The tree data structure
• Merging two clusters
• The hierarchy generation algorithm
• Displaying the result
• Performance issues
• K-means clustering
• Centroids
• The k-means algorithm step by step
• The k-means algorithm in code
• When is it finished?
• The Web Clustering Demonstrator
• Hierarchical vs. k-means
• K-means on the blog data

Summary

Discovering groups

Recommendation systems find users who are similar to you. Data clustering is a related method for discovering and visualizing groups of closely related items: people, products, blog entries, and so on. The lecture covers two algorithms, hierarchical clustering and k-means clustering, and both are part of A2.

The blog dataset

The lecture’s dataset is the one used in A2: blogs, with the number of times each of 706 pre-selected words appears in each blog’s feed. It is a tab-separated text file with one row per blog and one column per word. The word kids, for example, appears three times in Gothamist, never in GigaOM and twice in Quick Online Tips.

The idea is that blogs with similar word frequencies are likely to write about similar things, so clustering them can be useful for searching, cataloguing and discovering blogs. You can also build a new dataset of your own with a feed parser, or about another topic than blogs.

99 blogs, not 120

The recording says the dataset has 120 blogs. The dataset on the Datasets page, and in A2, has 99 blogs and 706 words. The lecture’s own numbers later on (98 merges, 161,700 comparisons) also match 99 blogs.

Hierarchical clustering

Hierarchical clustering builds a tree of groups by repeatedly merging the two most similar groups. Every blog starts as a group of its own. In each iteration the distance between every pair of groups is calculated and the two closest groups are merged into a new one. This repeats until only one group remains, the root of the tree.

The result can be drawn as a dendrogram: a tree diagram where each branch point is a merge, showing the order in which the groups were joined.

Pearson distance

The algorithm depends on a measure of how close two blogs are. Some blogs contain far more words than others because they have more or longer entries, and Euclidean distance handles that badly: two blogs about the same topic would look different just because one of them is longer. Pearson correlation only looks at whether the word counts go up and down together, so the lecture uses Pearson.

Pearson returns 1.0 for a perfect match and −1.0 for no match at all, but a distance should be small for similar blogs. It is therefore inverted:

distance = 1.0 - pearson(A, B)

The calculation is the same as for users in L02, but runs over all 706 words of the two blogs instead of over common ratings, so n is always 706.

The tree data structure

The clusters are modelled as a tree of Cluster nodes. A node holds either a blog (a leaf) or two branches, left and right, to other clusters. It also stores the distance between its two branches (0 for a leaf) and a reference to its parent.

Merging two clusters A and B means:

1. Create a new cluster P with P.left = A and P.right = B, and set A.parent and B.parent to P.
2. Create a new blog for P whose word counts are the average of the word counts in A and B, so that P can be compared with other clusters like any blog.
3. Store the distance between A and B in P.

To build the hierarchy, start with one cluster per blog. In each iteration, compare every cluster with every other cluster, merge the two closest ones, add the new cluster to the list and remove the two old ones. Repeat until one cluster remains. Displayed as a tree, related blogs end up in the same branches, such as the Google and search engine blogs, or a group of gadget and gaming blogs.

Performance

Comparing two blogs over 706 words takes time, and the algorithm does it a lot: with 99 blogs it merges 98 times and calculates 161,700 distances in total. Think about where you can avoid repeated work when you implement it. Each pair of clusters only needs to be compared once per iteration, for example, and most distances don’t change between iterations.

K-means clustering

Hierarchical clustering has two drawbacks: it doesn’t divide the data into distinct groups without extra work, and it is slow. K-means clustering is different: you decide in advance how many clusters you want (k), and the algorithm works out which items belong to each cluster.

The algorithm is built around centroids, points in the same 706-dimensional space as the blogs that represent the centre of a cluster. Each centroid starts at a random position: for every word, a random count between the lowest and highest count of that word in the dataset. The word china, for example, occurs between 0 and 11 times, so its random count must be between 0 and 11.

1. Place k centroids at random.
2. Assign every blog to the closest centroid (using Pearson distance).
3. Move each centroid to the average position of the blogs assigned to it, word by word.
4. Clear the assignments and repeat steps 2–3.

The algorithm has to stop at some point: either after a maximum number of iterations, for example 20, or when an iteration produces exactly the same assignments as the previous one. A2 asks for the first for grade E and the second for grade C–D.

A centroid without blogs

If a centroid ends up with no blogs assigned to it, the average in step 3 divides by zero. Leave such a centroid where it is (or give it a new random position) instead of dividing.

Hierarchical or k-means?

K-means needs very few iterations compared with hierarchical clustering and is much faster. Because the centroids start at random positions, the result can differ between runs, while hierarchical clustering always gives the same result. With k-means you also have to choose the number of clusters in advance.

Two k-means runs on the blog data give somewhat different clusters, but blogs that are very close to each other, such as the Google and search engine blogs, usually end up in the same cluster. That is a useful way to check that your own implementation works.