L06 - Data and Learning

Contents

• Data and Data representation
• Common datasets
• Types of learning problems
• Training and validation
• Performance metrics

Recommended reading

Programming Collective Intelligence

• Chapter 1

Mining of Massive Datasets

• Chapter 1
• Chapter 12.1

Lecture slides

Available for download here.

Recording

Video

2019-08-30 22:34

📕 Errata

The ImageNet challenge

The recording describes the ImageNet challenge as an annual contest. The challenge (ILSVRC) was held from 2010 to 2017 and is no longer run; the chart shows results up to 2016.

• Introduction
• Data and data representation
• The Iris dataset
• The MNIST dataset
• The CIFAR-10 dataset
• The ImageNet challenge
• Supervised learning
• Unsupervised learning
• Reinforcement learning
• Training and validation
• Generalization and overfitting
• Using separate datasets
• Cross-validation
• Train-test split or cross-validation?
• Good or bad result? ZeroR
• Accuracy
• Types of errors
• Precision and recall
• F1-score
• Confusion matrix
• Example: the Iris dataset
• Other important characteristics

Summary

Data and data representation

Data consists of inputs and outputs. Each set of inputs and outputs is an independent example (or instance): one image of a cat, the test results of one patient with diabetes, and so on.

• The inputs are one or more (often many) values called attributes or features. They can be numbers (integers or floats) or nominal (categorical) values from a finite set of categories: car, boat, plane, the digit 2, the letter A.
• The outputs are one or more values called classes or categories.

Common datasets

• Iris, the “hello world” of machine learning: classify an iris flower as one of three species from four numerical attributes (petal and sepal length and width). 150 examples, 50 of each species, so the dataset is balanced. You use it in A4.
• MNIST, the “hello world” of image recognition: images of handwritten digits, with a training set of 60,000 examples, a test set of 10,000 and 10 categories (0–9). Each image is 28×28 grey-scale pixels, a matrix of values where 0.0 is white and 1.0 black (sometimes integers 0–255 instead). To use an image as input, the matrix is flattened to a vector of 28 × 28 = 784 values.
• CIFAR-10, a much harder image recognition problem: 60,000 colour images of 32×32 pixels (50,000 for training, 10,000 for testing) in 10 categories such as airplane, automobile and bird. With three colour channels (RGB), each image becomes 32 × 32 × 3 = 3,072 input values.
• ImageNet, an image recognition challenge with 1.2 million training images in 1,000 categories and 50,000 validation images, about 482×415 pixels on average. It had two tasks: recognition (what is in the image) and localization (where in the image an object is). The error rate dropped sharply in 2012 with the first deep learning entry, and the best entries later made fewer mistakes than humans.

Types of learning problems

• Supervised learning: the algorithm gets example inputs together with their known outputs, and learns to map inputs to outputs. If the output is a category the task is classification (MNIST, CIFAR-10); if it is a continuous number it is regression.
• Unsupervised learning: no outputs are given, and the algorithm has to find patterns or structure in the inputs on its own. Grouping news articles about similar topics together, clustering, is an example (see L03).
• Reinforcement learning: the system learns by trial and error. It performs an action in its environment and gets a positive reward if it went well and a negative reward (punishment) if it didn’t. Over time it learns which actions work, much like small children do. An example is a robot learning to move around obstacles.

Training and validation

A machine learning algorithm is trained on a dataset of examples with known outputs, and the trained algorithm is called a model. How good the model is can be measured as its accuracy, the percentage of examples classified correctly: 143 correct out of 150 is 95.33%.

If you test on the same data you trained on, you only know how well the model maps examples it has already seen. What usually matters is how well it handles new, unknown examples: its generalization ability. Training too much on the data can make the accuracy on the training data keep going up while the accuracy on unknown examples goes down. That is overfitting, and the best point to stop training is where the error on unknown examples is lowest.

Two ways to measure generalization:

• Train-test split: train the model on a larger part of the data, typically 70–80%, and test it on the rest. Choose the model that performs best on the test data.
• Cross-validation: divide the data into a number of buckets (folds) of roughly equal size, commonly 3, 5 or 10. With 10 buckets, train on 9 and test on the remaining one; repeat until every bucket has been used for testing once, and calculate the average accuracy.

Cross-validation measures generalization better, but takes longer, since the model is trained once per bucket. Use it when it doesn’t take too long. A common combination is to use cross-validation on the training data while tuning the model, and keep a separate test set for the final validation.

Is the result good or bad?

Whether an accuracy is good depends on how many categories there are. 50–60% on a binary problem (two categories) is hardly better than chance, but the same accuracy can be good with 25 categories.

A useful baseline is the ZeroR classifier, which classifies every example as the most common category in the dataset. On the balanced Iris dataset ZeroR has an accuracy of 33.3%; a model that does no better than ZeroR hasn’t learned anything.

Types of errors

Accuracy only tells you how many errors the model makes, not what kind. For a problem like “is this a cat?”:

• True positive (TP): a cat classified as a cat.
• True negative (TN): a dog classified as not a cat.
• False positive (FP, type 1 error): a dog classified as a cat.
• False negative (FN, type 2 error): a cat classified as not a cat.

Which kind of error matters more depends on the task. An earthquake warning system that sounds the alarm when there is no earthquake spreads unnecessary fear, so there the lecture argues for minimizing type 1 errors.

Two metrics take the type of error into account, and both are calculated per category:

• Precision = TP / (TP + FP): of the examples classified as the category, how many really belong to it?
• Recall = TP / (TP + FN): of the examples that belong to the category, how many were classified correctly?

They are often combined into the F1-score = 2 / (1/precision + 1/recall).

The confusion matrix

A confusion matrix shows the correct and incorrect classifications for each category. Each row is the actual category and each column the predicted one: with categories A and B, the row for A might read 48 and 2, meaning 48 examples of A were classified as A and 2 as B. Correct classifications are on the diagonal, which should hold the highest values. A4 asks you to generate a confusion matrix.

In the lecture’s Iris example with 30 test examples, all 13 setosa and all 8 virginica are classified correctly, and 8 of the 9 versicolor; one versicolor is classified as virginica. That gives an accuracy of 29/30 = 96.67%, and versicolor a precision of 1.0 but a recall of 0.889.

Other things to consider

Accuracy is not the only thing that matters when choosing an algorithm:

• Training and classification time: some models are slow to train; fast ones may not be complex enough for the problem.
• Space consumption: some models have many parameters to store, which can matter in a mobile application.
• Explainability: can we understand what the model has learned? A bank that can’t explain why its system refused a loan has a problem.
• Online learning: can the model keep learning from new examples without being retrained from scratch on all data?