Summary - Naïve Bayes

Classification algorithms

There are many algorithms for classification (supervised learning): artificial neural networks, support vector machines, k-nearest neighbour, decision trees, and many more. The lecture focuses on Naïve Bayes, a basic but common algorithm that is very fast and often very effective, for example for text classification. You implement it in A4.

Bayes’ theorem

Bayes’ theorem describes the probability of an event based on prior knowledge of conditions that might be related to it:

P(A|B) = P(B|A) * P(A) / P(B)

P(A|B) reads “the probability that A occurs given B”.

Is a stiff neck a good sign of being a good FIFA player? Suppose 50% of good FIFA players have a stiff neck, P(stiff|good) = 0.5; one in 50,000 players is good, P(good) = 1/50000; and one in 20 players has a stiff neck, P(stiff) = 1/20. Then:

P(good|stiff) = 0.5 * (1/50000) / (1/20) = 0.0002

That is about one in 5,000 players. Even though half of all good players have a stiff neck, it is not a good indicator of being good at FIFA, because stiff necks are common and good players rare.

Naïve Bayes

Bayes’ theorem takes one attribute into account, but real problems usually have several: a stiff neck, a good gamepad, a large high-resolution TV, and so on. The Naïve Bayes classifier combines several attributes into a probability of belonging to a category.

It is called naïve because it assumes that the attributes are independent of each other: the probability of one attribute is unrelated to the others. That is false for most real tasks; “money” is a better spam indicator together with “casino” than with “programming”. As a result, the numbers Naïve Bayes calculates are not the actual probabilities that an example belongs to a category. What you can do is compare them between categories and choose the category with the highest value, and that works surprisingly well for many real-world problems.

Multinomial Naïve Bayes: a worked example

Multinomial Naïve Bayes works best when the inputs are categorical or text. The example dataset has nine players with two attributes, gamepad (great, average or junk) and stiff neck (yes or no), and the category player skill (good or bad). Four players are good and five bad.

1. Frequency table: count how often each attribute value occurs together with each category. An average gamepad occurs twice among the good players and three times among the bad.
2. Probabilities: divide by the number of examples in the category. P(average|good) = 2/4, P(average|bad) = 3/5, P(yes|good) = 3/4, P(yes|bad) = 1/5. The probability of each category is its share of all examples: P(good) = 4/9, P(bad) = 5/9.

That table is all the classifier needs. To classify a player with an average gamepad and a stiff neck, multiply the category probability by the probability of each attribute value in that category:

good: P(good) * P(average|good) * P(yes|good) = 4/9 * 2/4 * 3/4 = 0.1667
bad:  P(bad)  * P(average|bad)  * P(yes|bad)  = 5/9 * 3/5 * 1/5 = 0.0667

Good has the higher value, so the player is classified as good. A player with a great gamepad and no stiff neck gets 0.0278 for good and 0.0889 for bad and is classified as bad. In this dataset the gamepad has little influence on the result; the stiff neck decides.

Thresholds

Sometimes “don’t know” is a better answer than a wrong one; in spam filtering it is usually worse to put real email in the spam folder than to let some spam through. A threshold handles this: with a threshold of 3, the highest value must be at least three times the other one, otherwise the classifier answers “don’t know”.

The examples above used a threshold of 1, always choosing the highest category. With a threshold of 3, the first player (ratio 0.1667 / 0.0667 ≈ 2.5) becomes “don’t know”, while the second (ratio 0.0889 / 0.0278 ≈ 3.2) is still classified as bad.

Text classification

To classify texts, which differ in length, each text is first turned into a bag-of-words: the number of times each unique word occurs in it. Then the probability of each word in each category is calculated just like the attribute values above.

The example has two spam messages (“Buy cheap Rolex?”, “You want cheap Vicodin?”) and three non-spam messages (“Can you buy milk?”, “Want candy tonight?”, “Gym tonight?”). After removing special characters and lower-casing, the unique words are buy, cheap, rolex, you, want, vicodin, can, milk, candy, tonight and gym. For example, P(cheap|spam) = 2/2 and P(cheap|non-spam) = 0/3.

Classifying “buy cheap candy” as spam runs into a problem: candy never occurs in spam, so P(candy|spam) = 0, and multiplying by zero makes the whole result zero, whatever the other words say. Laplace correction avoids zero probabilities by adding a small constant. With the lecture’s correction of 1/3, “buy cheap candy” scores higher as spam than as non-spam and is classified as spam.

Is that right? The deciding word is cheap, which appears in both spam messages and in none of the non-spam ones. If cheap is not really a sign of spam, the model needs more training data where it occurs in normal messages. Text classification needs quite large amounts of data to be accurate.

The Laplace example and the usual formulation

The slides and the recording give the spam value as 2.9. With the numbers on the slide, 0.4 * 0.833 * 1.333 * 0.333, it is about 0.148, which is still higher than 0.089 for non-spam, so the message is still classified as spam. Note also that adding a constant to a probability can give values above 1, as with 2/2 + 1/3. The usual formulation of Laplace (additive) smoothing adds to the counts instead: (count + 1) / (number of examples in the category + number of possible values), which always gives proper probabilities.

Other variants

The approach so far is called multinomial Naïve Bayes; the other two main variants are Bernoulli and Gaussian. Bernoulli Naïve Bayes only looks at whether a word or attribute is present or not, instead of counting how often it occurs.

Gaussian Naïve Bayes assumes that the attributes are numeric and normally distributed, so it is used for numerical inputs, such as the Iris and Banknote datasets in A4. The lecture uses a reduced Iris dataset with two attributes (petal length and width) and two species (setosa and versicolor), six examples of each.

Training is fast: split the data by category and calculate, for each attribute in each category, the mean and the standard deviation:

s = sqrt(sum((x_i - mean)^2) / (N - 1))

For setosa the petal length has mean 1.45 and standard deviation 0.14; for versicolor, mean 4.40 and standard deviation 0.52.

Classification uses the Gaussian probability density function for each attribute value:

pdf(x, mean, std) = 1 / (sqrt(2 * PI) * std) * e^(-(x - mean)^2 / (2 * std^2))

Multiply the values for all attributes in a category, then normalize by dividing each category’s result by the sum over all categories, and choose the category with the highest value. A flower with petal length 1.6 and width 0.8 gets 0.102 for setosa and 0.898 for versicolor, so it is classified as versicolor. The normal distribution is available in many libraries and in Excel, so you may not need to implement the formula yourself.

Include the category probability

The Gaussian example doesn’t multiply by the probability of each category, P(category), as the multinomial example does. That works here only because both species have six examples, so the factor is the same for both. Include it when the categories are not equally common.

Log probabilities

The probability of each attribute value is usually small, and multiplying many small numbers can cause numerical underflow: the result becomes 0. The fix is to work with logarithms: take the natural logarithm of each probability (Math.log in many languages) and add them, since ln(x * y) = ln(x) + ln(y). Transforming the sum back with e^sum (Math.exp) gives the same result as multiplying, in the example 0.102 and 0.898 again, but without the risk of underflow in the multiplication.

Tip

To choose the most likely category you don’t need to transform back at all: the category with the highest sum of logarithms is also the category with the highest product.

Summary

Despite its simplicity, Naïve Bayes is often effective for text classification and for numerical datasets that are not too complex. It is very fast, which also makes it a good baseline to compare more complex algorithms with. On well-separated clusters it is almost as accurate as Random Forest; on data that is not separated well, such as the “moons” dataset, it reaches about 93% where Random Forest learns the data perfectly, because it only models each category’s mean and standard deviation.