Naive Bayes

Naive Bayes is classical ML at its purest: apply an 18th-century theorem (Bayes, 1763) with one bold simplifying assumption, and get a classifier that is fast, needs little data, and filtered your spam for two decades. It also ties Part IV back to text representation — bag-of-words is its natural habitat.

Bayes' theorem

For a class \(y\) and observed features \(x\):

\[ \underbrace{P(y \mid x)}_{\text{posterior}} = \frac{\overbrace{P(x \mid y)}^{\text{likelihood}}\;\overbrace{P(y)}^{\text{prior}}}{\underbrace{P(x)}_{\text{evidence}}} \]

Read as a learning rule: start from the prior (how common is each class?), weigh by the likelihood (how typical are these features for that class?), and obtain the posterior (how probable is the class, given what we observed?). Classify by the largest posterior — \(P(x)\) is the same for all classes and cancels:

\[ \hat{y} = \arg\max_y \; P(y)\, P(x \mid y) \]

The "naive" assumption

The likelihood \(P(x_1, \dots, x_d \mid y)\) is a joint distribution over all feature combinations — hopeless to estimate. Naive Bayes assumes features are conditionally independent given the class:

\[ P(x_1, \dots, x_d \mid y) \;\approx\; \prod_{j=1}^{d} P(x_j \mid y) \qquad\Longrightarrow\qquad \hat{y} = \arg\max_y \; P(y) \prod_{j=1}^{d} P(x_j \mid y) \]

Now each \(P(x_j \mid y)\) is a simple one-dimensional estimate: count frequencies (discrete features) or fit a Gaussian (continuous ones). Training = counting. One pass over the data.

The assumption is almost always false (in real spam, "free" and "offer" co-occur far more than independence predicts). Why does it still work? Classification needs only the ranking of posteriors, not their exact values. Naive Bayes usually gets \(\arg\max\) right even when the probabilities themselves are badly distorted — typically over-confident (pushed toward 0 or 1). Trust its labels, not its probabilities.

The classic: spam filtering

With bag-of-words features, Multinomial Naive Bayes models word counts per class. Estimate from training counts:

\[ P(\text{spam}) = \frac{\#\text{spam docs}}{\#\text{docs}}, \qquad P(w \mid \text{spam}) = \frac{\text{count}(w, \text{spam}) + 1}{\sum_{w'} \text{count}(w', \text{spam}) + |V|} \]

The \(+1\) is Laplace smoothing: without it, a single word never seen in spam training data gives \(P(w \mid \text{spam}) = 0\), and one zero annihilates the entire product — any e-mail containing that word could never be spam. Smoothing pretends every word was seen once.

In practice, multiply-many-small-numbers underflows, so implementations sum logs:

\[ \hat{y} = \arg\max_y \Big[ \log P(y) + \sum_j \log P(x_j \mid y) \Big] \]

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline

spam_filter = make_pipeline(CountVectorizer(), MultinomialNB(alpha=1.0))  # alpha = smoothing
spam_filter.fit(docs_train, y_train)
spam_filter.predict(["WIN a FREE prize now!!!"])

The Naive Bayes family

Variant	Feature type	\(P(x_j \mid y)\) model	Typical use
`MultinomialNB`	counts	multinomial over counts	text (bag-of-words / TF-IDF)
`BernoulliNB`	binary	Bernoulli (presence/absence)	short text, binary flags
`GaussianNB`	continuous	Gaussian per feature per class	numeric tabular data

For GaussianNB, each class simply stores a mean and variance per feature:

\[ P(x_j \mid y) = \frac{1}{\sqrt{2\pi\sigma_{jy}^2}} \exp\!\Big(-\frac{(x_j - \mu_{jy})^2}{2\sigma_{jy}^2}\Big) \]

Practical profile


Strengths	trains in one pass (fastest learner in the course); works with little data; handles very high dimensions (10⁵ word features); naturally multi-class; online-updatable (`partial_fit`); no scaling needed
Weaknesses	probability estimates poorly calibrated; correlated features double-count evidence; linear-ish decision boundaries; the independence fiction can genuinely hurt
Reach for it when	text baselines, many features + few samples, latency-critical or streaming prediction

The perfect baseline

Whatever fancy model you plan for a text problem, fit CountVectorizer + MultinomialNB first. It takes seconds, and anything that cannot beat it is not worth deploying.