Logistic Regression

Despite the name, logistic regression (Cox, 1958; roots in Verhulst's logistic curve, 1838) is the canonical classification algorithm — the default first model for any classification problem, a fixture of credit scoring and medicine, and the conceptual gateway to neural networks: a single neuron is a logistic regression.

From line to probability

Linear regression outputs any real number — useless as a probability. Logistic regression keeps the linear score

and squashes it through the sigmoid (logistic) function:

Predict class 1 when \(\hat{p} \geq\) threshold (0.5 by default → \(z \geq 0\)). The set \(w^\top x + b = 0\) is a hyperplane: logistic regression is a linear classifier — its decision boundary is a straight line/plane in feature space (curved boundaries require engineered features, e.g. polynomial ones, or other models).

Odds and interpretability

Inverting the sigmoid shows the linear score is the log-odds:

So each unit increase in \(x_j\) multiplies the odds \(\frac{p}{1-p}\) by \(e^{w_j}\). A coefficient of 0.7 on "number of overdue payments" means each one multiplies the odds of default by \(e^{0.7} \approx 2\) — the kind of statement regulators and doctors demand, and the reason logistic regression persists in high-stakes domains (Explainability).

The loss: cross-entropy

Squared error on probabilities creates a non-convex landscape. Instead, maximize the likelihood of the observed labels — equivalently minimize the log loss / binary cross-entropy:

Each term reads: the log of the probability the model assigned to what actually happened. As the right panel above shows, being confidently wrong (\(\hat{p} \to 0\) when \(y = 1\)) costs unboundedly much — the model is pushed toward calibrated honesty, not just correct labels.

Training by gradient descent

No closed form exists, but \(J\) is convex — one global minimum. The gradient is astonishingly clean:

— identical in form to linear regression's gradient, with \(\hat{p} = \sigma(Xw)\) replacing \(Xw\). The same gradient descent loop applies unchanged:

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

w = np.zeros(X.shape[1])
for _ in range(n_epochs):
    p = sigmoid(X @ w)
    w -= eta * X.T @ (p - y) / len(y)

Regularization

Everything from Ridge and Lasso transfers: add \(\alpha \lVert w \rVert_2^2\) (L2) or \(\alpha \lVert w \rVert_1\) (L1) to the loss. It is so essential — especially with many features, where unregularized weights can grow without bound on separable data — that scikit-learn regularizes by default, parameterized by \(C = 1/\alpha\):

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(C=1.0,            # SMALLER C = STRONGER regularization
                           penalty='l2',
                           class_weight='balanced',   # for imbalance
                           max_iter=1000)
model.fit(X_train_scaled, y_train)
model.predict_proba(X_test_scaled)[:, 1]     # probabilities for ROC/PR analysis

Tune \(C\) on a log grid by cross-validation; scale features first (regularized + gradient-based ⇒ doubly necessary).

Multi-class: softmax

For \(k\) classes, learn one weight vector per class and normalize scores with softmax:

Cross-entropy generalizes verbatim. LogisticRegression handles this automatically (multi_class='multinomial' is the modern default). This exact construction — linear scores + softmax + cross-entropy — is the output layer of essentially every neural classifier, including LLMs choosing their next token.

Practical profile

Class materials

Quiz