Feature Types

Feature Types

A network does not see a column. It sees numbers, and numbers come with two properties whether you meant them or not: an order and a distance. 7 is bigger than 3, and it is four away from 3 and one away from 6. The network has no way to be told otherwise — those facts are built into arithmetic, and arithmetic is all it does.

So this chapter is not really about taxonomy. Naming a column "nominal" changes nothing. What changes things is the encoding, and choosing one is making a claim:

The taxonomy below is just a way of working out which assertions are safe for a given column.

The map

graph TD
    D[a column] --> Q1{"is it a<br/>quantity?"}
    Q1 -->|yes| N["<b>numerical</b><br/><small>order ✓ distance ✓</small>"]
    Q1 -->|no| Q2{"do the values<br/>have an order?"}
    Q2 -->|yes| O["<b>ordinal</b><br/><small>order ✓ distance ✗</small>"]
    Q2 -->|no| Q3{"does it<br/>wrap around?"}
    Q3 -->|yes| C["<b>cyclical</b><br/><small>distance ✓ but circular</small>"]
    Q3 -->|no| M["<b>nominal</b><br/><small>order ✗ distance ✗</small>"]
    D --> Q4{"is it an<br/>identifier?"}
    Q4 -->|yes| I["<b>id</b><br/><small>not a feature</small>"]
    D --> U["<b>unstructured</b><br/><small>text · image · audio · graph</small>"]

    classDef ok   fill:#e6f4ea,stroke:#3fb950,color:#14532d
    classDef warn fill:#fff4e5,stroke:#f0883e,color:#7c2d12
    classDef bad  fill:#fdeaea,stroke:#ff7b72,color:#7f1d1d
    classDef q    fill:#eef2f7,stroke:#8b949e,color:#1f2937
    class N,O,C ok
    class M warn
    class I bad
    class D,Q1,Q2,Q3,Q4,U q

Seeing the claim

Pick an encoding for hour_of_day and look at what it asserts. The left panel is where the 24 hours land in the encoded space; the right is the distance the encoding claims between every pair, with the cell for 23:00 against midnight boxed in white.

The number at the bottom is the rank agreement between the distances the encoding claims and the real circular ones. An integer scores about 0.72 — it gets most pairs roughly right and is catastrophically wrong about the one that wraps. One-hot scores 0.00, because it claims every pair is equidistant. sin/cos scores 0.997.

1. Numerical

The easy case: the order is real and the distances are real. The only thing to repair is scale, because a column measured in the hundreds of thousands and a column measured in tens do not contribute equally to a gradient, no matter which one matters more to the problem.

from sklearn.preprocessing import StandardScaler
# inside a Pipeline — see the chapter overview for why

Two sub-cases worth separating:

• Counts (n_orders, n_visits) are numerical, but almost always right-skewed: most rows near zero, a long tail to the right. log1p before scaling usually helps, and the distributions page covers when it does not.
• Bounded quantities (percentages, probabilities, pixel intensities) already have a natural range, so min-max to \([0,1]\) is often more faithful than standardizing.

The full treatment of which scaler to choose is in preprocessing.

2. Ordinal

The order is real: small < medium < large. Integer encoding preserves it, and that is exactly right.

from sklearn.preprocessing import OrdinalEncoder
enc = OrdinalEncoder(categories=[['small', 'medium', 'large']])   # spell out the order

Always pass the categories explicitly. OrdinalEncoder with no categories sorts alphabetically, which turns {small, medium, large} into large < medium < small.

3. Cyclical

The trap the panel above is about. Hour, day of week, month, wind direction, phase angle: values that wrap around, where the largest and the smallest are neighbours.

import numpy as np
df['hour_sin'] = np.sin(2 * np.pi * df['hour'] / 24)
df['hour_cos'] = np.cos(2 * np.pi * df['hour'] / 24)

Two columns instead of one, and the period goes in the denominator: 24 for hours, 7 for weekdays, 12 for months, 360 for degrees. You need both sin and cos — either alone is ambiguous, since \(\sin\) takes the same value at two different hours.

4. Nominal

No order at all. {Recife, Curitiba, Belém} has no "middle". The encoding must not invent one, and the cost of not inventing one is columns.

import pandas as pd
df = pd.get_dummies(df, columns=['city'], drop_first=False)

import torch.nn as nn
# 40 000 SKUs → a learned 32-dimensional vector each
emb = nn.Embedding(num_embeddings=40_000, embedding_dim=32)

from sklearn.preprocessing import TargetEncoder
# replaces each category by an average of y, smoothed toward the global mean

counts = X_train['city'].value_counts(normalize=True)
X['city_freq'] = X['city'].map(counts).fillna(0)

df['city'] = LabelEncoder().fit_transform(df['city'])   # ❌

5. Identifiers are not features

user_id, transaction_id, row_number, patient_number. They are keys, not measurements — and the reason to be strict about it is that they work. Ids are assigned in blocks: by registration date, by source system, by hospital. So the id correlates with the target, the model finds it, the score improves, and nothing has been learned.

This is one of the documented leakage cases in the leakage chapter: in KDD Cup 2008, the patient id was among the most predictive features available.

Identifiers do have one proper use: defining the groups the split must respect, so that all rows of one patient land on the same side.

6. Unstructured

The encoding question does not go away here — it moves inside the model. A tokenizer is a claim about which character sequences are units of meaning; a spectrogram is a claim that frequency content matters more than phase. The difference is that these encodings are mostly standard and mostly learned, rather than chosen per column.

Practice

Lab: what a wrong encoding costs

Step 1 — inventing an order that is not there

Each neighbourhood has its own price level, and the levels are in no particular order. Label encoding hands the network one number and asks it to learn a function that jumps arbitrarily at every integer; one-hot hands it one switch per category. Scored by \(R^2\) under 5-fold cross-validation.

"""An integer code is an order, and the network believes it.

Each neighbourhood has its own price level, and the levels are in no particular
order — neighbourhood 7 is not between neighbourhood 6 and neighbourhood 8 in
any sense at all. Label encoding hands the network a single number and asks it
to learn a function that jumps arbitrarily at every integer. One-hot hands it
one switch per category and asks for nothing.

Watch what happens as the number of categories grows: the harder the invented
ordering is to unlearn, the further the label-encoded model falls behind.

Printed as a markdown table, in identifiers only, so one artifact serves both
the English and the Portuguese page.
"""

import numpy as np
from sklearn.model_selection import KFold, cross_val_score
from sklearn.neural_network import MLPRegressor
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

N = 1500

model = lambda: make_pipeline(
    StandardScaler(), MLPRegressor(hidden_layer_sizes=(32,), max_iter=4000, random_state=0)
)
cv = KFold(5, shuffle=True, random_state=0)

print("| `categories` | `label_int` | `one_hot` |")
print("|---:|---:|---:|")
for k in (10, 40, 100):
    rng = np.random.default_rng(9)
    level = rng.normal(size=k) * 2.0            # each category sits where it sits
    category = rng.integers(0, k, N)
    price = level[category] + 0.4 * rng.normal(size=N)

    as_integer = category.reshape(-1, 1).astype(float)   # 0, 1, 2, … — an invented order
    as_one_hot = np.eye(k)[category]                     # one switch per category

    r2 = lambda X: cross_val_score(model(), X, price, cv=cv, scoring="r2").mean()
    print(f"| {k} | **{r2(as_integer):.3f}** | **{r2(as_one_hot):.3f}** |")

With 10 categories the network mostly copes: 0.795 against one-hot's 0.940. It has enough capacity to learn ten jumps in a single input. At 40 categories it collapses to 0.163, and at 100 to 0.077 — while one-hot sits at 0.96 throughout, because for one-hot nothing got harder: each category still has its own switch.

That is the shape of the damage. Label encoding does not fail loudly; it fails as a slow leak that gets worse the more categories you have, and the only symptom is a model that underperforms for no visible reason.

Step 2 — the hour that wraps

Demand follows the hour of the day, and the busy period runs across midnight. The model is asked about two hours it never saw in training: first the pair straddling the wrap, then a pair in the middle of the day. Scored by mean absolute error, against the noise floor of 0.279 that no model can beat.

"""Midnight is next to 23:00, and only one of these encodings knows that.

Demand follows the hour of the day, and the busy period runs across midnight.
The model is then asked about two hours it never saw in training — first the
pair that straddles the wrap, then a pair in the middle of the day — and scored
by mean absolute error against the noise floor.

Three claims about geometry, three results:
  integer  23 and 0 are as far apart as two hours can be
  sin_cos  the hours lie on a circle, so 23 and 0 are neighbours
  one_hot  no hour is near any other, and an unseen hour is a column of zeros

Printed as a markdown table, in identifiers only, so one artifact serves both
the English and the Portuguese page.
"""

import numpy as np
from sklearn.metrics import mean_absolute_error
from sklearn.neural_network import MLPRegressor
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

N, NOISE = 4000, 0.35

rng = np.random.default_rng(5)
hour = rng.integers(0, 24, N)
angle = 2 * np.pi * hour / 24
demand = 3.0 * np.cos(angle) + 1.0 * np.cos(2 * angle) + NOISE * rng.normal(size=N)

encodings = {
    "integer": hour.reshape(-1, 1).astype(float),
    "sin_cos": np.c_[np.sin(angle), np.cos(angle)],
    "one_hot": np.eye(24)[hour],
}
holdouts = {"unseen_23_00": [23, 0], "unseen_11_12": [11, 12]}


def error(X, held):
    test = np.isin(hour, held)
    fitted = make_pipeline(
        StandardScaler(), MLPRegressor(hidden_layer_sizes=(16,), max_iter=5000, random_state=0)
    ).fit(X[~test], demand[~test])
    return mean_absolute_error(demand[test], fitted.predict(X[test]))


print(f"| `encoding` | `{'` | `'.join(holdouts)}` |")
print("|---|---:|---:|")
for name, X in encodings.items():
    scores = " | ".join(f"**{error(X, held):.3f}**" for held in holdouts.values())
    print(f"| `{name}` | {scores} |")
floor = NOISE * np.sqrt(2 / np.pi)                 # E|noise|, the best any model can do
print(f"| `noise_floor` | {floor:.3f} | {floor:.3f} |")

Three encodings, three different failures:

• sin_cos lands at 0.356 and 0.303, close to the 0.279 floor. The hours it never saw are surrounded by hours it did, so it interpolates along the circle.
• integer is worse everywhere, and worst at the wrap: 0.671 against 0.450 in the middle of the day. Midnight has neighbours on only one side, because the encoding put the other side 23 units away.
• one_hot is off by an order of magnitude — 2.945 and 3.325 — and this is not a subtlety. An unseen category is a column that is zero in every training row. The network never saw that input fire, so it has nothing to say. One-hot gives up the ordering, and with it any ability to generalize to a value it has not met.