2. Classification or Regression

Deadline and Delivery

Nov 13 (Friday) 2026

Commits until 23:59

The same team as the EDA

60% of the Project grade

docs/projects/classification/ or docs/projects/regression/

Project: Classification or Regression with neural networks

Second delivery of the project. The team trains a neural network on the same dataset as the EDA, on the task chosen in the proposal: classification or regression, only one. The EDA's pipeline is the starting point. The question of this delivery: does the network learn something simpler models do not? And how far can the number it reports be trusted?

Rules

Network required: an MLP (or deeper) in PyTorch, Keras/TensorFlow or NumPy written by the team. scikit-learn's MLPClassifier/MLPRegressor only counts as a baseline.
Same dataset, same task and same train/test split as the EDA, with the same seed. The validation set comes out of the training set.
The test set is used once, in step 5, after everything was decided on validation. If it helped choose anything, say so in the report: step 5 is then worth up to half.
Fixed seeds, figures numbered with a title, axes and a legend; every number asked for appears in the text.

Steps

0 · Starting point · 0.5
- The EDA pipeline imported from code/, fitted on the training set only, with the fixes the EDA's grading pointed out.
- The main metric and the success criterion written down before training: "MAE at least 15% lower than the best baseline's".
1 · Baselines · 1.0
- A trivial one (majority class or training mean, Dummy*) and a linear one (logistic or linear/Ridge), on the same pipeline, evaluated on validation.
- Optional: random forest or boosting. On tables they often tie with the network; if so, say it.
2 · The network · 2.0
- Layers, widths, activations and initialization justified; number of parameters compared with the number of training rows.
- Output and loss consistent with the task (below). Network and loop in files under code/.
3 · Training · 1.5
- Optimizer (SGD with momentum, Adam or AdamW), learning rate, batch and epochs, with the reason.
- Early stopping on validation, restoring the best weights.
- Curves of training and validation per epoch, read: does it converge? where does overfitting start?
4 · Regularization and hyperparameters · 2.0
- Two or more techniques (weight decay, dropout, BatchNorm/LayerNorm…) compared with the network without any.
- A search of 8 or more configurations (learning rate, width/depth, regularization) chosen on validation, in a sorted table.
5 · Test evaluation · 1.5
- The chosen configuration, retrained and evaluated once, with the task's metrics (below).
- One table with the network and the baselines on the same test set: was the criterion of step 0 met?
- 3 or more seeds (mean ± std): is the gap to the baseline larger than that variation?
6 · Error analysis · 1.0
- The error per subgroup (a category from the EDA, ranges of the target) and the worst examples, with a hypothesis for each pattern.
- Permutation importance: does it match what the EDA suggested?
7 · Conclusion · 0.5
- Was the network worth it? Limitations, who would be affected by an error and what you would do with more time.
- The results summary table filled in.

What changes with the task

ClassificationRegression

Item	What to do
Trivial baseline	`DummyClassifier(strategy="most_frequent")`
Output and loss	binary: 1 unit + `BCEWithLogitsLoss`; multiclass: 1 unit per class + `CrossEntropyLoss`. The network returns logits
Imbalance	`weight`/`pos_weight` in the loss, or resampling on the training set only
Test metrics	confusion matrix, precision, recall and F1 per class, macro F1, ROC-AUC; if imbalanced, the precision-recall curve and PR-AUC
Threshold	chosen on validation, justified by the cost of the errors

Item	What to do
Trivial baseline	`DummyRegressor(strategy="mean")` (or `"median"`)
Output and loss	1 linear unit + MSE, MAE or Huber, justified by the outliers the EDA found
Skewed target	training on `log1p(y)` is allowed; the metrics go back to the original scale
Test metrics	MAE, RMSE and R² (MAPE only if the target stays away from zero)
Figures	predicted vs. actual and residuals vs. predicted

MLP · Optimization · Regularization · Classification metrics · Regression metrics · Splitting

Skeleton of the training loop with early stopping (PyTorch)

best, wait = float("inf"), 0
for epoch in range(max_epochs):
    model.train()
    for xb, yb in train_loader:
        optimizer.zero_grad()
        loss = loss_fn(model(xb), yb)
        loss.backward()
        optimizer.step()

    model.eval()
    with torch.no_grad():
        val_loss = loss_fn(model(X_val), y_val).item()
    if val_loss < best:
        best, wait = val_loss, 0
        torch.save(model.state_dict(), "best.pt")   # keep the best weights
    else:
        wait += 1
        if wait == patience:
            break
model.load_state_dict(torch.load("best.pt"))

Results summary

The last section of the report, with the rows in order:

#	Summary	Value
1	Dataset, task, target and main metric
2	Train · validation · test (rows)
3	Best baseline and its test metric
4	Chosen network: layers × units, number of parameters, regularization
5	Optimizer, learning rate, batch and the epoch it stopped at
6	The network's main metric on the test set (mean ± std, 3 seeds)
7	Success criterion met?
8	Subgroup with the largest error and its value

Delivery

Report in docs/projects/classification/index.md or docs/projects/regression/index.md (by task, as in the template), in the same repository as the EDA, with the step headings in the order of this page, code in code/ (runs from a clean clone) and figures in figures/ — see Submission Format. The deadline is not extended · plagiarism zeroes the delivery · AI is allowed, but every member explains every number and every decision: oral exams may happen and the Project Assessment caps the grade.

---
project: classification          # or regression
dataset: https://...
team: [First Last, ...]
ai_use: "none"
---

Competition bonus (up to +1, grade capped at 10): if the dataset belongs to an open competition (Kaggle, DrivenData, Zindi), +0.5 for a valid submission of the network and +0.5 if it ranks in the top 50% of the leaderboard, with a link and a screenshot in the report.

Rubric

Rubric under review

The rubric of this delivery may still be revised. Any change will be announced in class and recorded on this page before the deadline.

Each step is scored full, half or zero: full when everything its card asks for is there and justified; half when the justification or the number that supports the analysis is missing; zero when it is missing or wrong.

Step	Pts	Full marks when
0. Starting point	0.5	EDA pipeline reused without leakage; metric and numeric criterion set before training.
1. Baselines	1.0	Trivial and linear on the same pipeline, evaluated on validation and interpreted.
2. The network	2.0	Architecture justified, the right output and loss for the task, code in `code/` that runs.
3. Training	1.5	Optimizer choices justified, early stopping on validation, curves read.
4. Regularization and hyperparameters	2.0	Two techniques compared, 8 or more configurations chosen on validation.
5. Test evaluation	1.5	Test set used once, the task's metrics, comparison with the baselines and 3 seeds.
6. Error analysis	1.0	Error per subgroup with hypotheses and permutation importance compared with the EDA.
7. Conclusion	0.5	Limitations and responsible use discussed, results summary complete.
	10