1. Data

Activity: Data Preparation and Analysis for Neural Networks

This activity is designed to test your skills in generating and manipulating synthetic datasets, handling real-world data challenges, and preparing data to be fed into neural networks.

The thread running through the activity is the spread of the data: how much a point cloud spreads out, in which direction it spreads, and how that changes the difficulty of the classification problem.

Exercise 1

Point Clouds: Geometry and Spread in 2D

Understanding how data is distributed is the first step before designing a network architecture. In this exercise you will generate and measure two-dimensional point clouds, observing how the distribution affects the complexity of the decision boundaries a neural network would need to learn.

A — Generate the clouds

Create a synthetic dataset with a total of 400 samples, divided equally among 4 classes (100 samples each). Use a Gaussian distribution to generate the points for each class based on the following parameters:

• Class 0: Mean = \([2, 3]\), Standard Deviation = \([0.8, 2.5]\)
• Class 1: Mean = \([5, 6]\), Standard Deviation = \([1.2, 1.9]\)
• Class 2: Mean = \([8, 1]\), Standard Deviation = \([0.9, 0.9]\)
• Class 3: Mean = \([15, 4]\), Standard Deviation = \([0.5, 2.0]\)

Produce Figure 1: a 2D scatter plot with all the points, one color per class, with the center of each cloud (its mean) marked on the plot.

B — More or less spread out

The standard deviation controls how much each cloud spreads around its center. Generate the same 4 classes from item A, four times over, multiplying all standard deviations by a scale factor \(s\) — the means never change, only the spread:

So this is 4 datasets of 4 classes each — not one dataset of 4 scales with different classes.

1. Produce Figure 2: 4 subplots (one per value of \(s\)), sharing the same axis limits, so the comparison is honest.

2. Compute, for \(s = 1\) only, the separation ratio of each pair of classes \((i, j)\):

   \[ r_{ij} = \frac{\lVert \mu_i - \mu_j \rVert}{\bar{\sigma}_i + \bar{\sigma}_j}, \qquad \bar{\sigma}_k = \frac{\sigma_{k,x} + \sigma_{k,y}}{2} \]

   High values mean well separated clouds; low values, clouds that blend into each other. There are 6 pairs — report them in a table and say which is the smallest. Since the means do not change, \(r_{ij}\) scales with \(1/s\): state what that smallest \(r_{ij}\) becomes at \(s = 2\) without generating anything new. 1. Compute, for each \(s\), the mixing rate: the fraction of points whose nearest class center is not the one of their own class. With NumPy this is comparing each point against the 4 means — a purely geometric measure, with nothing to train. 1. Produce Figure 3: a plot of mixing rate \(\times\) \(s\). Answer: from which scale factor on can the clouds no longer be separated by straight lines? What happens to the smallest \(r_{ij}\) at that point?

C — Analysis

1. Describe the overlap of the four classes in the original dataset (\(s = 1\)). Could a single linear boundary separate all classes? What about a set of linear boundaries?
2. Sketch on Figure 1 the decision boundaries you think a trained neural network might learn.
3. Relate your sketch to item B: the more spread out the clouds are, what happens to the region where the network necessarily makes mistakes?

Exercise 2

Non-Linearity in Higher Dimensions

Simple neural networks (like a Perceptron) can only learn linear boundaries. Deep networks shine when the data is not linearly separable. This exercise contrasts two datasets of the same dimensionality to make that difference explicit.

A — Dataset I: shifted Gaussians

Generate 500 samples for Class A and 500 for Class B, using a multivariate normal distribution with the parameters below.

• Class A:

  Mean vector:

  \[\mu_A = [0, 0, 0, 0, 0]\]

  Covariance matrix:

  \[ \Sigma_A = \begin{pmatrix} 1.0 & 0.8 & 0.1 & 0.0 & 0.0 \\ 0.8 & 1.0 & 0.3 & 0.0 & 0.0 \\ 0.1 & 0.3 & 1.0 & 0.5 & 0.0 \\ 0.0 & 0.0 & 0.5 & 1.0 & 0.2 \\ 0.0 & 0.0 & 0.0 & 0.2 & 1.0 \end{pmatrix} \]

• Class B:

  Mean vector:

  \[\mu_B = [1.5, 1.5, 1.5, 1.5, 1.5]\]

  Covariance matrix:

  \[ \Sigma_B = \begin{pmatrix} 1.5 & -0.7 & 0.2 & 0.0 & 0.0 \\ -0.7 & 1.5 & 0.4 & 0.0 & 0.0 \\ 0.2 & 0.4 & 1.5 & 0.6 & 0.0 \\ 0.0 & 0.0 & 0.6 & 1.5 & 0.3 \\ 0.0 & 0.0 & 0.0 & 0.3 & 1.5 \end{pmatrix} \]

Note that the two classes have different spreads: \(\Sigma_B\) has larger variances and a negative correlation between the first two features, while \(\Sigma_A\) has a positive one.

B — Dataset II: concentric shells

Generate a second dataset, also with 500 samples per class and also in 5 dimensions, but with radial structure:

1. Draw directions uniformly on the unit sphere of \(\mathbb{R}^5\) — draw \(v \sim \mathcal{N}(0, I_5)\) and normalize, \(u = v / \lVert v \rVert\);
2. Class C (core): radius \(\rho \sim \mathcal{N}(2.0,\ 0.4)\);
3. Class D (shell): radius \(\rho \sim \mathcal{N}(5.0,\ 0.4)\);
4. Each point is \(x = \rho \cdot u\).

C — Visualize and compare

You cannot plot a 5D graph directly, so reduce the dimensionality.

1. Produce Figure 4: apply PCA to project each dataset into 2 dimensions and plot the two scatter plots side by side, colored by class;
2. Report the explained variance of the first two components in each case. In which dataset does the 2D projection better preserve the information relevant for classification?
3. For each dataset, compute in 5D and report:
   • the distance between the class centers, \(\lVert \mu_1 - \mu_2 \rVert\);
   • Figure 5: the histogram of the radius \(\lVert x \rVert\) of each point, with both classes overlaid on the same axis.

D — Analysis

1. In Dataset II the distance between the centers is close to zero, yet the radius histograms are well separated. What does that combination tell you about the possibility of separating the classes with a hyperplane?
2. Explain why the structure of Dataset II cannot be solved by a linear boundary, no matter how much data is collected.
3. PCA is a linear transformation. Discuss: does a 2D projection in which the classes look mixed prove that they are inseparable in the original space? Justify with your own results, and write a simple function of the inputs that separates Dataset II (hint: look at \(\lVert x \rVert^2 = \sum_i x_i^2\)).

Exercise 3

Preparing Real-World Data for a Neural Network

This exercise uses a real dataset from Kaggle. Your task is to perform the preprocessing required to make it suitable for a neural network that uses the hyperbolic tangent (tanh) activation function in its hidden layers.

A — Get to know the data

1. Download the Spaceship Titanic dataset (use train.csv, the only labeled file).
2. Describe the goal of the dataset: what does the Transported column represent? What is the class balance between the two labels?
3. List the features, separating numerical (e.g., Age, RoomService) from categorical (e.g., HomePlanet, Destination).
4. Build a table of missing values per column, in absolute count and in percentage.
5. For the spending columns (RoomService, FoodCourt, ShoppingMall, Spa, VRDeck), report mean, median, and maximum. Compare mean and median: what does that difference tell you about the spread and the skewness of those distributions?

B — Split before you transform

1. Split train and test 80/20, stratified by the target, with a fixed seed;
2. Explain, in two or three sentences, why this split comes before imputation and scaling.

C — Preprocess

The tanh activation outputs values in \([-1, 1]\), so the inputs must be on a compatible scale.

1. Missing data: define and implement a strategy per column type (numerical × categorical) and justify each choice. Fit the imputer on the training set and apply it to the test set;
2. Categorical features: convert HomePlanet, CryoSleep, Destination, and VIP to numerical format (one-hot encoding is a good choice). Explain how your code handles a category that appears in the test set but not in the training set;
3. Feature engineering: create the TotalSpend feature, the sum of the five spending columns (drop Cabin, Name, and PassengerId);
4. Heavy tails: apply \(\log(1 + x)\) to the spending columns and show the histogram of one of them before and after. Why does this transformation help a network with tanh?
5. Scaling: scale the numerical columns with Standardization (mean 0, std 1) or Normalization to \([-1, 1]\). Implement one of them, explain the choice, and report the resulting minimum and maximum values.

D — Verify and visualize

1. Figure 6: histogram of one heavy-tailed feature (FoodCourt, for instance) before and after preprocessing;
2. Final checks, explicitly reported: no remaining NaN; the final shape of the feature matrix; the value range is compatible with tanh;
3. In one paragraph: which of your preprocessing decisions do you think would most affect the network's training, and why?

Results summary

Close the report with this table filled in. It replaces no analysis — it is an index of the numbers you already computed, gathered in one place so grading can check each value without hunting for it.

Evaluation Criteria

The deliverable for this activity is a report that includes:

1. A brief description of your approach to each exercise.
2. The code used to generate the datasets, preprocess the data, and create the visualizations, with comments explaining each step.
3. The plots and visualizations requested in each exercise.
4. Your analysis and answers to the questions posed in each exercise.

Important Notes:

• The deliverable must be submitted in the specified format: GitHub Pages. No other format will be accepted. There is a course template you may use — template;

• There is a strict no-plagiarism policy. Any form of plagiarism will result in a zero grade for the activity;

• The deadline for each activity is not extended — NO EXCEPTIONS will be made for late submissions.

• AI collaboration is allowed, but each student MUST UNDERSTAND and be able to explain every part of the submitted code and analysis. Any use of AI tools must be properly cited. ORAL EXAMS may be conducted.

Grading Criteria:

Each row is worth the points indicated, awarded in full, partially (half), or not at all: in full when the item is complete and correct; partially when it is implemented but missing the requested analysis, or when the analysis lacks the numerical result that supports it; zero when absent or incorrect.

Exercise 1 — Point clouds (3 points):

Exercise 2 — Non-linearity in 5D (3 points):

Exercise 3 — Real-world data (4 points):

Deductions: