Machine Learning · Insper · E3.2

NMF

Fundamental approach

Non-negative matrix factorization: a document as a sum of parts

The path of this part

  1. X ≈ WHtopics in H, mixtures in W, all ≥ 0
  2. Why non-negativewhole themes, not contrasts
  3. NMF on the postsfour forums, four topics
Each stage answers one question: can a matrix factorization find topics? → why must nothing be negative? → does it work on real posts?
Stage 1 · can a matrix factorization find topics?

NMF: \(X \approx WH\), with nothing negative

\[ X_{\,n \times V} \;\approx\; W_{\,n \times k}\; H_{\,k \times V}, \qquad W \ge 0,\; H \ge 0 \]

\(n\) documents, \(V\) words, \(k\) topics. \(X\): the TF-IDF matrix (term frequency × inverse document frequency).

rows of \(H\)

the topics: a weight for every word

rows of \(W\)

how much of each topic a document contains

Non-negative matrix factorization (Lee & Seung, 1999): the shape of PCA (principal component analysis), plus the constraint that every entry is \(\ge 0\). It minimizes \(\lVert X - WH \rVert^2\).

NMF on the 11 sentences

Predict. LDA failed on these sentences in 19 seeds out of 20. Will NMF find the three themes?

Topics: launch rocket satellite orbit · bread oven bake fresh · team game goal won

Clean split: every single-theme sentence in its theme.

init='nndsvda' starts from an SVD (singular value decomposition), not a random point: one reason the result is stable.

sentencespacecookingsports
0 (sports)0.000.000.71
3 (space)0.620.000.00
9 team watched the rocket launch…0.290.000.45
10 astronauts ate fresh bread…0.100.410.00
The rows of \(W\) are amounts, not probabilities: sentence 9 is mostly sports plus some space.
Stage 2 · why must nothing be negative?

Why non-negative? A comparison with SVD

Heat maps of the two 3 x 33 topic-word matrices: TruncatedSVD has 25 negative weights and contrasts themes; NMF has none and each component lights up one theme
TruncatedSVD (the PCA of sparse matrices): 25 negative weights; a component reads "sports words minus space words". NMF can only add parts, so each part is a whole theme.

Simulator: NMF parts versus SVD directions

Predict. From random weights, press step: will the NMF rows end up as whole themes or as contrasts like the SVD's? Will its error reach the SVD's?

Stage 3 · does it work on real posts?

NMF on the posts

Predict. Same 2,155 posts and vocabulary as LDA, now as TF-IDF, four topics. Will NMF blur the forums as LDA did?

graphics thanks files image program file know windows
god jesus people believe christ faith bible christians
game team hockey players play games season nhl
space nasa launch shuttle moon orbit lunar earth

LDA

0.456
ARI (adjusted Rand index) with the forums

NMF

0.767
the four topics are the four forums

Summary

  1. \(X \approx WH\) with \(W, H \ge 0\): rows of \(H\) are topics, rows of \(W\) the amount of each topic in a document.
  2. Nothing negative means parts that add up: whole themes, not "sports minus space".
  3. The weights are amounts, not probabilities: read them directly.
  4. Here NMF beat LDA (0.767 against 0.456); in the lab the order is reversed. No model wins everywhere.
Practice

To close the fundamental part

  • What is in a row of \(H\), and what is in a row of \(W\)?
  • Why can an SVD component not be read as a topic?
  • Why is NMF's error always a little above the SVD's?

In the lab: step 3 and the simulation more topics for NMF (overview page). Next: the fundamental slides of E3.3 BERTopic.