Topic 7b - Unsupervised Learning and Pattern Discovery

Clustering

1. K-means works in an iterative cycle: (1) Initialize — place K centroids randomly in the data space. (2) Assign — assign each data point to the nearest centroid; that centroid's cluster now "owns" that point. (3) Update — recalculate each centroid as the mean (average position) of all points currently assigned to it; centroids move. (4) Repeat — reassign all points to the (now-moved) nearest centroid. The algorithm stops when no point changes its cluster assignment between iterations — the centroids have stabilized. A centroid is the geometric center of a cluster — the average of all data points currently assigned to it, not necessarily an actual data point itself.
2. The algorithm contributed the numerical groupings — it identified which students were mathematically similar to each other across whatever attributes were in the dataset, without any human guiding what "similar" should mean. The human contributed the labels and their interpretation — K-means outputs group memberships, not names or meanings. The counselor looked at the characteristics of students in each cluster and decided what to call them. No, the algorithm could not produce labels on its own: unsupervised learning discovers structure but has no concept of what that structure means. Meaning requires human judgment and domain knowledge.
3. Requiring K in advance is a limitation because the "right" number of clusters in real data is often unknown before analysis. If K is too small: genuinely distinct groups are merged together, obscuring real structure and producing misleading centroids that represent no actual student profile well. If K is too large: the algorithm artificially splits natural groups into sub-clusters that don't correspond to meaningful distinctions, producing clusters that are hard to interpret and may reflect noise rather than real patterns.

Dimensionality Reduction

4. PCA preserves the directions of greatest variance in the data — the combinations of attributes that explain the most variation between students. The 3-dimensional representation captures the major axes of difference across the student population. What is lost: the fine-grained details captured by the other 37 dimensions — subtle distinctions that explain less of the overall variation. The 3D version may be more useful for some purposes because: it is small enough to visualize, analyze, and apply clustering or other techniques to efficiently; it filters out noise (minor variation that may be measurement error rather than signal); and it reveals the dominant structure that would be buried in 40-dimensional complexity.
5. Humans can only perceive and visualize 2 or 3 spatial dimensions simultaneously. There is no physical or cognitive way to directly perceive a 40-dimensional space. Dimensionality reduction mathematically compresses the data into 2 or 3 dimensions that preserve as much structure as possible, making it viewable as a scatter plot. The risk: any 2D projection of 40-dimensional data necessarily discards information. Clusters that appear separate in 2D might actually overlap in the other 38 dimensions; clusters that look merged might actually be distinct. Conclusions drawn from the 2D view may not accurately reflect the full-dimensional structure of the data.

Autoencoders and Pretraining

6. The autoencoder is learning something meaningful, despite having no external labels: it is learning a compact internal representation of its inputs. To successfully compress an input into a much smaller bottleneck layer and then reconstruct it accurately, the network must learn which features of the input are essential and which are redundant. The bottleneck forces it to discover the underlying structure of the data — the compact representation captures what matters most. This is precisely what makes autoencoders useful for dimensionality reduction and for initializing other models.
7. To predict the next word in a sentence, the model must implicitly represent an enormous amount of world knowledge. Predicting that "The patient took the medication prescribed by the ___" ends with "doctor" (or similar) requires understanding medical contexts, professional roles, and grammatical structure. Predicting "The Eiffel Tower is located in ___" requires geographic knowledge. The task forces the model to build internal representations that encode facts about the world, relationships between concepts, grammatical structure, and contextual coherence — all inferred purely from patterns in text, without any explicit labels for any of these things.
8. Labeled data requires human effort to create — experts must examine each example and assign a correct label. For many problems, labeling is extremely expensive or requires rare expertise. This limits supervised learning to problems where labeled datasets can realistically be assembled. Unlabeled data, by contrast, is abundant: the entire internet, all books ever written, all scientific papers, all news articles — none of it requires labeling to use as training data. Unsupervised pretraining makes it possible to train models at a scale that would be completely impossible with supervised learning, enabling capabilities that no labeled dataset could ever provide.