library(tidyverse)
library(caret)
library(matrixStats)
library(Rborist)
library(dslabs)

mnist <- read_mnist()

Principal Component Analysis (PCA)

In previous section we applied several machine learning techniques to the entire mnist dataset. We had a preprocessing step, where we filtered the most relevant pixels. However many predictors were still redundant and it is possible to improve the models.

We will apply SVD and PCA analysis on the mnist digit dataset. The original data has 784 pixels, used as predictors, and this is a good example for dimension reduction.
Because the pixels are so small, pixels close to each other on the grid tend to be correlated, allowing dimension reduction.

pca <- prcomp(mnist$train$images)
names(pca)

1. 'sdev'
2. 'rotation'
3. 'center'
4. 'scale'
5. 'x'

The following graphic shows the variability of each of the 784 principal components.

options(repr.plot.width=12, repr.plot.height=7)
pc <- 1:ncol(mnist$test$images)
plot(pc, pca$sdev)

We can see that the first few PCs already explain a large percent of the variability.

summary(pca)$importance[,1:10]

A matrix: 3 × 10 of type dbl

	PC1	PC2	PC3	PC4	PC5	PC6	PC7	PC8	PC9	PC10
Standard deviation	576.82291	493.23822	459.89930	429.85624	408.56680	384.50613	334.93015	314.44305	307.72756	284.27069
Proportion of Variance	0.09705	0.07096	0.06169	0.05389	0.04869	0.04312	0.03272	0.02884	0.02762	0.02357
Cumulative Proportion	0.09705	0.16801	0.22970	0.28359	0.33228	0.37540	0.40812	0.43696	0.46458	0.48815

The importance of the first PCs is more evident by looking the cumulative sum of variabilities.

cumulative_var <- summary(pca)$importance[3,]
plot(pc, cumulative_var)

With over 40 PCs we already are able to explain about 80% of the variability of the entire data.
We take a random sample of 2,000 digits and look how they are scattered between the first two principal components. It is possible to see that some classes have their points gathered in groups.