Singular Value Decomposition (SVD)

Singular value decomposition (SVD) is a mathematical technique that is widely used in machine learning, both in practice and to understand the mathematical properties of some algorithms.
The SVD decomposes a matrix into orthogonal components and allows for PC analysis.

We will apply SVD to the mnist data set and compare the data obtained from the four first principal components. For easy visualization, we use a sample of 1,000 digits.
First we show this sample of 1,000 digits with 784 predictors. Since there are too many digits and predictors, the graphics serves for illustrative purpose only.

my_image <- function(x, zlim = range(x), ...){
  colors = rev(RColorBrewer::brewer.pal(9, "RdBu"))
  cols <- 1:ncol(x)
  rows <- 1:nrow(x)
  image(cols, rows, t(x[rev(rows),,drop=FALSE]), xaxt = "n", yaxt = "n",
        xlab="", ylab="",  col = colors, zlim = zlim, ...)
  axis(side = 1, cols, colnames(x), las = 2)
}

options(repr.plot.width=10, repr.plot.height=10)
index <- sample(c(1:60000), 1000)
my_image(mnist$train$image[index,])
title("Original Data")

Now we apply SVA over the entire data and reconstruct it with the first four principal components.

y <- as.matrix(mnist$train$image)
s <- svd(y)
names(s)

1. 'd'
2. 'u'
3. 'v'

Below we show the dataset obtained from the first matrix factorization, obtained from the first principal component:
$Y \approx d_{1,1}U_{1}V_{1}^{T}$

matrix <- (s$u[index,1, drop=FALSE]*s$d[1]) %*% t(s$v[,1, drop=FALSE])
my_image(matrix)
title("PC1")

We also show the residual, the difference between the approximation and original data.