Techniques analysis

For each machine learning technique we calculate their confusion matrix, accuracy, F1 Score and show their conditional probability. This will help us summarize each technique.

Logistic regression

Logistic regression is an extension of linear regression, that assures that the estimate of conditional probability is between 0 and 1. This approach applies the logistic transformation:
$g(p) = log\frac{p}{1-p}$
The conditional probability is modeled by a line, which is computed by the maximum likelihood estimate (MLE).

train_glm <- train(y ~ ., method = "glm", data = mnist_27$train)
train_glm$finalModel

Call:  NULL

Coefficients:
(Intercept)          x_1          x_2  
     -2.164       24.169       -7.416  

Degrees of Freedom: 799 Total (i.e. Null);  797 Residual
Null Deviance:	    1107 
Residual Deviance: 717.7 	AIC: 723.7

y_hat_glm <- predict(train_glm, mnist_27$test, type = "raw")
confusionMatrix(data = y_hat_glm, reference = mnist_27$test$y)

# Or:
# fit_glm <- glm(y ~ x_1 + x_2, data=mnist_27$train, family = "binomial")
# p_hat_glm <- predict(fit_glm, mnist_27$test)
# y_hat_glm <- factor(ifelse(p_hat_glm > 0.5, 7, 2))

Confusion Matrix and Statistics

          Reference
Prediction  2  7
         2 82 26
         7 24 68
                                         
               Accuracy : 0.75           
                 95% CI : (0.684, 0.8084)
    No Information Rate : 0.53           
    P-Value [Acc > NIR] : 1.266e-10      
                                         
                  Kappa : 0.4976         
                                         
 Mcnemar's Test P-Value : 0.8875         
                                         
            Sensitivity : 0.7736         
            Specificity : 0.7234         
         Pos Pred Value : 0.7593         
         Neg Pred Value : 0.7391         
             Prevalence : 0.5300         
         Detection Rate : 0.4100         
   Detection Prevalence : 0.5400         
      Balanced Accuracy : 0.7485         
                                         
       'Positive' Class : 2              
                                         

confusionMatrix(data = y_hat_glm, reference = mnist_27$test$y)$overall["Accuracy"]
cat("F1 Score:", F_meas(data = y_hat_glm, reference = mnist_27$test$y))

Accuracy: 0.75

F1 Score: 0.7663551

y_hat_glm <- predict(train_glm, newdata = mnist_27$true_p, type="prob")[,2]

options(repr.plot.width=12, repr.plot.height=7)
grid.arrange(plot_cond_prob(),
             plot_cond_prob(y_hat_glm, title = "Logistic Regression"),
             ncol = 2)

To illustrate the accuracy achieved, we can see the boundary line of the logistic regression applied to the training and testing data:

mnist_27$true_p %>%
  mutate(p_hat = y_hat_glm) %>%
  ggplot() +
  stat_contour(aes(x_1, x_2, z=p_hat), breaks=c(0.5), color="black") +
  geom_point(mapping = aes(x_1, x_2, color=y), data = mnist_27$train)

mnist_27$true_p %>%
  mutate(p_hat = y_hat_glm) %>%
  ggplot() +
  stat_contour(aes(x_1, x_2, z=p_hat), breaks=c(0.5), color="black") +
  geom_point(mapping = aes(x_1, x_2, color=y), data = mnist_27$test)