Quadratic Discriminant Analysis (QDA)

In Generative Models we consider the predictors distribution and how they affect the outcome of the conditional probability.
QDA technique is a version of Naive Bayes, in which we assume that the conditional probabilities of the predictors are multivariate normal.
In this analysis, where we use 2 predictors for 2 outcomes, the assumption is that their conditional probability is bivariate normal.
The average and standard deviations for the 2 predictors are shown below, as well as the correlation.

params <- mnist_27$train %>%
  group_by(y) %>%
  summarize(avg_1 = mean(x_1), avg_2 = mean(x_2),
            sd_1 = sd(x_1), sd_2 = sd(x_2),
            r = cor(x_1, x_2))
params

A tibble: 2 × 6

y	avg_1	avg_2	sd_1	sd_2	r
<fct>	<dbl>	<dbl>	<dbl>	<dbl>	<dbl>
2	0.1287798	0.2831542	0.07017481	0.05779893	0.4007501
7	0.2341133	0.2881862	0.07188006	0.10479565	0.4551607

The following graphic shows the two estimated normal densities. It is possible to see that the normal density is a good approximation for the label 2, however the label 7 does not really follow a normal density.

mnist_27$train %>% mutate(y = factor(y)) %>%
  ggplot(aes(x_1, x_2, fill = y, color = y)) +
  geom_point(show.legend = FALSE) +
  stat_ellipse(type="norm") +
  facet_wrap(~y)

Below are the model, confusion matrix and conditional probability of this technique.

train_qda <- train(y ~., method = "qda", data = mnist_27$train)
train_qda$finalModel

Call:
qda(x, grouping = y)

Prior probabilities of groups:
      2       7 
0.47375 0.52625 

Group means:
        x_1       x_2
2 0.1287798 0.2831542
7 0.2341133 0.2881862

y_hat_qda <- predict(train_qda, mnist_27$test)
confusionMatrix(data = y_hat_qda, reference = mnist_27$test$y)

Confusion Matrix and Statistics

          Reference
Prediction  2  7
         2 87 17
         7 19 77
                                          
               Accuracy : 0.82            
                 95% CI : (0.7596, 0.8706)
    No Information Rate : 0.53            
    P-Value [Acc > NIR] : <2e-16          
                                          
                  Kappa : 0.6391          
                                          
 Mcnemar's Test P-Value : 0.8676          
                                          
            Sensitivity : 0.8208          
            Specificity : 0.8191          
         Pos Pred Value : 0.8365          
         Neg Pred Value : 0.8021          
             Prevalence : 0.5300          
         Detection Rate : 0.4350          
   Detection Prevalence : 0.5200          
      Balanced Accuracy : 0.8200          
                                          
       'Positive' Class : 2               
                                          

confusionMatrix(data = y_hat_qda, reference = mnist_27$test$y)$overall["Accuracy"]
cat("F1 Score:", F_meas(data = y_hat_qda, reference = mnist_27$test$y))

Accuracy: 0.82

F1 Score: 0.8285714