y_hat_rf <- predict(train_rf, mnist_27$true_p, type = "prob")[,2]

grid.arrange(plot_cond_prob(),
             plot_cond_prob(y_hat_rf, title = "Random Forest"),
             ncol = 2)

Random Forest Rborist method

Rborist has less features, but is a faster method than random forest.

train_rborist <- train(y ~ ., method = "Rborist",
                       tuneGrid = data.frame(predFixed = 2, minNode = c(3, 50)),
                       data = mnist_27$train)
train_rborist$bestTune

A data.frame: 1 × 2

	predFixed	minNode
	<dbl>	<dbl>
2	2	50

y_hat_rborist <- predict(train_rborist, mnist_27$test)
confusionMatrix(data = y_hat_rborist, reference = mnist_27$test$y)

Confusion Matrix and Statistics

          Reference
Prediction  2  7
         2 86 20
         7 20 74
                                          
               Accuracy : 0.8             
                 95% CI : (0.7378, 0.8531)
    No Information Rate : 0.53            
    P-Value [Acc > NIR] : 1.662e-15       
                                          
                  Kappa : 0.5986          
                                          
 Mcnemar's Test P-Value : 1               
                                          
            Sensitivity : 0.8113          
            Specificity : 0.7872          
         Pos Pred Value : 0.8113          
         Neg Pred Value : 0.7872          
             Prevalence : 0.5300          
         Detection Rate : 0.4300          
   Detection Prevalence : 0.5300          
      Balanced Accuracy : 0.7993          
                                          
       'Positive' Class : 2               
                                          

confusionMatrix(data = y_hat_rborist, reference = mnist_27$test$y)$overall["Accuracy"]
cat("F1 Score:", F_meas(data = y_hat_rborist, reference = mnist_27$test$y))

Accuracy: 0.8

F1 Score: 0.8113208

y_hat_rborist <- predict(train_rborist, mnist_27$true_p, type = "prob")[,2]

grid.arrange(plot_cond_prob(),
             plot_cond_prob(y_hat_rborist, title = "Rborist"),
             ncol = 2)

Random Forests usually have same or better accuracy than single decision trees, since they have more flexibility in their conditional probability. However it depends on the problem.

Local Weighted Regression (loess), degree = 1 (linear):

The loess technique applies smoothing to the data, searching for the best estimate, in the presence of uncertainty. It applies the Tukey tri-weight density to assign weights to the kernel.
With degree 1, the technique fits the estimates as local lines.

grid <- expand.grid(span = seq(0.15, 0.65, len = 10), degree = 1)

train_loess <- train(y ~ ., method = "gamLoess",
                     tuneGrid = grid,
                     data = mnist_27$train)
train_loess$bestTune

Loading required package: gam

Loading required package: splines

Loading required package: foreach


Attaching package: ‘foreach’


The following objects are masked from ‘package:purrr’:

    accumulate, when


Loaded gam 1.20

A data.frame: 1 × 2

	span	degree
	<dbl>	<dbl>
6	0.4277778	1

y_hat_loess <- predict(train_loess, mnist_27$test, type = "raw")
confusionMatrix(data = y_hat_loess, reference = mnist_27$test$y)

Confusion Matrix and Statistics

          Reference
Prediction  2  7
         2 92 17
         7 14 77
                                          
               Accuracy : 0.845           
                 95% CI : (0.7873, 0.8922)
    No Information Rate : 0.53            
    P-Value [Acc > NIR] : <2e-16          
                                          
                  Kappa : 0.6883          
                                          
 Mcnemar's Test P-Value : 0.7194          
                                          
            Sensitivity : 0.8679          
            Specificity : 0.8191          
         Pos Pred Value : 0.8440          
         Neg Pred Value : 0.8462          
             Prevalence : 0.5300          
         Detection Rate : 0.4600          
   Detection Prevalence : 0.5450          
      Balanced Accuracy : 0.8435          
                                          
       'Positive' Class : 2               
                                          

confusionMatrix(data = y_hat_loess, reference = mnist_27$test$y)$overall["Accuracy"]
cat("F1 Score:", F_meas(data = y_hat_loess, reference = mnist_27$test$y))

Accuracy: 0.845

F1 Score: 0.855814

ggplot(train_loess, highlight = TRUE)

y_hat_loess <- predict(train_loess, mnist_27$true_p, type = "prob")[,2]

grid.arrange(plot_cond_prob(),
             plot_cond_prob(y_hat_loess, title = "Local Weighted Regression"),
             ncol = 2)

Ensembles

Ensembles combine the results of different algorithms, thus improving the final result of the predictions. We compute new class probabilities by taking the average of the class probabilities provided by different techniques.

We will test out 10 of the most common machine learning models, from which 6 we already discussed:

• Logistic regression (glm)
• Linear Discriminant Analysis (lda)
• Naive Bayes (naive_bayes)
• Support Vector Machines with Linear Kernel (svmLinear)
• k-Nearest Neighbors (knn)
• Local Weighted Regression (gamLoess)
• Penalized Multinomial Regression (multinom)
• Quadratic Discriminant Analysis (qda)
• Random Forest (rf)
• AdaBoost Classification Trees (adaboost)

models <- c("glm", "lda", "naive_bayes", "svmLinear", "knn", "gamLoess", "multinom", "qda", "rf", "adaboost")

For simplicity, we will use all default parameters to train their models.