Validating and Choosing Models

Learning Objective 5 Explain how to validate the predictions of full versus refined decision trees

• cross-validation using prediction accuracy and maximum likelihood and testing on holdout samples to balance the bias–variance tradeoff

Now that you understand full versus pruned decision trees and the related tradeoffs, we introduce three techniques: cross-validation using prediction accuracy, cross-validation using maximum likelihood, and testing on holdout samples that management accountants and data scientists use to choose among models such as the full and pruned decision tree.

Cross-Validation Using Prediction Accuracy to Choose Between Full and Pruned Decision Trees

Cross-validation is the process of comparing predictions of different models on a new set of data for which the actual outcomes (default or repay) are already known. Managers choose the model that predicts most accurately.

To determine whether the pruned tree performs better than the full tree, data scientists at Sierra test both models on 10 randomly selected new borrower records listed in Exhibit 11-13. For each new borrower listed in column 1, column 2 lists income, column 3 lists the credit score, and column 4 lists the actual outcome: default or repay. Of the 10 new observations, eight repaid and two defaulted; that is, the cross validation (also called validation) sample has the same 20% default rate as the population.

Exhibit 11-13 shows the predictions for each borrower made by the full tree (column 5) and the pruned tree (column 6).^*

Fully Grown Decision Tree

To compare how the fully grown tree and pruned tree classify each of the borrowers, consider ­record 2, with income of $67,000 and credit score of 710. Follow how the fully grown decision tree in Exhibit 11-11 classifies this record.

1. The cutoff at node 1 is credit score 675; with a credit score of 710, record 2 goes to the right.
2. The cutoff at node 2 is income of $56,000; with an income of $67,000, record 2 again goes to the right.
3. The cutoff at node 3 is credit score 712; with a credit score of 710, record 2 now goes left.
4. At the fourth and final node, which is at a depth of 4, the cutoff is credit score 704.5. With a credit score of 710, the tree sends record 2 to the right and classifies the record as default (column 5).

That default classification is wrong because the known actual outcome for record 2 in Exhibit 11-13 is repay.

Pruned Decision Tree

Now turn to the pruned tree in Exhibit 11-12 and follow how the pruned tree classifies record 2, noting that the path of the record for the first three nodes is the same as for the full tree:

1. The cutoff at node 1 is credit score 675; with a credit score of 710, record 2 goes to the right.
2. The cutoff at node 2 is income of $56,000; with an income of $67,000, record 2 again goes to the right.
3. The cutoff at node 3 is credit score 712; with a credit score of 710, record 2 goes to the left.
4. Record 2 now flows into a mixed pool that contains two repay loans and one default loan. The pruned tree cannot classify this record as either default or repay. It instead assigns a probability based on the contents of the pool. Only one of three observations in the pool is default, so there is a 1/3 probability that record 2 is default. In the simplest version of the model, any pool where the probability of default is less than 1/2 is classified as repay. Therefore, the pruned tree predicts record 2 as repay.

The actual outcome of record 2 in Exhibit 11-13 is also repay. The pruned tree correctly predicts the actual outcome.

Exhibit 11-13 shows the predictions of the fully grown decision tree and the pruned decision tree for the 10 new borrowers. The pruned tree more accurately classifies the validation set overall by correctly predicting 9 out of 10 loans (9/10), while the fully grown tree correctly predicts only 8 out of 10 loans (8/10). Data scientists at Sierra Investments recommend using the pruned tree model on new data because it performs better in predicting default and repay loans.

The data scientists also suggest a more sophisticated approach to comparing the full decision tree model to the pruned decision tree model.