11.5.2: Using Maximum Likelihood Values to Choose Between Fully Grown and Pruned Decision Trees

Using Maximum Likelihood Values to Choose Between Fully Grown and Pruned Decision Trees

An alternative approach used by data scientists to compare the performance of the fully grown decision tree and the pruned decision tree is to calculate the maximum likelihood value of the predictions. Exhibit 11-14 describes the maximum likelihood computations for the fully grown decision tree and the pruned decision tree. In the description below, we focus on the pruned decision tree.

bold italic p to the power of bold italic y bold cross times bold left parenthesis bold 1 bold minus bold italic p bold right parenthesis to the power of bold 1 bold minus bold italic y end exponent — ^a ${(0.01)}^{1} \times {(1 - 0.01)}^{1 - 1} = 0.01 \times 1 = 0.01$ ; ^b ${(0.01)}^{0} \times {(1 - 0.01)}^{1 - 0} = 1 \times 0.99 = 0.99$ ; ^c ${(0.99)}^{1} \times {(1 - 0.99)}^{1 - 1} = 0.99 \times 1 = 0.99$ ; ^d $0.01 \times 0.01 \times 0.99 \times 0.99 \times 0.99 \times 0.99 \times 0.99 \times 0.99 \times 0.99 \times 0.99 = 0.000092$ ; ^e ${(0.33)}^{1} \times {(1 - 0.33)}^{1 - 1} = 0.33 \times 1 = 0.33$ ;^f ${(0.33)}^{0} \times {(1 - 0.33)}^{1 - 0} = 1 \times 0.67 = 0.67$ ; ^g ${(0.33)}^{0} \times {(1 - 0.33)}^{1 - 0} = 1 \times 0.67 = 0.67$ ; ^h $0.33 \times 0.67 \times 0.99 \times 0.67 \times 0.99 \times 0.99 \times 0.99 \times 0.99 \times 0.99 \times 0.99 = 0.13807$

				For Fully Grown Tree	For Pruned Tree
Observation	Income	Credit Score	Actual Outcome `(y)`	Probability of Default (`p`)	Likelihood Value $bold italic p to the power of bold italic y bold cross times bold left parenthesis bold 1 bold minus bold italic p bold right parenthesis to the power of bold 1 bold minus bold italic y end exponent$	Probability of Default (`p`)	Likelihood Value $bold italic p to the power of bold italic y bold cross times bold left parenthesis bold 1 bold minus bold italic p bold right parenthesis to the power of bold 1 bold minus bold italic y end exponent$
(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)
1	$60,000	690	1 (Default)	0.01	0.01^a	0.33	0.33^e
2	$67,000	710	0 (Repay)	0.99	0.01	0.33	0.67^f
3	$55,000	772	0 (Repay)	0.01	0.99^b	0.01	0.99
4	$61,000	702	0 (Repay)	0.01	0.99	0.33	0.67^g
5	$58,000	715	0 (Repay)	0.01	0.99	0.01	0.99
6	$54,000	725	1 (Default)	0.99	0.99^c	0.99	0.99
7	$87,000	665	0 (Repay)	0.01	0.99	0.01	0.99
8	$90,000	660	0 (Repay)	0.01	0.99	0.01	0.99
9	$59,000	718	0 (Repay)	0.01	0.99	0.01	0.99
10	$53,000	775	0 (Repay)	0.01	0.99	0.01	0.99
			$L_{f} = {0.000092}^{d}$		$L_{p} = {0.13807}^{h}$

The first four columns are the cross-validation data from Exhibit 11-13 with the loan coded as 1 if it is a default and 0 if it is a repay. Now focus on column 7, which shows the probabilities of default for each of the cross-validation loans taken from Exhibit 11-13, column 6:

The pruned decision tree predicts observations 1, 2, and 4 will repay with a probability of 2/3 or, alternatively stated, will default with probability $1 / 3 (1 - 2 / 3)$ . Exhibit 11-14, column 7 records the probability of default as 0.33 for observations 1, 2, and 4.
The pruned decision tree predicts observations 3, 5, 7, 8, 9, and 10 in Exhibit 11-13, column 6, will repay with probability 1 (or alternatively stated, default with probability 0). Exhibit 11-14, column 7 records the probability of default as 0.01.^*
Finally, the pruned tree predicts observation 6 in Exhibit 11-13, column 6 will default with probability 1.00 so Exhibit 11-14, column 7 records the probability of default as 0.99.^*

For each observation in the validation sample, Exhibit 11-14, column 8 calculates the likelihood value $L = p^{y} \times (1 - p)^{1 - y}$ , based on the pruned decision tree model, where p is the predicted probability of default and y equals 1 if the loan defaults and 0 otherwise. What does the likelihood value do? It ranges between 0 and 1 with a value near 0 when the predicted probability is very far from the actual value and a value close to 1 when the predicted probability is close to the actual value.

Consider a loan that defaults so $y equals 1$ . If the prediction model predicts a high probability of default, say $p equals 0.99$ , the likelihood value $L = {0.99}^{1} \times {(1 - 0.99)}^{ 1 - 1} = 0.99$ . If, however, the prediction model predicts a low probability of default, $p equals 0.01$ , the likelihood value $L = {0.01}^{1} \times {(1 - 0.01)}^{ 1 - 1} = 0.01$ . When $inline style y equals 1$ , the likelihood value is maximized when p is close to 1.

Now consider a loan that repays so $inline style y equals 0$ . If the prediction model predicts a low probability of default, say $p space equals space 0.01$ , the likelihood value $L = {0.01}^{0} \times {(1 - 0.01)}^{ 1 - 0} = 0.99$ . If, however, the prediction model predicts a high probability of default in this case, $p space equals space 0.99$ , the likelihood value $L = {0.99}^{0} \times {(1 - 0.99)}^{1 - 0} = 0.01$ . When $y space equals space 0$ , the likelihood value is maximized when p is close to 0.

Consider the likelihood calculation for observation 1 in column 8 for the pruned tree. The actual outcome is a default $left parenthesis y space equals space 1 right parenthesis$ while the predicted probability of default is 0.33. The likelihood value $L = {(0.33)}^{1} \times {(1 - 0.33)}^{ 1 - 1} = 0.33 \times 1 = 0.33$ , indicating that for this observation the model did not do a good job of assessing default. Exhibit 11-14, column 8 shows the likelihood value calculations for all 10 observations in the cross-validation sample. The likelihood values range from a low of 0.33 to a high of 0.99 for these observations.

How well does the model do in predicting defaults across all observations taken together? To measure this performance, data scientists multiply the likelihood values calculated for the individual observations.^* Exhibit 11-14, column 8 shows the overall likelihood value for the pruned decision tree model, $begin inline style L subscript p equals 0.13807. end style$ Exhibit 11-14, column 7 shows the overall likelihood value for the full decision tree model, $begin inline style L subscript f equals 0.000092 end style$ . In this example, the overall likelihood is maximized for the pruned decision tree model because it has a higher overall likelihood value than the fully grown decision tree model. This means that, when measured across all observations, the predictions of the pruned decision tree model are closer to actual values. Sierra Investments prefers the pruned decision tree model to the fully grown decision tree model for predicting defaults. In other contexts, the fully grown decision tree model might have a higher likelihood value than the pruned decision tree model.

Paige still has some doubts about this approach. Choosing a model based on feedback from the validation sample means that the data scientist is, in effect, using the validation sample to make decisions about which model performs better. The real test is how well the pruned decision tree model performs against data it has not seen. Paige proposes testing the pruned decision tree model on a completely new dataset called a holdout sample (sometimes also called a test sample). She would be more confident of the model’s ability to predict defaults if the overall likelihood value is similar to the overall likelihood value in the cross-validation set.