11.5.3: Testing the Pruned Decision-Tree Model on the Holdout Sample

Testing the Pruned Decision-Tree Model on the Holdout Sample

Managers randomly select ten new borrower records listed in Exhibit 11-15. For each new borrower listed in column 1, column 2 lists the income, column 3 lists the credit score, and column 4 lists the actual outcome: default or repay. Of the 10 new observations, eight repaid and two defaulted. Based on the pruned decision tree model, the table shows the predicted probability of default for each borrower (column 5) and the likelihood value calculation (column 6). The overall likelihood value is 0.18972, very similar to the overall likelihood value in the validation sample. Paige is now more confident using the pruned decision tree to predict probability of defaults. If the holdout sample had given results very different from the validation sample, Sierra’s data science team would have needed to go back and rework the model.

p to the power of y text ? end text cross times text ? end text not stretchy left parenthesis 1 minus p not stretchy right parenthesis to the power of 1 minus y end exponent — $^{a} {(0.01)}^{0} \times {(1 - 0.01)}^{1 - 0} = 1 \times 0.99 = 0.99$ ; $^{b} {(0.99)}^{1} \times {(1 - 0.99)}^{1 - 1} = 0.99 \times 1 = 0.99$ ; $^{c} {(0.33)}^{0} \times {(1 - 0.33)}^{1 - 0} = 1 \times 0.67 = 0.67$ ; $^{d} 0.99 \times 0.99 \times 0.67 \times 0.99 \times 0.67 \times 0.67 \times 0.67 \times 0.99 \times 0.99 \times 0.99 = 0.18972$

Observation	Income	Credit Score	Actual Outcome `(y)`	Probability of Default (`p`)	Likelihood Value $p to the power of y text ? end text cross times text ? end text not stretchy left parenthesis 1 minus p not stretchy right parenthesis to the power of 1 minus y end exponent$
(1)	(2)	(3)	(4)	(5)	(6)
1	$87,000	650	0 (Repay)	0.01	0.99^a
2	$79,000	670	1 (Default)	0.99	0.99^b
3	$70,000	708	0 (Repay)	0.33	0.67^c
4	$51,000	695	1 (Default)	0.99	0.99^b
5	$64,000	700	0 (Repay)	0.33	0.67^c
6	$68,000	710	0 (Repay)	0.33	0.67^c
7	$59,000	690	0 (Repay)	0.33	0.67^c
8	$92,000	670	0 (Repay)	0.01	0.99^a
9	$52,000	778	0 (Repay)	0.01	0.99^a
10	$65,000	720	0 (Repay)	0.01	0.99^a
	Overall likelihood value		$L_{f} = {0.18972}^{d}$

Criteria for the Model Choice

Two important principles guide the choice of models:

FEEDBACK LOOPS ARE FUNDAMENTAL TO DATA SCIENCE. Data scientists use a training dataset to train models. They choose among competing models based on how well the models perform in making classifications or predictions on a separate randomly selected cross-validation set. Data scientists do not rely on statistical measures like $R^{2}$ , t-values, and F-values to choose among models. Instead, the feedback loop from the cross-validation set becomes the basis for choosing among competing models.
THE CHOICE OF MODEL IS BASED ON PERFORMANCE ON THE CROSS-VALIDATION DATA, NOT THE TRAINING DATA. In the Sierra Investments example, the fully grown tree does better than the pruned tree in identifying default loans and repay loans in the training data by fitting the model to the peculiarities and specifics of the training data. As a result, it overfits the model to the training data and so performs poorly on the validation data. The pruned tree predicts default probabilities more accurately compared to the fully grown tree in the validation sample. This raises the question of whether further pruning would yield more accurate predictions.

The Bias–Variance Tradeoff

What are the tradeoffs of further pruning? As we have already discussed, the benefit of pruning is that it avoids overfitting the model to noise, so a different training dataset would have yielded a fairly similar model. The risk of further pruning is that it might weaken the signals in the feature variables that help more accurately assess the probability of default. The model would be underutilizing information signals in the data. Pruning the tree too much would “underfit” the data, biasing the model from fully understanding features that help distinguish between default loans and repay loans.

In contrast, building out the tree allows features to be more fully used to separate default loans from repay loans. A fully grown tree captures more richness of the data and so more accurately reflects the underlying reality. But, as mentioned, it also risks overfitting the model to the noise and randomness in the training data. Slightly different datasets would lead to very different models. Building fully grown trees would increase the variance of the model, making it less reliable in predicting default probabilities in a new sample of loans. In general, the more complex the model, the lower the bias and the higher the variance. Inversely, the less complex the model, the higher the bias and the lower the variance.

Data scientists try to balance the bias–variance tradeoff by, for example, pruning trees to various depths. The greater the depth, the lower the bias and the higher the variance of the model. But at what point is the tradeoff optimal? There is nothing that Sierra Investments can do a priori to determine whether pruning the tree to a depth of 2 or a depth of 3 is better. It must simply try both models and see which model has a higher overall likelihood value. This is the model that optimizes the bias–variance tradeoff. In the context of decision trees, the depth to which to prune is called a hyperparameter. A hyperparameter is a parameter that cannot be learned by running the model. It must be chosen prior to doing the analysis.^*

Pruning Guidelines

In large datasets, it would be difficult to detect the anomalies that result from fully growing the tree and then pruning, as we did in our example. Data scientists use some rules of thumb to guide their pruning.

STOP GROWING THE TREE AFTER A CERTAIN DEPTH. In the example, the data scientist grows the tree to a maximum depth of 3. Therefore, no further cuts are made even though rectangle 9 (R9) is impure with two repay loans and one default loan.
STOP GROWING THE TREE IF A NODE CONTAINS FEWER THAN A CERTAIN NUMBER OF DATA POINTS. In the example, the data scientist might choose to stop growing the tree if it has fewer than four data points. If this rule were applied, the cut at a credit score of 768 of rectangle 3 (R3) into rectangles 7 and 8 (R7 and R8) would not be made. Why? Because there are only three data points for income levels less than $56,000. The decision tree would classify loans with a credit score greater than 675 and income less than $56,000 as an impure node—a default loan with probability 2/3.

Management Accounting Insights

Management accountants provide important insights into the models that data scientists evaluate. For example if the tree were pruned to a depth of 2, then it follows from Exhibit 11-10 that the rule for defaults would be

If credit score $less than 675$ and income $less than $ text 85,000 end text comma text then end text equals 1 thin space left parenthesis text default end text right parenthesis$

If the borrower’s features do not satisfy this condition, then classify as 0 (repay) because for a decision tree pruned to depth 2, the probability of repayment at all other nodes would be greater than 0.5.

Even if cross-validation using prediction accuracy or maximum likelihood was slightly better for the model pruned to a depth of 2 than for a model pruned to a depth of 3, the management accountant might suggest that the model provides only weak signals about the probability of default.

It signals default when both the credit score and income levels are below certain thresholds. Based on business knowledge, the management accountant might suggest that the model is largely being driven by the credit score since many borrowers have incomes below $85,000.
From a business standpoint, the management accountant might ask whether the model could be strengthened to identify defaults when income is low but credit score is high.

Balanced Versus Unbalanced Samples

Paige is curious about one more point. Although the number of default loans in the dataset is close to 20%, the training sample consists of an equal number of default and repay loans, while the validation and holdout samples have 20% default loans. She wonders whether the number of default and repay loans in the training data should also be representative of the population (20% default loans and 80% repay loans) or should the training data comprise 50% default loans and 50% repay loans. To her the tradeoffs seem clear: If the decision tree models are built using the population distribution of default loans and repay loans, the models would be driven by repay loans since they are greater in number. But, just as in the case of detecting credit card fraud, the main goal of the “ask” is to identify default loans. Choosing more default loans in the training data than appear in the population will emphasize default loans and give the model more default data to use to help identify the characteristics of default loans.

Try It! 11-3

Rafael Alvarez, the management accountant at Wyatt Manufacturing, obtains the validation sample for product SB171, shown below. The sample contains data on workers’ experience (measured in years), the level of machine automation (measured on an interval scale from 0 to 5), and the actual outcome $(defective = 1 and good = 0)$ .

Use the format of Exhibit 11-14 to calculate (a) the probability of defect for each observation using the pruned tree shown in Try It 11-2. (If the observation is predicted to be a defective unit at a pure (terminal) node, write the probability as 0.99; if the observation is predicted to be a good unit at a pure (terminal) node, write the probability as 0.01; if the observation is predicted to be in a mixed node, write the probability of a defective unit for that node as defined by the decision tree rules you wrote.) (b) Calculate the likelihood value for each observation in the cross validation set using the equation, $L = p^{y} \times {(1 - p)}^{1 - y}$ . Remember: $x^{1} = x$ and $x^{0} = 1$ . (c) Calculate the overall likelihood value for the cross validation set.

Observation #	1	2	3	4	5	6	7	8	9	10
Automation Level	3.5	1.5	2.25	2.1	2.6	1.7	2.3	3.4	3.2	2.3
Years’ Experience	9	15	10	11	12	19	8.5	11	10	11.5
Actual Outcome	0	1	0	1	0	0	0	0	0	0

Watch Try It Video

The downside is that the decision tree models are being built on data that are not representative of the population (20% default loans, not 50%). Of course, the cross validation and holdout samples are representative of the population (they each contain 20% of default loans) and so the models are being selected and tested using the proportion of default loans the model will be encountering in practice. There is no clear answer, but in many circumstances choosing a “balanced” sample of default and repay loans (50% of each) in the training sample often performs better than an “unbalanced” sample (20% default loans and 80% repay loans). The data scientist will often run different training models using both “balanced” and “unbalanced” data and see which approach has a higher overall likelihood value and gives better predictions in the cross validation and holdout samples.