Refining the Decision Tree

Learning Objective 4 Describe how to refine a decision tree model to ensure the data represent the business context

• pruning the decision tree to prevent overfitting and improving predictions

Management accountants have a unique ability to contribute to the development of decision models because they have a deep understanding of the economics of the business. We look at two specific areas that management accountants address when refining a decision tree to better match the economics of the business:

• Addressing overfitting, when a model matches the specific details of a dataset too closely, limiting its predictive powers
• Pruning the decision tree to sharpen the model’s predictive powers

Developing effective decision models requires that management accountants and data scientists work together. It is this understanding that causes Paige to question the decision tree model.

Overfitting

Based on her understanding of the business, Paige concludes that the decision tree developed from the dataset of 24 borrowers accounted for 21 borrowers in a way that makes intuitive sense. However, the rules for the last two nodes, credit score 712 and credit score 704.5, are

If income $>\mathrm{\$}\text{56,000}$ and $712>$ credit score $>704.5\hspace{0.17em}\mathrm{then}=1\mathrm{\hspace{0.17em}}\left(\text{default}\right)$

If income $>\mathrm{\$}\text{56,000}$ and credit score $<704.5\hspace{0.17em}\mathrm{then}=0\mathrm{\hspace{0.17em}}\left(\text{repay}\right)$

Consider two borrowers with identical incomes of $65,000. Borrower 1 has a credit score of 700. Borrower 2 has a credit score of 710. These rules say that Borrower 1, with a credit score less than 704.5, lower than Borrower 2, will repay and that Borrower 2, with a credit score greater than 704.5 but less than 712 will default $(=1)$. This prediction contradicts the overall trend within the data—the signal—in which repayers have higher credit scores and incomes and defaulters have lower credit scores and incomes. This is an example of overfitting, which is a direct outcome of the flexibility and power of these models. Overfitting occurs when a model adheres too closely to the specific details of a dataset such that, in addition to signal, it captures noise from random chance, making it less effective at accurately classifying observations from a new dataset. Overfitting limits a model’s ability to predict future outcomes.

Overfitting is an important concept for management accountants to recognize when using data analytic techniques. The decision tree algorithm, like many data analytic methods, is extremely flexible. It does not a priori assume a model for predicting defaults. Recall that in Chapter 10 we assumed a linear relation, $y=a+b\mathrm{X\; \&nbsp;}$ between machine hours (X) and indirect manufacturing labor costs (y) and then proceeded to estimate the coefficients a and b. In contrast, the decision tree algorithm simply partitions the data to separate repay loans from default loans (by reducing Gini impurity) and creates rules to predict new loan applications as either repay or default loans. Given the management accountant’s training and skills, he or she will quickly realize that the decision tree method “fits” to observations even when:

1. Credit scores and income levels may have been recorded in error.
2. Other factors not considered by the model (for example, the amount of the loan; the model only considers income and credit score) may have affected whether a loan was repaid.
3. The borrower with the higher credit score defaulted because of some other chance or random event (for example, having to pay health care bills because of a freak accident to a family member).