Data Algorithms and Models

Learning Objective 3 Explain the elements of a decision tree model

• using rules to segment the target variable into different classes

In the last several years, data scientists have developed myriad models to analyze vast quantities of data. These range from simple regression techniques to sophisticated neural networks. The hallmark of most of these models is that they are extremely flexible in fitting to the data and rely heavily on computational power. This flexibility, however, comes with its own problems that the management accountant’s skills can help address. In this section, we examine functional versus flexible relationships and introduce a specific model: the decision tree.

Step 4: Build a Model

In this step, the management accountant works with the data scientists to build models. They must decide what kinds of models to build and how to validate them.

Functional Relationship Model

Paige and the data science team recognize that the target (dependent variable) is binary: 1 if the loan defaults and 0 if the loan repays. As we saw in Chapter 10, the team could use logistic regression to estimate the relationship between independent feature variables (such as [annual] income, credit score, and amount of the loan) and the target variable. This approach assumes a particular functional form of the relationship between feature variables and the target variable to fit the curve shown in Exhibit 10-10. The data scientists on the team suggest that they could get better predictions of default if they fit more flexible models that do not assume a particular functional relationship between feature variables and the target variable. They propose to use a very popular data science technique called decision trees.

THE DECISION TREE: A PREDICTIVE MODELING TECHNIQUE The decision tree is a technique for segmenting the target variable into different regions based on a set of rules. These rules make the model easier to interpret than some other models. Industry analyst Dean Abbot writes, “All trees can be read as a series of ‘if-then-else’ rules that ultimately generate a predicted value . . . [They are] much more accessible to decision makers than mathematical formulas.” They are also easy to build and perform better with more data.

Paige is eager to learn why more flexible models yield better default predictions. She is also keen to understand the problems and challenges of the decision tree technique. This seems very difficult to do in the context of thousands of variables and hundreds of features. She asks the data science team whether it is possible to describe the approach in a simple visual way. She would then be able to use the output of the model to guide managers in their decision making.

To explain the technique, the team starts with a simple example that links directly to Sierra Investments’ objective. They select a dataset of 24 loans containing 12 default loans (labeled as 1) and 12 repay loans (labeled as 0). Two features, income and credit score, describe each data point. Exhibit 11-4 presents the default loans in columns 1 through 3 and the repay loans in columns 4 through 6.

Exhibit 11-5 plots the data in Exhibit 11-4. The x-axis represents the income level. The y-axis represents credit score. Each default loan from Exhibit 11-4, column 3 is represented as a red dot. Each repay loan from Exhibit 11-4, column 6 is represented as a blue dot. For example, the red dot at an income level of $50,000 and credit score of 530 represents the first default loan (from columns 1, 2, and 3 in Exhibit 11-4). In this initial plot the 12 repay loans and 12 default loans sit within one rectangle. If Paige selects one dot at random from this pool, it is equally likely to be a blue dot as a red dot.

THE DECISION TREE ALGORITHM An algorithm is a set of instructions. The decision tree emerges from an algorithmic process of subdividing the data along the two possible features, income or credit score. The decision tree algorithm instructs Paige to divide the data with a cut or line parallel to the x-axis (income) or y-axis (credit score) such that each of the two smaller rectangles contains a mix of dots that are purer than the original rectangle; that is, one rectangle contains more repay loans than default loans and the other rectangle contains more default loans than repay loans. For example, in the scatter plot on the left side of Exhibit 11-6, Paige places a cut at credit score of 675 (a horizontal line parallel to the x-axis).^* This creates rectangle R1 above the line and rectangle R2 below the line. All borrowers in R1 have a credit score greater than 675 and there are more repay loans (10 blue dots) than default loans (3 red dots). All borrowers in R2 have credit scores less than 675 and there are fewer repay loans (2 blue dots) than default loans (9 red dots). Therefore, the mix of dots in rectangles R1 and R2 are each purer than the mix in the original rectangle, which had an equal number of red and blue dots. However, the mix in each is still impure: Rectangle R1 contains mostly repay loans, but also a few default loans, and rectangle R2 contains mostly default loans, but also a few repay loans. The table in Exhibit 11-6 shows the tally of default and repay loans across each of the two new rectangles.

At the top right of Exhibit 11-6 is a visualization of a decision tree as a series of decision nodes and connecting lines. In this example, the feature credit score with a value of 675 is the first decision node. The 13 borrowers with credit scores higher than 675 go to the right, to R1, while the 11 borrowers with credit scores less than 675 go to the left, to R2.

The process of subdividing the data into smaller subsets that get purer with each subdivision is the essence of the decision tree. The algorithm uses a measure of impurity, the focus of the following section, to calculate precisely how pure the resulting subsets will be at any specific cut.

MEASURING IMPURITY Gini impurity is a way to measure the purity of a collection of observations in a rectangle (set). If a rectangle is very mixed it is “impure” and the Gini impurity is high. As a rectangle becomes more pure, that is, it contains more members of one class than another, the Gini impurity decreases. Gini impurity can be calculated for any number of classes; in the Sierra Investments example, there are two classes: repay and default. The reduction in the Gini impurity from the level in the original rectangle to the levels in the new rectangles created by the cut (in our example R1 and R2) is called information gain. Information gain and Gini impurity are inversely related.

To place the first cut, as it did at credit score 675, the algorithm tests all possible values for vertical and horizontal cuts. It calculates the Gini impurity measure for each pair of rectangles that result from that cut and then compares the Gini impurity of the rectangles after the cut is made to the impurity of the rectangle(s) before the cut is made. The algorithm selects the value that reduces Gini impurity the most.

The equation for calculating Gini impurity for any point on the credit score or income axis is

The process for calculating Gini impurity is shown below:

1. ESTABLISH THE BASELINE GINI IMPURITY. This calculation is based on the number of classes in the population. In this example there are two classes comprising 12 repay loans and 12 default loans. The initial calculation of the Gini impurity for the original rectangle is

The Gini impurity is $\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.$, which in fact is the maximum value for the measure, given two classes.^*

2. COMPARE NEW GINI IMPURITIES FOR ANY POSSIBLE CUT. With the baseline established, a software program then (a) tests each possible placement of a cut (at every value on either the credit score or income axis), (b) calculates the new Gini impurity values for each possible partition, R1 and R2, (c) calculates the difference between baseline and new values, and then (d) places the cut to maximize reduction in the Gini impurity. Note the very large number of calculations that need to be done to determine the optimal way to partition the data compared to the formulae used when doing multiple regressions in Chapter 10. These calculations would not be possible without access to cheap computing power. Note also the extreme flexibility of the techniques to separate default loans from repay loans. The partitions are based on the actual sample data without assuming any functional form (such as the linear relationship assumed in Chapter 10).

Exhibit 11-7 shows the distribution of classes (repay and default loans) in the original scatter plot and in the rectangles R1 and R2 that the first cut creates.

For this cut at a credit score equal to 675, the Gini impurity calculations for R1 and R2 are as follows:

Weighted average Gini impurity for R1 and R2 $=$

By placing the first cut at credit score 675, the decision tree algorithm reduces the Gini impurity from 0.500 to 0.329.

Why did the algorithm choose to make the first cut at a credit score of 675? Return to Exhibit 11-6 and lay a ruler along this line, then pull it down so that it sits at a credit score of 600. Count and record the number of red and blue dots in the new R1 and the new R2. We can use these counts to calculate the reduction in Gini impurity if the algorithm had made the first cut at 600.

At credit score equals 600, the Gini impurity calculations for R1 and R2 are

Weighted average Gini impurity for R1 and R2 $=$

By placing the first cut at credit score 600, the decision tree algorithm reduces the Gini impurity from 0.500 to 0.429. Exhibit 11-8 shows the counts and the Gini impurity values if the first cut had been made at several different values of credit score. It also shows counts and the Gini impurity values if the first cut had been made at several different values of income. A first cut at credit score 675 reduces the Gini impurity the most.