CONTINUING TO BUILD THE TREE

The algorithm next evaluates all possible values for a horizontal cut on credit score or a vertical cut on income in any one rectangle. In the scatter plot in Exhibit 11-9, it places the second cut in rectangle R1 on the income feature of $56,000. This line cuts R1 into two smaller rectangles, R3 to the left and R4 to the right. The three loans in R3 have incomes less than $56,000, of which one is a repay loan and two are default loans. The 10 loans in R4 have incomes greater than $56,000, where nine are repay loans and one is a default loan. There are now three rectangles, R2, R3, and R4, and none is fully pure. In the Exhibit 11-9 decision tree, the node to the right of the first node now has an income value of $56,000. At the first node, the 13 borrowers whose credit score is greater than 675 go to the right; those 10 borrowers whose income is greater than $56,000 move again to the right, while the three borrowers whose income is less than $56,000 go to the left.

Exhibit 11-9

Second Cut of the Decision Tree (with income shown in $000s)

A scatter plot shows the second cut decision tree with income shown in 000s dollar.

The x-axis shows x subscript income (dollar 000s) from 30 to 110 in increments of 10. The y-axis shows x subscript 2 credit score from 500 to 850 in increments of 50. A series of 24 dots comprising of 12 each of red and blue color represents default and y: repay respectively are scattered across the grid. The repay dots are scattered on the top part of the grid, while the default dots are scattered on the bottom part of the grid. A horizontal line is marked from 675 of credit score cutting the grid into two rectangular segments. The rectangle above the line is labeled as R1 and the rectangle below the line is labeled as R2. A vertical line is marked for the second cut in rectangle R1 cutting it into two smaller rectangles, R3 to the left and R4 to the right. The line is placed at 56,000 dollars of income range.

A visualization of the decision tree is shown on the right side of the plot. The credit score with a value of 675 is at the first decision node. The 13 dots with credit scores higher than 675 are on the right, to R1, while the 11 dots with credit scores less than 675 are on the left, to R2. The node of R1 is labeled as cut 2 with an income of 56,000 dollars. At the first node, the 13 borrowers whose credit score is greater than 675 go to the right of which 10 borrowers whose income is greater than 56,000 dollars move to the right, while the three borrowers whose income is less than 56,000 dollars go to the left.

A table below the plot shows the tally of default and repay loans across each of the four rectangles. R1 is consistently not applicable for all segments. The details of loan status, R2, R3, R4 and total respectively are as follows: Repay (blue): 2, 1, 9, 12; Default (red): 9, 2, 1, 12; Total: 11, 3, 10, 24.

For the second cut at income equals $56,000, the Gini impurity calculations for R3 and R4 are

R3: 13×(1−13)+23×(1−23)=13×23+23×13=49=0.444R4: 910×(1−910)+110×(1−110)=910×110+110×910=18100=0.180  

Weighted average Gini impurity for R2, R3, and R4 =

proportion of observations in R2×Gini impurity for R2+proportion of observations in R3×Gini impurity for R3 + proportion of observations in R4×Gini impurity for R4       1124×0.298+324×0.444+1024×0.180=0.267

By placing the second cut at income $56,000, the decision tree algorithm reduces the Gini impurity from 0.329 to 0.267.

On its third iteration, the algorithm makes a cut on the income value of $85,000 in rectangle R2. In the scatter plot in Exhibit 11-10, it splits rectangle R2 into R5 to the left and R6 to the right. The nine borrowers in R5 have incomes less than $85,000 and zero are repay loans and nine are default loans. The two borrowers in R6 have incomes greater than $85,000; two are repay loans and zero are default loans. Both rectangles R5 and R6 are now pure. The purity of these nodes in the decision tree is indicated in Exhibit 11-10 by a rectangle around the name of the class. The third node has an income value of $85,000; the two borrowers with incomes higher than this move right. They both repay, so the terminal node is labeled “repay.” The nine borrowers with incomes below $85,000 move to the left and, because they all default, this too is a terminal node and labeled “default.”

Exhibit 11-10

Third Cut of the Decision Tree (with income shown in $000s)

A scatter plot shows the third cut decision tree with income shown in 000s dollar.

The x-axis shows x subscript income (dollar 000s) from 30 to 110 in increments of 10. The y-axis shows x subscript 2 credit score from 500 to 850 in increments of 50. A series of 24 dots comprising of 12 each of red and blue color represents default and y: repay respectively are scattered across the grid. The repay dots are scattered on the top part of the grid, while the default dots are scattered on the bottom part of the grid. A horizontal line is marked from 675 of credit score cutting the grid into two rectangular segments. The rectangle above the line is labeled as R1 and the rectangle below the line is labeled as R2. A vertical line is marked for the second cut in rectangle R1 cutting it into two smaller rectangles, R3 to the left and R4 to the right. Another vertical line is placed at 85,000 dollars of income range cutting rectangle R2 cutting it into two rectangles, R5 to the left and R6 to the right.

A visualization of the decision tree is shown on the right side of the plot. The credit score with a value of 675 is at the first decision node. The 13 dots with credit scores higher than 675 are on the right, to R1, while the 11 dots with credit scores less than 675 are on the left, to R2. The node of R1 is labeled as cut 2 with an income of 56,000 dollars. At the first node, the 13 borrowers whose credit score is greater than 675 go to the right of which 10 borrowers whose income is greater than 56,000 dollars move to the right, while the three borrowers whose income is less than 56,000 dollars go to the left. The node of R2 is labeled as cut 3 with an income of 85,000 dollars. The nine borrowers in R5 with incomes below 85,000 dollars move to the left to default. The two borrowers in R6 with incomes higher than 85,000 dollars move right to repay.

A table below the plot shows the tally of default and repay loans across each of the six rectangles. R1 R2 is consistently not applicable for all segments. The details of loan status, R3, R4, R5, R6 and total respectively are as follows: Repay (blue): 1, 9, 0, 2, 12; Default (red): 2, 1, 9, 0, 12; Total: 3, 10, 9, 2, 24.

For the third cut in our example, at income equals $85,000, the Gini impurity calculations for R5 and R6 are

R5: 09×(1−09)+99×(1−99)=09×99+99×09=0R6: 22×(1−22)+02×(1−02)=22×02+02×22=0  

Weighted average Gini impurity for R3, R4, R5, and R6 =

proportion of observations in R3×Gini impurity for R3+proportion of observations in R4×Gini impurity for R4 +proportion of observations in R5×Gini impurity for R5+proportion of observations in R6×Gini impurity for R6324×0.444+1024×0.180+924×0+224×0=0.131  

By placing the third cut at income of $85,000, the decision tree algorithm reduces the Gini impurity from 0.267 to 0.131.

The algorithm continues this process over and over until all rectangles are pure. The technical term for this process is recursive partitioning. Recursion means to apply a procedure again and again. In this case, the algorithm is placing “cuts” to partition the space into smaller and smaller subsets. Exhibit 11-11 shows the full decision tree.

Exhibit 11-11

Fully Grown Decision Tree (with income shown in $000s)

A scatter plot shows fully grown decision tree with income shown in 000s dollar.

The x-axis shows x subscript income (dollar 000s) from 30 to 110 in increments of 10. The y-axis shows x subscript 2 credit score from 500 to 850 in increments of 50. A series of 24 dots comprising of 12 each of red and blue color represents default and y: repay respectively are scattered across the grid. The repay dots are scattered on the top part of the grid, while the default dots are scattered on the bottom part of the grid. A horizontal line is marked from 675 of credit score cutting the grid into two rectangular segments. The rectangle above the line is labeled as R1 and the rectangle below the line is labeled as R2. A vertical line is marked for the second cut in rectangle R1 cutting it into two smaller rectangles, R3 to the left and R4 to the right. Another vertical line is placed at 85,000 dollars of income range cutting rectangle R2 cutting it into two rectangles, R5 to the left and R6 to the right.

A horizontal line at 768 credit score cuts the rectangle R3 into two parts, R7 and R8. Two horizontal lines at credit score 712 and 704.5 cuts the rectangle R4 into R10, R11, and R12.

A visualization of the decision tree is shown on the right side of the plot. The credit score with a value of 675 is at the first decision node. The 13 dots with credit scores higher than 675 are on the right, to R1, while the 11 dots with credit scores less than 675 are on the left, to R2. The node of R1 is labeled as cut 2 with an income of 56,000 dollars. At the first node, the 13 borrowers whose credit score is greater than 675 go to the right of which 10 borrowers whose income is greater than 56,000 dollars move to the right, while the three borrowers whose income is less than 56,000 dollars go to the left. The node of R2 is labeled as cut 3 with an income of 85,000 dollars. The nine borrowers in R5 with incomes below 85,000 dollars move to the left to default. The two borrowers in R6 with incomes higher than 85,000 dollars move right to repay.

The node of R3 is labeled as cut 4 with a credit of 768. The two borrowers in R7 with credit below 768 move to the left to default. The one borrower in R8 with credit higher than 768 move right to repay. The node of R4 is labeled as cut 5 with a credit of 712. The ten borrowers in R10 with credit above 712 move to the right to repay. The three borrower in R9 with credit below 712 move right to the node cut 5 with credit of 704.5. The two borrowers in R11 with credit below 704.5 move to the left to repay while the one borrower in R12 with credit above 704.5 move right to default.

The full set of classification rules for defaults in the decision tree model in Exhibit 11-11 are as follows:

  1. If credit score <675 and income <$85,000, then =1 (default), identified by red line (dashed). There are nine defaults identified by this rule.
  2. If 675< credit score <768 and income <$56,000, then =1 (default), identified by green line (solid). There are two defaults identified by this rule.
  3. If 704.5< credit score <712 and income >$56,000, then =1 (default), identified by blue line (dotted). There is one default identified by this rule.

If the borrower’s features do not satisfy any of these conditions, then classify as = 0 (repay).

Paige is puzzled by this analysis. She observes that at the node labeled 704.5 the default terminal node is to the right of the repay terminal node. This suggests that for the same level of income (greater than $56,000), the decision tree will classify a borrower with a higher credit score (greater than 704.5) as a default, while classifying a borrower with a lower credit score (less than 704.5) as a repay. She wonders why this happened and whether it makes sense.

Watch Try It Video