11.4.2: Pruning

Pruning

One solution to the problem of overfitting is pruning. Pruning is a technique in which the tree is not grown to its full size, but instead is only allowed to grow to a certain depth. Exhibit 11-12 shows the decision tree from the previous example grown to a depth of three layers.

A tree diagram shows pruned to depth of three with income shown in 000s dollars.

The diagram shows three layers of the tree. The details of the diagram are as follows:

The credit score with a value of 675 is at Cut 1. The 13 dots with credit scores higher than 675 are on the right, to R1, while the 11 dots with credit scores less than 675 are on the left, to R2. The node of R2 is labeled as cut 3 with an income of 85,000 dollars. The nine borrowers in R5 with incomes below 85,000 dollars move to the left to default. The two borrowers in R6 with incomes higher than 85,000 dollars move right to repayer.

The node of R1 is labeled as cut 2 with an income of 56,000 dollars. At the first node, the 10 borrowers whose income is greater than 56,000 dollars go to the right, while the three borrowers whose income is less than 56,000 dollars go to the left.

The node of R3 is labeled as cut 4 with a credit of 768. The two borrowers in R7 with credit below 768 move to the left to default. The one borrower in R8 with credit higher than 768 move right to repay. The node of R4 is labeled as cut 5 with a credit of 712. A dashed line labels the level as pruning at depth 3. The ten borrowers in R10 with credit above 712 move to the right to repayer. The three borrower in R9 with credit below 712 move left to 2 repayers and 1 defaulter.

If we prune the tree to a depth of three layers, we get the following tree and rule for defaulters:

If credit score $begin inline style less than 675 end style$ and income $begin inline style less than straight $ text 85,000 end text comma space text then end text equals 1 straight thin space left parenthesis text default end text right parenthesis end style$ .
If $begin inline style 675 less than end style$ credit score $begin inline style less than 768 end style$ and income $begin inline style less than straight $ text 56,000 end text comma space text then end text equals 1 straight thin space left parenthesis text default end text right parenthesis end style$ .
If $begin inline style 675 less than end style$ credit score $begin inline style less than 712 end style$ and income $inline style begin inline style greater than straight $ text 56,000 end text end style$ , then default with probability 1/3.

If the borrower’s features do not satisfy any of these conditions, then $inline style begin inline style text classify?as end text equals 0 thin space left parenthesis text repay end text right parenthesis end style$ .

The first two rules for classifying defaulters in the pruned tree are identical to the rules for classifying defaulters in the fully grown tree (also called full tree). The third rule is different. Pruning creates one impure node in the lower right part of the tree, as shown in Exhibit 11-12. Of the three borrowers whose credit scores are less than 712, two borrowers at that node are repayers and one is a defaulter. Without purity in the node, the model classifies the three borrowers in probabilistic terms. When a borrower’s credit score is less than 712 but greater than 675, and income is greater than $56,000, the pruned decision tree model indicates there is a 1/3 probability of a borrower defaulting (1 default loan out of 3 total loans).

Try It! 11-2

Rafael Alvarez, the management accountant at Wyatt Manufacturing, builds the following decision tree. He prunes the tree at a depth of 4. Write the rules for classifying defective products based on the pruned tree.

The details of the tree depth are as follows:

Depth 1: Auto level at 1.95 branches in to two, 7 and 9.
Depth 2: 7 leads to 18 years’ experience and 9 leads to 6.25 years’ experience.
Depth 3: 18 years’ experience branches into 6 and 1 leading to defect and good respectively. 6.25 years’ experience branches into 1 and 8 leading to defect and 2.5 auto level respectively.
Depth 4: 2.5 auto level branches into 3 and 6 leading to 9.25 years’ experience and good. A dashed line marks the level as pruned. The node 9.25 years’ experience branches into 2 and 1 leading to good and defect respectively.

Watch Try It Video

Building models by pruning trees raise two questions: (1) To what depth should the tree be pruned? For example, why not prune the tree up to depth 2 rather than depth 3? (2) How can a data scientist compare trees when there are thousands of data points and hundreds of features? In some contexts, as was the case in the prior example, the management accountant can use his or her understanding of the economics of a business to recognize overfitting, but in other, more complex models with many features, this may be more difficult. To choose among models and to decide where to prune, data scientists cross-validate the model. The management accountant uses his or her business knowledge to interpret and evaluate the selected model.