Confusion Matrix

A confusion matrix is a matrix that shows the predicted and actual classifications at a given threshold value. In the Sierra Investments example, this threshold value is the cutoff value for the default probability above which the model would indicate not to invest in the loan and below which the model would suggest investing in the loan. We will label the decision not to invest in the loan as the model predicting default (or, more accurately, high risk of losses because of default risk) and the decision to invest in the loan as the model predicting repay (or, more accurately, high returns because of the likelihood of repayment).

Just as we did in Exhibit 11-13, the most intuitive cutoff value is a default probability of 0.50. That is, for all values greater than 0.50, the model would predict not to invest (default) and for all values less than 0.50, the model would predict invest (repay). Exhibit 11-19 shows the confusion matrix for the validation sample using the pruned decision tree and a cutoff probability of 0.50.

To see how to build the confusion matrix at a cutoff value of 0.50, consider the line drawn at a value of 0.50 in Exhibit 11-16 between the probability values of 0.33 and 0.99. The cutoff probability of 0.50 means that Sierra Investments will reject all loans for which the predicted default probability is greater than or equal to 0.50 (that is, all loans above the line) and invest in all loans for which the predicted default probability is less than 0.50 (that is, all loans below the line). Ideally, Sierra would like (1) all loans above the line to be default loans (coded as 1) with no repay loans (coded as 0) and (2) all loans below the line to be repay loans (coded as 0) with no default loans (coded as 1).

From Exhibit 11-16, Sierra Investments will predict loan 6 as default and will not invest in this loan (which has a predicted probability of default of 0.99). It will predict all other loans as repays and invest in these loans (which have a predicted probability of default of 0.33 or 0.01). The cutoff probability of 0.50 correctly predicts that loan 6 will actually default, but it predicts that Sierra should invest in loan 1 when it actually defaults. It correctly predicts the other eight loans below the cutoff as repays. Using the cutoff probability of 0.50 results in only one misclassification.

This leads to the confusion matrix in Exhibit 11-19. The first column shows the only predicted default, loan 6, which actually defaults and appears in the box labeled (A) as true positives (TP)—positives correctly predicted as positives, that is, default loans correctly predicted as defaults. Sierra Investments does not invest in loans predicted as default and avoids those losses. Loans that are actually repays but are falsely predicted as defaults appear in the box labeled (Y) as false positives (FP)—negatives incorrectly predicted as positives. These are good loans that Sierra would like to invest in and earn interest on but will not because the model wrongly predicts them as default loans. At a cutoff probability of 0.50 there are no such loans, so the box labeled Y has a value of 0.

The second column in Exhibit 11-19 shows the predicted repay loans – 1, 2, 3, 4, 5, 7, 8, 9, and 10. Of these nine loans, eight actually repay and appear in the box labeled (B) as true negatives (TN)—negatives correctly predicted as negatives, that is, repay loans correctly predicted as repays. Sierra invests in the loans classified as repay and gets the benefit of interest payments on these fully paid loans. There is one loan (loan 1) in the box labeled (Z) or false negatives (FN)—positives incorrectly predicted as negatives, that is loans predicted to repay, leading Sierra to invest, but which actually default. Sierra would like to avoid loans in box Z because they lead to loss of principal as a result of a default.

So, what is the consequence of setting the cutoff probability at 0.50? The good news is that Sierra invests in all loans that repay. The downside is that Sierra invests in one loan, loan 1, which the model misclassifies as repay but which actually defaults (false negative).

As the management accountant, Paige has insight into this problem that data scientists do not have. She understands that it is very costly for Sierra to invest in loans that default. Even though a cutoff probability of default of 0.50 may sound very appealing for separating default loans from repay loans, Sierra may want to be conservative and minimize investing in loans that default even if it means not investing in some loans that repay. One way for Sierra to invest in fewer default loans is to set the cutoff probability lower at, say, 0.30. At this cutoff, Sierra will only invest in loans when the predicted default probability is less than 0.30 and not invest in loans when the predicted default probability is greater than 0.30.^*

The lower cutoff probability means lowering the line in Exhibit 11-16 below loans 1, 2, and 4, which have a predicted probability of default of 0.33. This means loans 6, 1, 2, and 4 with a predicted default probability of 0.99 and 0.33 are above the cutoff line. Sierra will not invest in these loans because they are above the cutoff probability. It will invest in loans 3, 5, 7, 8, 9, and 10 that are below the cutoff line (with a predicted default probability of 0.01). This leads to the confusion matrix in Exhibit 11-20. The first column shows the loans Sierra will not invest in (called predicted defaults)—loans 6, 1, 2, and 4. Of these loans two loans, 6 and 1, actually default and appear in the box labeled (A) as true positives (TP). Sierra does not invest in these loans and avoids these losses. Two loans, 2 and 4, actually repay but are above the cutoff probability and appear in the box labeled (Y) as false positives (FP). These are repay loans that Sierra would have liked to invest in and earn interest on but will not because the predicted default probability is above the cutoff probability.

The second column in Exhibit 11-20 shows the loans Sierra will invest in (called predicted repay)—3, 5, 7, 8, 9, and 10. All six of these loans actually repay and appear in the box labeled (B) as true negatives (TN). There are no loans in the box labeled (Z) as false negatives (FN). At the 0.30 cutoff, Sierra avoids investing in loans that are predicted to repay but actually default.

The two confusion matrices indicate the nature of the tradeoff in choosing the cutoff default probability. If the cutoff probability is set low (0.30), Sierra would only invest in loans that have a default probability of less than 0.30. It would avoid investing in loans that default (false negatives) but would also miss out on investing in loans that repay (true negatives). By raising the cutoff default probability to 0.50, Sierra would be more aggressive and invest in loans with a default probability less than 0.50. It would then invest in more loans that repay (true negatives) but would also be stuck with some loans that default (false negatives). In summary, raising the cutoff decreases true positives and false positives and increases true negatives and false negatives.

So which cutoff should Sierra choose? At first glance, the cutoff probability of 0.50 appears more appealing—it is intuitive in that loans with default probabilities above 0.50 are rejected while those with default probabilities below 0.50 are accepted. The confusion matrix in Exhibit 11-19 at a cutoff probability of 0.50 also misclassifies only one loan (loan 1 as a false negative). In contrast, the confusion matrix in Exhibit 11-20 at a cutoff probability of 0.30 misclassifies two loans (loans 2 and 4 as false positives). But choosing cutoffs is seldom obvious. It depends on the payoffs for each quadrant of the confusion matrix.

Quantifying Predictions Using the Payoff Matrix

Management accountants use their insights and knowledge of the business to estimate the payoffs. In the Sierra Investments example, Paige must choose between the 0.30 default probability cutoff of Exhibit 11-19 or the 0.50 default probability cutoff of Exhibit 11-20. Paige estimates the payoffs to Sierra for each of the four quadrants of the confusion matrix over the 3-year period of the loans. For simplicity, she ignores the time value of money. She prepares the payoffs assuming she invests $100 in a loan.

1. The payoff for the true negatives in quadrant B—loans that Sierra invests in that actually repay—is relatively easy. If Sierra invests $100 in each loan, Sierra will receive a payoff of 15% per year or $45 over 3 years, in addition to getting back the $100 the firm invested for a net payoff of $45.
2. The computations for the false negatives in quadrant Z—loans Sierra invests in but which actually default—is not so straightforward. Sierra would not lose all the $100 invested in these loans. That’s because defaulting loans start defaulting sometime along the way during the next 3 years. Paige does a detailed analysis of how much borrowers repay before they default. She estimates that Sierra will, on average, receive $30 as interest and principal repayments on the loan before the loan defaults. She therefore concludes that the payoff for false negatives in quadrant Z is $-\$70$.
3. Next, she considers the payoffs for all outcomes where Sierra does not invest in the PD loan because the model predicts a high default risk. These are the true positives in quadrant A—loans that are predicted to have high default risk and actually default and the false positives in quadrant Y—loans that are predicted to have high default risk but actually repay. In either case, once the model predicts these loans as having high default risk, Sierra will not invest in these loans even though in the false positive case, it would have been beneficial for Sierra to invest. The key question is the payoff Sierra would get elsewhere from the money not invested in PeerLend Digital.

Case A: Paige learns that Sierra would invest $100 not invested in PeerLend Digital in a bond fund that is expected to earn 4% per year or $12 over 3 years. She uses a payoff of $12 for both quadrant A and quadrant Y. Exhibit 11-21 presents the payoff matrix.

If Paige chooses a default probability cutoff of 0.30, her payoff for that strategy equals the number of loans in each quadrant in Exhibit 11-20 multiplied by the payoff for the corresponding quadrant in Exhibit 11-21:

The total investment for all 10 loans is $\$\text{100}\times 10=\$\text{1,000}$ so a $318 net payoff would equal a return of $31.8\mathrm{\%\hspace{0.17em}}(\$318÷\$\text{1,000})$ over 3 years or an average of 10.6% per year $(31.8\%÷3)$.

If Paige chooses a default probability cutoff of 0.50, her payoff for that strategy equals the number of loans in each quadrant in Exhibit 11-19 multiplied by the payoff for the corresponding quadrant in Exhibit 11-21:

What if Paige had chosen not to develop the decision tree model to predict defaults and had done no analysis? She could then simply invest in all 10 loans in the validation sample. In this case, Sierra’s expected payoff would be $45 for the eight repay loans and $-\mathrm{\$}70$ for the two default loans or $(\$45\ \times \mathrm{\ 8})\ +\ (-\$70\ \times \mathrm{\ 2})=\$220$, or a return of 22% $(\mathrm{\$}\mathrm{\ 220}÷\mathrm{\$}\text{1,000})$ over 3 years or 7.33% $(22\mathrm{\%}÷3)$ per year.

On the basis of these assumptions, Paige will recommend to her managers that Sierra should be conservative and use a default probability cutoff of 0.30 to invest in the loans. That is, only invest in loans for which the predicted probability of default is less than 0.30. This means that Sierra will only invest in 6 of the 10 loans so as to avoid the losses that come from defaulting loans. Using the decision tree model to invest in loan decisions increases Sierra’s profitability from 7.33% to 10.6%.

Case B: If the outside investment opportunities yield a payoff of only 2% per year, or $6 over 3 years, the expected payoffs are as follows:

In this scenario, the outside investment opportunities are sufficiently unattractive that Sierra is willing to be more aggressive (invest in loans with a default probability less than 0.50). Sierra will invest in many more repaying PeerLend Digital loans (true negatives) even though this means accepting some false negatives (default loans) as well.

Depending on the outside investment opportunities (Case A or Case B), the decision tree model helps Sierra increase the payoff to approximately $300 or $30\mathrm{\%\hspace{0.17em}}(\$300÷\mathrm{\$}\text{1,000})$ over 3 years, or $10\%(30\%÷3)$ per year. Paige can now clearly visualize what the model is doing and is eager communicate her takeaways to managers.