Evaluating Data Science Models

Learning Objective 6 Evaluate the predictions of different data science models to choose the best one for the business need and visualize and communicate model insights

• use likelihood values, feature variables, receiver operating characteristic curve, confusion matrix, payoff matrix

Management accountants play a pivotal role in helping managers evaluate and use data science models. Does the model make economic sense? Does it reflect underlying reality? In the Sierra Investments context, can the pruned decision tree model classify loans as repay or default well enough to inform investment decisions? We will present Step 5 in our data science framework by examining five methods for evaluating a model and then conclude with Step 6, in which the management accountant visualizes and communicates insights.

Step 5: Evaluate the Model

Having selected a model, in Step 5 of the decision-making process, Paige helps to evaluate the model and answer important questions. There are several ways to evaluate the model and Paige considers each of them in turn: (1) the magnitude of the likelihood values, (2) evaluation of feature variables, (3) receiver operating characteristic (ROC) curve, (4) confusion matrix classifications, and (5) quantifying predictions using the payoff matrix.

Magnitude of the Likelihood Values

Models with high likelihood values give better predictions. Sometimes even the most sophisticated data science models do not do a good job of prediction because the problem itself is complex, important variables are missing, or the data has errors or is incomplete. It appears from Exhibits 11-14 and 11-15 that the model does very well in predicting the probability of default in both the validation and holdout samples. The average of the likelihood values is about 0.86.^* Average values above 0.65 indicate that the model’s predictions are good. A likelihood value of 0.5 would indicate that it is just a random chance that the model would distinguish default loans from repay loans.

Evaluation of Feature Variables

Management accountants use their knowledge of the business to judge whether the feature variables used to make predictions about the target variable make economic sense. In the Sierra Investments example, Paige is comfortable with the model features used to separate default loans from repay loans. Income levels indicate an ability to repay, and credit scores indicate how borrowers have managed debt in the past. The decision tree partitions identify borrowers with low income levels and low credit scores to be more likely to default and those with high income levels and high credit scores as more likely to repay. These results make intuitive sense and reflect what she believes to be the underlying economic reality. She cannot judge the exact cutoff values derived from the model, but the broad insights are consistent with her business experience.

Identifying Misclassifications

Data science models are not perfect and misclassify predictions. In the Sierra Investments case, the model (1) correctly classifies some default loans but misclassifies other default loans as repay loans and (2) correctly classifies some repay loans but misclassifies other repay loans as default loans. Classifying a default loan as a repay loan and a repay loan as a default loan creates risk.

To identify misclassifications, Paige constructs Exhibit 11-16 using information from the pruned decision tree validation sample in Exhibit 11-14, column 7. Columns 1 and 2 in Exhibit 11-16 order the information in Exhibit 11-14, column 7 from the highest to the lowest predicted probabilities of default. Observation 6 has the highest predicted probability of default of 0.99. She records this as the first observation in Exhibit 11-16, columns 1 and 2. Observations 1, 2, and 4 each have a predicted default probability of 0.33. She records these as the next three observations in Exhibit 11-16, columns 1 and 2. Finally, observations 3, 5, 7, 8, 9, and 10 each have a predicted default probability of 0.01. She records these as the final six observations in Exhibit 11-16, columns 1 and 2. Column 3 records the actual outcome for each observation.

Columns 4 and 5 record the cumulative number of zeroes (repay loans) and ones (default loans) based on the information in column 3. For example, the first observation in column 3 is a 1 (default) so the cumulative number of zeroes after the first observation is 0 and the cumulative number of ones is 1. The second, third, and fourth observations have the same probability of default and are all part of a group. The actual values are 1, 0, and 0. The cumulative number of zeroes after this group of observations is 2 and the cumulative number of ones is also 2.^* After the fifth, sixth, seventh, eighth, ninth, and tenth observations, all of which are zeroes, the cumulative number of zeroes is 8 and the cumulative number of ones is 2.

Columns 6 and 7 in Exhibit 11-16 calculate the false positive rate and the true positive rate. At Sierra Investments, the target (positive) feature variable is defaults. The false positive rate in a classification problem is the fraction of negatives (repay loans) incorrectly identified as positives (default loans) at a given threshold value. The true positive rate in a classification problem is the fraction of positives (default loans) correctly identified as positive (default loans) at a given threshold value. Exhibit 11-16, column 6 calculates the false positive rate equal to the cumulative number of repay loans divided by the total number of repay loans $(\text{column} (4)÷8)$. Exhibit 11-16, column 7 calculates the true positive rate equal to the cumulative number of default loans divided by the total number of default loans $(\text{column} (5)÷2)$

There are two primary ways that Paige can use Exhibit 11-16. First, she can plot the receiver operating characteristic curve, and, second, she can construct the confusion matrix.

Receiver Operating Characteristic (ROC) Curve

A useful and commonly used tool to evaluate a model is the receiver operating characteristic (ROC) curve that plots the false positive rate (FP) on the x-axis and the true positive rate (TP) on the y-axis. Comparing these two rates provides insight into how well a model correctly classifies borrowers at any threshold value. Exhibit 11-17 presents the ROC curve based on the false positive rate (column 6) and the true positive rate (column 7) in Exhibit 11-16. Point A on the ROC curve indicates that the default prediction model can choose a cutoff to get 50% of the true positives (value on the y-axis) without attracting any false positives. This is the case when the cutoff value is set at a default probability above 0.33, say 0.50. At this cutoff default probability, the model correctly predicts loan 6—with a predicted probability of default of 0.99, which is greater than 0.50—to be a default (true positive for one of the two defaulting loans). The model does not predict any repaying loan as a default (zero false positives) because all repaying loans have a predicted probability of default less than 0.50. If, however, Paige wants to avoid lending to both actual defaulters (100% of true positives), she must move to point B on the ROC curve (and choose a default probability below 0.33, say 0.30). At this point, however, Sierra must accept 25% false positives. That is, although this cutoff probability correctly predicts both default loans, it also misidentifies two of eight repay loans as default loans (25%). Increasing the true positive rate also increases the false positive rate.

The more accurately a model predicts defaults as defaults and repays as repays, the closer the ROC curve will go up along the y-axis on the left and then move horizontally across the top. Exhibit 11-18, Panel A shows such an ROC curve. In the context of Sierra Investments, such an ROC curve would mean there is a cutoff probability such that all defaulting loans have a probability of default above the cutoff (100% true positives) without classifying any repayer loans as defaults (0% false positives). In other words, the model has a very good hit rate without any false alarms! The closer an ROC curve follows the diagonal line from the origin (0, 0) to the upper-right corner (1, 1) as in Exhibit 11-18, Panel B, the less effective the model is at correctly predicting a new loan (increasing the true positives increases the false positives at the same rate). Exhibit 11-18, Panel C shows a reasonably good ROC curve, much like the ROC curve in Exhibit 11-17. If the actual ROC curve for the full dataset is like this ROC curve, Paige will be more confident that the model predictions will be useful for predicting defaults without the risk of too many false positives.

Data scientists cannot make perfect predictions. As we have seen, they may predict a loan to default that ultimately is repaid. They may also predict a loan to be repaid that eventually defaults. Based only on the ROC curves, the data scientist favors a point corresponding to a default probability cutoff where the ROC curve starts to flatten out resulting in a high true positive rate without a correspondingly high false positive rate. Paige points out that a far better way to choose a default probability cutoff is to estimate the payoffs—the rewards and costs of each of the four quadrants of the confusion matrix.