Visualizing and communicating the insights of data science models is an important task for the management accountant because it helps managers understand the value and tradeoffs from using these models. We have already seen ways of visualizing how models work (for example, using a decision tree diagram to explain how different values of the feature variables separate different classes of the target variable). The ROC curve and the confusion matrix help managers visualize the performance of a model by identifying the tradeoff between false positives and true positives (and the corresponding false negatives and true negatives). The visual shape of the ROC curve communicates how accurately a model can classify the target value of interest (default loans in Exhibit 11-18).
Communicating the output of a model can help managers grasp the choices they face and the judgments they must make. For example, Paige should try to communicate and explain intuitively why the decision tree model helps to generate higher returns. Suppose the outside investment opportunities are as described in Case B. At a default probability cutoff of 0.50, the decision tree model identifies one of the two defaulting loans without classifying any of the repaying loans as defaults. As a result, Sierra will invest in nine PD loans, of which one loan is a default. In the absence of the model, Sierra would have invested in 10 PD loans, of which two would be defaults. Correctly identifying one default loan and not investing in it increases Sierra’s payoff by $70 from $220 to . In addition, Sierra earns $6 by investing in a loan outside PeerLend Digital, resulting in a total payoff of $296.
What if the manager chooses the lower default probability cutoff of 0.30? Paige explains that at this cutoff, Sierra would avoid investing in both default loans but will misclassify two repay loans as defaults and so not invest in those repay loans. Correctly identifying two default loans and not investing in them increases Sierra’s payoff by . But not investing in two of the repay loans lowers Sierra’s payoff by . The result is an increase in the payoff by from $220 to $270. In addition, Sierra earns $24 by investing in four loans outside PeerLend Digital at $6 per loan, resulting a total payoff at the 0.30 cutoff of .
Paige should also communicate to her manager when the model would not be effective in increasing returns. Suppose that at the default probability cutoff of 0.30, the model correctly identified only one of the two defaulting loans (true positives), but in the process also classified two repaying loans as defaults (false positives). Once again, correctly identifying and not investing in the default loan will increase Sierra’s payoff by $70 but now Sierra will also not invest in two of the repay loans, resulting in the payoff decreasing by . Sierra would be worse off by $20. By investing in three outside loans, Sierra will gain , but is still worse off by using the model. The model only has value if it can identify true positives (default loans) without classifying too many repay loans as defaults (false positives).