Management accountants work with managers to operationalize the data science model to make decisions. In doing so, they evaluate what elements need to be modified in light of the model inputs and how best to balance quantitative and qualitative assessments. We use the Sierra Investments context to describe these general insights.
As the manager prepares to implement the model, the management accountant must underscore critical model inputs and areas of significant judgment. These normally pertain to data and cutoff values.
Data issues can be tricky and require judgment. Are the data adequate for the intended purpose? How accurate are the data? How representative are the past data for predicting the future? Are past conditions different from what might be encountered in the future? Paige knows from her understanding of data science models and the bias–variance tradeoff that the models are powerful enough to deal with spotty data. But if she judges the data to be more suspect, it is a problem that data scientists cannot overcome.
For example, Paige trusts that PD is providing relevant and accurate data. That’s because creating and maintaining the integrity of pertinent and appropriate data is one of PD’s major value propositions to lender-investors. It is an issue PD takes very seriously. However, it is very costly for PD to verify all lender information. Paige reviews the data that individual applicants provide to PD to secure a loan and uses qualitative judgment to evaluate whether the data are accurate enough to use in the models.
Paige is also aware that the model reflects PD loan activity up to a specific moment in time. These are the data she uses to build a model to predict the future. The model only works if historical data are representative of the future. Market conditions change all the time. Paige will have to use qualitative judgment to assess whether the model is still useful 2 years from now or whether she needs to obtain newer data from PD to train new models. These are important judgments that she will have to make. She thinks back to the recession that began in October 2008. Wall Street’s most sophisticated models failed to predict the calamitous stock market crash and resulting recession caused by a particular mix of conditions and dynamics that had never happened before. Paige will make the manager aware of these issues. But in her own judgment, she concludes that the data are rich, accurate, and representative enough to use to predict defaulting loans in the future.
The choice of the cutoff depends heavily on the payoff matrix, so it is important to get good estimates of the payoffs. The management accountant best understands the economic consequences of different decisions and how certain he or she is about the different payoff values. Where there is uncertainty, the management accountant evaluates the sensitivity of different payoffs on the cutoff decision.
Paige has a strong understanding of the effect of alternate investment strategies on the PeerLend Digital investment decision. She is fairly confident about the payoffs if the loans repay since these are contracted amounts and the returns from alternative investments if Sierra does not invest in PD loans. She is less sure about the value she should assign to false negatives. She varies the loss from $60 to $80 on a $100 loan. The cutoff values change but, as in Exhibit 11-25, the overall payoffs are reasonably close to each other and well above the payoffs from investing in all 500,000 PD loans. She plans to share these analyses with the manager to support her recommendation to use the model to invest in PD loans.
Advances in data science create many exciting opportunities for organizations. Although the algorithms vary, the basic approach to data science follows the methods described in this chapter—avoiding overfitting, balancing the bias–variance tradeoff, identifying false positives and false negatives, evaluating models, and measuring economic impact. Management accountants bring deep insights about the economic consequences of decisions. By understanding the statistical and computer science tools that data scientists use, management accountants can take advantage of these tools to build models that create value.