Decision Points

1. How can management accountants work with data scientists to create value?

Management accountants can contribute in each step of the seven-step decision-making process for applying machine learning techniques in business situations: (1) Gain a business understanding of the problem, (2) Obtain and explore relevant data, (3) Prepare the data, (4) Build a model, (5) Evaluate the model, (6) Visualize and communicate insights, and (7) Deploy the model.

2. Will solving the proposed problem create value, and are the relevant data available?

The management accountant must work with data scientists to judge whether addressing the problem will create value. The management accountant must avoid target leakage and ensure that the data are objective, correctly measured, and relevant for predicting the target feature.

3. What are the strengths and weaknesses of a decision tree model?

A decision tree model is an algorithmic predictive modeling technique that subdivides data along features to reduce Gini impurity so that each partition is purer (contains more members of one class than another). Its strength is that it is very flexible to fit the dataset on which it is trained. Its weakness is that it will fit to noise, not just the signal, and so will not perform well on a brand-new dataset.

4. What is pruning, and why is it useful?

Pruning is a technique by which the tree is not grown to its full size, but instead only allowed to grow to a certain depth. The benefit of pruning is that it helps to avoid overfitting that occurs when a model adheres too closely to the specific details of a dataset such that, in addition to signal, it captures noise from random chance.

5. How do managers choose among different data science models?

Managers ask data scientists to cross-validate the model by comparing predictions of different models on a new set of data for which the actual outcomes (for example, default or repay) are already known. Managers choose the model that predicts most accurately based on the overall number of correct predictions or a higher overall likelihood value. The selected model best balances the bias–variance tradeoff. In the context of decision trees, the greater the depth, the lower the bias and the higher the variance of the model. Data scientists also verify that the overall likelihood value of the validation sample used to choose the model is similar to the overall likelihood value of a brand-new and yet-unseen (by the model) hold-out sample.

6. How do management accountants help managers evaluate data science models?

Management accountants help managers answer the following questions when evaluating models: (1) Are the likelihood values big enough? (2) Are the confusion matrix and receiver operating characteristic (ROC) curve useful to inform decisions? (3) Do the model and feature variables make economic sense? (4) Does the model reflect underlying reality? (5) Is there confidence in the payoff matrix to guide decisions? To help managers evaluate models, management accountants visualize and communicate insights from the model.

7. How can management accountants help managers to operationalize data science models?

Management accountants help managers understand the critical inputs to be monitored and evaluated to implement models. They combine quantitative and qualitative judgment about how the data and the model should be used to reach conclusions.