Assignment Material

Questions

1. 11-1. How do management accountants work with data scientists to create value for an organization?
2. 11-2. What is the seven-step decision-making process for applying machine-learning techniques in business situations?
3. 11-3. Define target leakage.
4. 11-4. Describe the decision tree technique of predictive modeling.
5. 11-5. What is Gini impurity?
6. 11-6. Why does overfitting occur?
7. 11-7. What is pruning? Why is it helpful?
8. 11-8. How do data scientists use cross-validation and holdout samples?
9. 11-9. What is the likelihood value? Why do prediction models attempt to maximize it?
10. 11-10. Explain the bias–variance tradeoff.
11. 11-11. What is the receiver operating characteristic (ROC) curve?
12. 11-12. What is the confusion matrix?
13. 11-13. Explain false positives (FP) and false negatives (FN).
14. 11-14. How does the management accountant use the payoff matrix to make decisions using data science models?
15. 11-15. How do management accountants help managers to operationalize data science models?

Multiple-Choice Questions

1. 11-16. Which of the following is not a primary component of business applications of data science?
   1. a. Computer science and data skills
   2. b. Substantive expertise in natural sciences
   3. c. Math and statistics
   4. d. Substantive expertise in an applicable business
2. 11-17. Which of the following statements describes cross-validation?
   1. a. Cross-validation is close and intense monitoring of all the manufacturing steps and critical points in at least the first three production-scale batches.
   2. b. Cross-validation is an automatic computer check to ensure that the data entered is sensible and reasonable and conforms to general data-type expectations. It does not check the accuracy of data.
   3. c. Cross-validation is using historical data to provide documentary evidence that practices, policies and/or products exist in the state in accordance with their design specifications
   4. d. Cross-validation is the process of comparing predictions of different models on a new set of data for which the actual outcomes are already known.
3. 11-18. What is the proper order of the steps in the data science framework?
   1. a. Visualize and communicate insights, build a model, obtain and explore relevant data, prepare data, deploy the model, evaluate the model, gain a business understanding of the model.
   2. b. Build a model, obtain and explore relevant data, prepare data, deploy the model, evaluate the model, gain a business understanding of the problem, visualize and communicate insights.
   3. c. Gain a business understanding of the problem, obtain and explore relevant data, prepare data, build a model, evaluate the model, visualize and communicate insights, deploy the model.
   4. d. Build a model, evaluate the model, deploy the model, obtain and explore relevant data, prepare data, visualize and communicate insights, gain a business understanding of the problem.
4. 11-19. Which of the following statements about decision trees is true?
   1. a. Evaluate and choose among candidate decision trees based on their performance on the cross-validation data, not the training data they were built from.
   2. b. The more branches a tree has, the more precisely and correctly it will make predictions.
   3. c. Decision trees should take into account all available data on a topic, even data which is unavailable at the time a choice is to be made.
   4. d. To make accurate predictions, decision trees need to be fully built out so that all nodes are pure.
5. 11-20. At what point does an analyst stop the building and/or pruning process of a decision tree model?
   1. a. Several models with different amounts of pruning are evaluated against the validation sample to find the best model.
   2. b. Several models with different amounts of pruning are evaluated against the training sample to find the best model.
   3. c. Decision trees continue adding branches until all resulting rectangles are pure. This is therefore the perfect model and should be deployed.
   4. d. Decision trees continue adding branches until the Gini impurity reaches a threshold set by the analyst.

©2016 DeVry/Becker Educational Development Corp. All Rights Reserved.

Exercises

1. 11-21. GINI IMPURITY. Kevin Brown is the management accountant at Boehm and Sons Bank. Some of his colleagues attended a conference recently where they learned that machine learning can be used to accurately predict loan performance. Excited by the possibilities, they asked Kevin to work with the data science team to develop a machine learning model that predicts whether a loan will repay or default. Kevin knows that defaulting loans have significantly hurt Boehm and Sons’ profitability.
2. Kevin tells the data science team that his colleagues will appreciate a model that is interpretable and easy to understand, so they decide to use a decision tree. The team selects a random sample of eight loans (three loans that defaulted and five loans that repaid) from Boehm and Sons’ internal loan database and begins the analysis. 

1. Required

1. Use your ruler to identify two possible places to make the first cut. Visually, is one of these cuts better than the other at separating the data?

2. For each potential cut, calculate the Gini impurity and the information gain. Based on these calculations, which cut should be used to form the first node of the decision tree? Is this consistent with your visual intuition from requirement 1?

3. How could Kevin use the decision tree analysis to help senior managers at Boehm and Sons improve profitability?

1. 11-22. GINI IMPURITY. Laurie Rech is a management accountant at Donnelly Bank, which has recently suffered significant loan losses. Rech and her team are worried that they do not fully understand the risk profile of their loans, so they would like some way of identifying loans that are likely to default. One team member suggests developing a decision tree so that loans can be quickly and easily classified as “likely to default” or “likely to repay.”
2. After plotting and inspecting a sample of seven loans (three loans that defaulted and four loans that repaid), Rech and her team notice there are two different cuts that result in only one misplaced observation: $\text{Credit\hspace{0.17em}Score}=650$ and $\text{Income}=\mathrm{\$}\text{70,000}$. Based on this, they conclude that either would be fine to use as the first node for their decision tree.

1. Required

1. Are Rech and her team correct that the two cuts are equivalent? Which one would you choose to be the first node of the decision tree?

2. Why might Rech care about Gini impurities when making decisions?

1. 11-23. DECISION TREES. The data science team at Swift Investments uses a sample of 16 loans to build a decision tree that predicts whether a loan will repay or default. Although the data science team is very familiar with decision trees and other machine learning models, they do not have much experience in the finance or lending industries.
2. The team shows the decision tree to Royce Brown, the management accountant at Swift, who indicates that some of the decisions made by the model seem incorrect. To understand how the decision tree partitions the data, the team writes out the rules and thinks through some examples.

1. Required

1. Write out the rules dictated by this decision tree.

2. How would the decision tree classify the following loans?

a. Borrower has a credit score of 650 and income of $62,000

b. Borrower has a credit score of 670 and income of $75,500

c. Borrower has a credit score of 690 and income of $58,000

d. Borrower has a credit score of 725 and income of $62,000

3. Is Brown correct that something is wrong with the model? If so, what might explain this? Why is understanding the model important to Brown in his role as a management accountant?

1. 11-24. DECISION TREES. TelMark Mobile Services provides mobile phone services to millions of customers in the U.S. TelMark’s customer base is constantly changing as old customers leave the service and new customers join. This is called churn and it is a very important dynamic for the company to manage effectively because it has significant profit implications. Churn has increased recently, so managers at TelMark have started a program to offer targeted promotions to customers who are likely to abandon the service. They would like to develop a machine learning model to help identify these customers. Sylvia Restler is the management accountant assigned to work on the project with the data science team.
2. Sylvia speaks with a few members of the customer service team to better understand why customers leave their platform. The team tells her that two variables are highly related to customer churn: number of lines per customer family plan and number of months the customer has been with the company. According to their data, customers who have already been with the company for a long time tend to stay, as do customers with many lines on their family plan. Sylvia asks the data science team to use these variables to create a decision tree.

1. The team uses the following as a validation set:

1. Required

1. Write out the rules for this decision tree.

2. Using the full decision tree, classify each customer in the validation sample as leave or stay. Calculate the proportion of customers correctly classified as in Exhibit 11-13, column 5.

1. 11-25. DECISION TREES AND PRUNING (CONTINUATION OF 11-24). Assume the same information for TelMark as in Exercise 11-24. A member of the data science team points out that overfitting is often an issue with decision trees. To avoid this issue, he suggests pruning the tree at depth 3 so that the third cut is not made.

1. Required

1. Using the pruned tree, classify each customer in the validation sample as leave or stay (if the probability of leave is greater than 0.5, classify the customer as leave). Calculate the proportion of customers correctly classified as in Exhibit 11-13, column 6.

2. Based on your answer to requirement 1 of this problem and Exercise 11-24, requirement 2, which tree should Sylvia use to identify customers who are likely to leave?

3. As the management accountant, what business recommendations could Sylvia make to management based on the decision tree model?

1. 11-26. MAXIMUM LIKELIHOOD. Margo London is the management accountant at Norse Credit. Norse Credit spends a lot of time and resources trying to detect fraudulent activity within customers’ accounts. For most customers this is a low probability event. However, if it happens and Norse Credit does not detect it, it is very costly for the company.
2. London is working with the data science team to improve models for predicting fraudulent activity in customers’ accounts. The table below lists six observations in a model’s validation sample and the probability of default predicted by the (pruned) decision tree.

1. Required

1. Calculate the likelihood value for each observation in the validation set as in Exhibit 11-14, column 8 using the following equation $L=p^y\times {(1-p)}^{1-y}(\text{remember}\hspace{0.17em}{\mathrm{x}}^1=\mathrm{x}\hspace{0.17em}\text{and}\hspace{0.17em}{\mathrm{x}}^0=1)$.

2. Calculate the overall likelihood value for this set of predictions by multiplying the likelihood values for each observation together.

3. As the management accountant, would you use this model for decision making? Explain.

4. After becoming more familiar with the data, London’s team returns with a second model. The table below lists the six observations in the model’s validation sample and the probability of fraud predicted by the (pruned) decision tree. How does this model compare to the previous one? Should London be satisfied with the accuracy of the new model? Explain.

1. 11-27. RECEIVER OPERATING CHARACTERISTIC (ROC) CURVE. An ROC curve plots the false positive rate (x-axis) against the true positive rate (y-axis) for different model thresholds. The graph below shows three different ROC curves labeled 1, 2, and 3.

1. Required

1. Compare the curves labeled 1, 2, and 3 in the chart above. Which curve represents the model that would be most useful for a management accountant? Explain.

2. Explain what a “straight line” ROC curve actually represents.

1. 11-28. CONFUSION MATRICES, PAYOFF MATRIX, AND CHOOSING CUTOFF VALUES. Sun TV sells TV sets. It does not sell smart TVs so customers do not come to Sun TV if they want to purchase smart TVs. Sun TV wants to start selling smart TVs and will only sell smart TVs to customers to whom they advertise. Managers use customer information (income level, previous purchase history) to decide which customers they should target.
2. The team needs to decide how sure it must be in predicting customer interest in a smart TV. If it is too cautious, it will choose a very high cutoff probability and only market to customers who it believes are very likely to be in the market for a smart TV. This may cause them to miss out on many customers. If they are too aggressive and choose a low cutoff probability, they may identify more individuals interested in buying smart TVs but also end up wasting marketing dollars on customers who are not interested in purchasing smart TVs.
3. To choose a cutoff probability, the team develops the confusion matrices below for two cutoff probabilities on a validation sample of 1,000 households comprising 100 buyers and 900 non-buyers of smart TVs.

1. Required

1. Complete the confusion matrices for the validation set as in Exhibits 11-19 and 11-20.

2. A team of management accountants at SunTV estimates the payoffs from their actions. For every customer it targets, SunTV will spend $20 to market to that customer. For every smart TV it sells, SunTV makes a profit of $200 after taking into account the $20 it spends on that customer. Construct the payoff matrix as in Exhibit 11-21 and determine which cut off value SunTV should use.

3. Are there any other factors SunTV should consider before building such a model?

1. 11-29. MODEL THRESHOLDS AND PAYOFF MATRICES. Blanda Brothers is a produce processing company that specializes in fruits, with apple sales representing the majority of their revenue. Their main facility receives a daily shipment of apples, which are then sorted and shipped to grocery stores and to producers of apple jelly and juice, depending on the apple quality (sweetness, taste, color, etc.). The acceptable-quality apples are sold to grocery stores and low-quality apples are sold to apple processors. If a low-quality apple is sold to a grocery store, the apples are returned and Blanda must pay the stores a fee to compensate the store for lost sales and for the extra work of shipping the apples back. Blanda’s reputation also suffers, affecting future business. Blanda has an algorithm to determine apple quality, but the algorithm is not perfect and misclassifies apples.
2. The data science team presents their work to Cindy Hansen, the management accountant at Blanda, to help them choose between two cutoff prediction probabilities for low-quality apples of 0.50 and 0.30. Apples with a predicted probability above the cut-off probability are classified as low quality, and apples below the cut-off probability are classified as acceptable quality apples. The following confusion matrices are based on the validation sample.

1. Required

1. Cindy estimates that apples of acceptable quality result in a profit of $0.30 per apple. She also knows that low-quality apples can be sold to juice companies at a profit of $0.04. Finally, she estimates that the cost to Blanda of selling a low-quality apple to the grocery store as an acceptable-quality apple is $1.05 for each low-quality apple. Use this information to construct a payoff matrix as in Exhibit 11-21.

2. Using Cindy’s knowledge about the payoff of each outcome, which threshold should the team choose?

1. 11-30. MODEL THRESHOLDS AND PAYOFF MATRICES (CONTINUATION OF 11-29). Assume the same information for Blanda Brothers as in Problem 11-29. Due to increased competition among grocery stores, the cost to Blanda of selling a low-quality apple to the grocery store as an acceptable-quality apple is now $0.75 instead of $1.05 for each low-quality apple.

1. Required

1. Cindy realizes that the change in the cost of selling a low-quality apple to the grocery store as an acceptable-quality apple may change the optimal model cutoff. Re-evaluate the two cutoffs based on the new payoff matrix. How does this compare to the decision in Problem 11-29, requirement 2?

2. Why is it important for Cindy as the management accountant and the data science team to partner with each other?

Problems

1. 11-31. THINKING THROUGH THE DATA. James Silva is a management accountant at Keebler-Olson, where he is in charge of their investment portfolio. In 2015, James worked with a data scientist to develop a model that predicts how a given loan will perform in the future based on the characteristics of the borrower available on the peer-to-peer lending platform Mandel Credit. On April 1, 2016, he purchased $100,000 worth of loans with 36-month terms (3 years). His investments had performed well. James planned to invest another $100,000 on January 1, 2020. Looking ahead, he considers some strategic questions around the model.

1. Required

1. James wonders whether he should use new data to train a new model in December 2019, just prior to his planned investment on January 1, 2020. What should he do with the older data?

2. How much of the actual performance of the loans (such as repayments, loan restructuring, delayed repayments, hardship plans, etc.) should James use in classifying loans as default or repay when building his prediction model?

1. 11-32. DECISION TREES. Assume the same information for Keebler-Olson as in Problem 11-31. James Silva and the data scientist on his team work together to develop the following decision tree:

1. The data science team tests the model on the following validation set:

1. Required

1. Write out the rules for this full decision tree.

2. Classify each loan in the validation sample as repay or default. Calculate the proportion of loans correctly classified as in Exhibit 11-13, column 5.

3. Based on his experience in management accounting, James reviews the model and notes that something appears to be wrong with the tree at a depth of 3. What problem did James observe? What should James propose?

1. 11-33. DECISION TREES AND PRUNING (CONTINUATION OF 11-32). Assume the same information for Keebler-Olson as in Problems 11-31 and 11-32.

1. Required

1. Prune the tree at depth 3. Using the pruned tree, classify each loan in the validation sample as repay or default (if the probability of default is greater than 0.5 classify the loan as default). Calculate the proportion of loans correctly classified as in Exhibit 11-13, column 6.

2. Based on your answer to requirement 1 of this problem and Problem 11-32, requirement 2, which decision tree should James use to identify default and repay loans?

3. James has to present both models and the conclusions to the president of Keebler-Olson. He knows that in the past the president has preferred using models based on full decision trees because they seem to fit the training data more closely. How should James explain the pruned decision tree model?

1. 11-34. CALCULATE LIKELIHOOD VALUES (CONTINUATION OF 11-32 AND 11-33). Assume the same information for Keebler-Olson as in Problems 11-31, 11-32, and 11-33. While James is presenting his work to the president and his team, someone asks whether he had calculated the overall likelihood value for the trees. Wanting to make sure that he had fully evaluated the model, James agrees that this is a good idea.

1. Required

1. For the full and pruned trees, calculate likelihood values for the validation set in Problem 11-32 as in Exhibit 11-14, columns 6 and 8.

2. Calculate the overall likelihood value for the full and pruned trees.

3. As the management accountant, which model would you use for decision making based on the likelihood values? Does this differ from your answer to Problem 11-33, requirement 2? Explain.

1. 11-35. PAYOFF MATRIX AND CHOOSING CUTOFF VALUES (CONTINUATION OF 11-32, 11-33, AND 11-34). Assume the same information for Keebler-Olson as in Problems 11-31, 11-32, 11-33, and 11-34. James plans to invest $100 in each of the 1,000 loans he is examining. If he decides not to invest in a loan, he will invest the $100 in an alternative investment. As James and the data scientist work to improve their model, James mentions that different model predictions are associated with different payoffs for Keebler-Olson. Based on his management accounting experience, James estimates that the average payout for a repay loan of 3 years is $48 dollars. James also estimates that Keebler-Olson loses $60 over 3 years if it purchases a default loan. If Keebler-Olson does not invest in a loan, it places the money in an alternative investment with a payout of $14 over 3 years. James uses the pruned tree to evaluate the total payoff at a cutoff value of 0.25 and a cutoff value of 0.50.

1. Required

1. Fill in the payoff matrix below as in Exhibit 11-21.

2. James is deciding whether to use a cutoff of 0.25 or 0.5. Complete the confusion matrices below using the pruned tree for each cutoff as in Exhibits 11-19 and 11-20.

3. What cutoff value should James use? Explain.

1. 11-36. MAXIMUM LIKELIHOOD. Sarah Letourneau is the management accountant at Turcotte Manufacturing, an auto supplier that produces brake pads. Turcotte’s customers have high safety standards, and managers at Turcotte believe they can gain a competitive advantage by reducing the number of defective brake pads they produce. Sarah works with an external data science consulting company to develop a prediction model that identifies defective brake pads $(y=1)$ before they are shipped to Turcotte’s clients. After building the model on a training dataset, Sarah tests its accuracy on the validation set below.

1. Required

1. Calculate the likelihood value for each observation in the validation set using the following equation: $L=p^y\times {(1-p)}^{1-y}$ (remember $x^1=x$ and $x^0=1$)

2. Calculate the overall likelihood value for the validation set.

3. As the management accountant, what should Sarah do with this information?

1. 11-37. PAYOFF MATRIX AND CHOOSING CUTOFF VALUES (CONTINUATION OF 11-36). Assume the same information for Turcotte as in Problem 11-36. Working brake pads yield a profit of $25. If defective brake pads are identified before they are shipped, they can be fixed and sold as working brake pads. However, the process of inspecting and repairing brake pads that are predicted to be defective incurs costs. If a brake pad is predicted to be defective, it costs $8 to re-inspect it and, if found to be actually defective, it costs an additional $12 to repair. Turcotte incurs a net loss of $50 for defective brake pads shipped to customers because of additional costs (including reputation costs).

1. Required

1. Fill in the payoff matrix below as in Exhibit 11-21.

2. Sarah is debating two different model cutoffs of 0.50 and 0.30. Using the table below, fill in the confusion matrices as in Exhibits 11-19 and 11-20 based on the validation set presented in Problem 11-36.

3. What cutoff value should Sarah choose? Explain.

4. Sarah is uncertain about the $50 net loss if a defective brake pad is shipped to customers. At what loss would Sarah prefer to use the other cutoff value (i.e., the cutoff value Sarah did not choose in requirement 3)?

1. 11-38. RECEIVER OPERATING CHARACTERISTIC (ROC) CURVE. It is June, and the manager of the retail chain Stapleton wants to increase revenues during the back-to-school period. To do that, he plans to attract more customers in August whose children are entering school for the first time. By marketing to households who have kids entering school for the first time, Stapleton expects these households will purchase more products from Stapleton throughout the year. His question for the data science team is how to identify and target those households.
2. Caitlin Finch is the management accountant assigned to the project. The data science team has built a predictive model. The nodes of the decision tree result in four possible prediction probabilities (0.99, 0.55, 0.33, and 0.01). The validation sample of 1,000 records comprises 100 households with a child going to school for the first time and 900 households with no children going to school for the first time. The team prepares the following table, like the one in Exhibit 11-16. The table orders the predicted probabilities of households having kids entering school for the first time ranked from highest to lowest (as in Exhibit 11-16, column 2) based on the model, the cumulative number of households with no kids entering school for the first time (as in Exhibit 11-16, column 4), and the cumulative number of households with kids entering school for the first time (as in Exhibit 11-16, column 5). To ease exposition, we refer to households with kids entering school for the first time as “Households With Kids” or simply as “Kids” and households with no kids entering school for the first time as “Households With No Kids” or simply as “No Kids.” This means, for example, that at a cutoff probability of 0.50, the model would correctly predict 50 households with kids entering school for the first time but also incorrectly predict 135 households with kids entering school for the first time when those households do not have kids entering school for the first time.

1. Required

1. Calculate the false positive rate and true positive rate for the above table as in Exhibit 11-16, columns 6 and 7.

2. Draw the ROC curve. Would you recommend using this ROC curve to make predictions about households with kids entering school for the first time?

3. Use the preceding table to determine how many true positives and false positives the model identifies at a cutoff of 0.25 and 0.50. Fill in the confusion matrices below as in Exhibits 11-19 and 11-20.

4. For those households predicted to have kids going to school for the first time, Stapleton will spend $2 promoting products to each household. Stapleton expects to make $25 (after taking into account the $2 spent on promotion) from those households that it correctly identifies as having kids. Without doing promotion, Stapleton will earn an average of $10 from every household with a kid entering school for the first time. Using this information, create a payoff matrix as in Exhibit 11-21 and determine whether Caitlin should use a cutoff probability of 0.5 or 0.25.

5. When might a large increase in false positives be acceptable to Stapleton if the number of true positives also increases?

1. 11-39. MODEL THRESHOLDS AND PAYOFF MATRICES. David Porter is the management accountant at Spruce Bank, where the data science department is leading an initiative to predict whether loans will default or repay. The default rate in the training set is 15%. After building a model on the training set that predicts whether a loan will default or repay, the data scientist applies it to the validation set of 400 observations to evaluate its performance.

1. Required

1. Help the data scientist complete the confusion matrixes below for different model thresholds as in Exhibits 11-19 and 11-20.

2. Assume that Spruce Bank has $1,000 to invest in each loan of the validation sample. If Spruce Bank does not invest in a loan, it keeps the money in a risk-free investment at 3% a year for 3 years (ignore the time value of money). If Spruce invests in a loan that eventually repays, it receives 10% a year for 3 years. If Spruce invests in a loan that eventually defaults, Spruce loses 65% of the amount of the loan. Fill in the payoff matrix below as in Exhibit 11-21. Which model threshold should David and the data scientist use?

1. 11-40. MODEL THRESHOLDS AND PAYOFF MATRICES, SENSITIVITY ANALYSIS (CONTINUATION OF 11-39). Assume the same information for Spruce Bank as in Problem 11-39. David believes that within the pool of borrowers he invests in, the timing of when borrowers ultimately default could be different from what he has assumed. That means he might lose more or less than the 65% he has assumed in the payoff matrix. He decides to model “worst case” and “best case” scenarios of losing 75% and 55%, respectively, of the amount of the loan in the event of a default.

1. Required

1. Calculate the model payoffs for both the worst- and best-case scenarios for each cutoff.

2. If you were David, how would this analysis impact your decision making?