End of Chapter: Data Analytic Thinking and Prediction

Problem for Self-Study

It is June, and the manager of the retail chain PriceTrimmer wants to increase revenues during the back-to-school period. He plans to identify parents in early August whose children are entering school for the first time in September. His question for the management accountant and the data science team is how to identify and target these households whose children will need to purchase school supplies.

PriceTrimmer’s data science team extracts prior-year data on 10,000 households from its customer database and trains a new model. The team partitions the sample into three subsamples: a training sample of 6,000 observations, a validation sample of 2,000 observations, and a holdout sample of 2,000 observations. The rate of households with a child entering school for the first time in each subsample is 18%.

Required

Is this an interesting problem for PriceTrimmer to solve? Why?
What data might be available to help solve this problem?
Complete the confusion matrices below for the validation sample.

The exhibit depicts a confusion matrix at cutoff value of 0.5. The details of the table are as follows:

Actual outcome of first-time kids and predicted outcome of first-time kids: True positives, 100
Actual outcome of first-time kids and predicted outcome of no first-time kids: False negatives, number unknown
Total actual outcomes of first-time kids: 360
Actual outcome of no first-time kids and predicted outcome of first-time kids: False positives, number unknown
Actual outcome of no first-time kids and predicted outcome of no first-time kids: True negatives, 1,440
Total actual outcomes of no first-time kids: 1,640
Total predicted outcomes of first-time kids: 300
Total predicted outcomes of no first-time kids: 1,700
Total: 2,000

The exhibit depicts a confusion matrix at cutoff value of 0.3. The details of the table are as follows:

Actual outcome of first-time kids and predicted outcome of first-time kids: True positives, number unknown
Actual outcome of first-time kids and predicted outcome of no first-time kids: False negatives, number unknown
Total actual outcomes of first-time kids: 360
Actual outcome of no first-time kids and predicted outcome of first-time kids: False positives, 900
Actual outcome of no first-time kids and predicted outcome of no first-time kids: True negatives, number unknown
Total actual outcomes of no first-time kids: 1,640
Total predicted outcomes of first-time kids: 1,210
Total predicted outcomes of no first-time kids: Number unknown
Total: 2,000

As the management accountant you have determined the following. Without doing any promotion, PriceTrimmer earns an average of $20 from every family sending a child to school for the first time. If it decides to promote the program, it will cost PriceTrimmer $10 per child in the form of mailers and promotions sent to households that are predicted to have a child entering school. This money would yield no return if the family was not sending a child to school for the first time. If, however, the family was sending a child to school for the first time, PriceTrimmer would earn $50 (net of the cost of the promotion) instead of $20. Develop a payoff matrix for this situation.
Which cutoff probability of first-time kids would you select: (a) 0.3 or (b) 0.5? Comment briefly on the results.

Solution

1. This is an interesting problem for PriceTrimmer to solve because households with children who are going to school for the first time are more likely to purchase many more school supplies than households with children who have already been to school and have many of the necessary supplies. The first-time households are looking for a store that can help them with all that they need both now and later in the year. If PriceTrimmer can accurately identify these households and their needs and promote their products to the households just at the time they are looking for these materials, it is a good opportunity to increase revenues.
2. PriceTrimmer may already have these data for households with loyalty cards. Some loyalty programs ask for data on all household members. For those who have not provided such explicit data, PriceTrimmer may be able to identify these households based on their past purchasing behavior. For example, there may be a progression of items a household buys for a child who grows from age 3, to 4, and then 5 when he or she enters kindergarten and needs a backpack, pencil case, and lunchbox. If PriceTrimmer can identify this pattern for past customers, it could build a model to predict those households that will behave similarly in the future.
3. The rate of households with children attending school for the first time is 18%, so the number of instances of this target feature in the validation set is $360 (2, 000 \times 0.18 = 360)$ .

The exhibit depicts a confusion matrix at cutoff value of 0.5. The details of the table are as follows:

Actual outcome of first-time kids and predicted outcome of first-time kids: True positives, 100
Actual outcome of first-time kids and predicted outcome of no first-time kids: False negatives, 360
Total actual outcomes of first-time kids: 360
Actual outcome of no first-time kids and predicted outcome of first-time kids: False positives, 200
Actual outcome of no first-time kids and predicted outcome of no first-time kids: True negatives, 1,440
Total actual outcomes of no first-time kids: 1,640
Total predicted outcomes of first-time kids: 300
Total predicted outcomes of no first-time kids: 1,700
Total: 2,000

The exhibit depicts a confusion matrix at cutoff value of 0.3. The details of the table are as follows:

Actual outcome of first-time kids and predicted outcome of first-time kids: True positives, 310
Actual outcome of first-time kids and predicted outcome of no first-time kids: False negatives, 50
Total actual outcomes of first-time kids: 360
Actual outcome of no first-time kids and predicted outcome of first-time kids: False positives, 900
Actual outcome of no first-time kids and predicted outcome of no first-time kids: True negatives, 740
Total actual outcomes of no first-time kids: 1,640
Total predicted outcomes of first-time kids: 1,210
Total predicted outcomes of no first-time kids: 790
Total: 2,000

4. The payoff matrix follows.

The exhibit depicts a payoff matrix. The details of the table are as follows:

Actual outcome of first-time kids and predicted outcome of first-time kids: True positives, 50 dollars
Actual outcome of first-time kids and predicted outcome of no first-time kids: False negatives, 20 dollars
Actual outcome of no first-time kids and predicted outcome of first-time kids: False positives, negative 10 dollars
Actual outcome of no first-time kids and predicted outcome of no first-time kids: True negatives, 0 dollars

Predicted First-Time Kids
1. • TRUE POSITIVES (TP): These are households PriceTrimmer correctly predicts have a child going to school for the first time (true positives). The store sends them a promotion believing they are likely to buy many supplies, which on average will yield $50.
2. • FALSE POSITIVES (FP): These are households PriceTrimmer predicts to have a child going to school for the first time, but actually, they do not. The $begin inline style negative text $ end text 10 end style$ represents the marketing cost of reaching out to these customers with no benefit.
Predicted No First-Time Kids
1. • TRUE NEGATIVES (TN): These are households the model correctly predicts do not have a child going to school for the first time. PriceTrimmer does not target them and they generate $0 incremental cash since these customers are not in the market for first-time supplies.
2. • FALSE NEGATIVES (FN): These are households PriceTrimmer predicts to not have a child going to school for the first time, but actually they do. The $20 represents the additional revenue these households spend at PriceTrimmer based on what they would buy without any targeted promotion. It is much less than the payoff from households PriceTrimmer correctly predicts had a child going to school for the first time because they do not receive a promotion and shop at competitor stores for some of the other items.

5. To calculate the payoff for the model at a particular cutoff probability of first-time kids, multiply the payoff matrix by the confusion matrix and add the products from each quadrant together.
At the 0.3 cutoff,

$\begin{array}{r} \begin{aligned} \begin{aligned} \begin{aligned} Payoff = ($ 50 \times 310) + ($ 20 \times 50) & + & (- $ 10 \times 900) + ($ 0 \times 740) \\ = $ 15,500 + $ 1,000 - $ 9,000 + 0 = $ 7,500 \end{aligned} \end{aligned} \end{aligned} \end{array}$

At the 0.5 cutoff,

$\begin{array}{rcl} Payoff = ($ 50 \times 100) + ($ 20 \times 260) & + & (- $ 10 \times 200) + ($ 0 \times 1,440) \\ = $ 5,000 + $ 5,200 - $ 2,000 + 0 = $ 8,200 \end{array}$

The management accountant will select the cutoff of 0.5 because the payoff of $8,200 is higher than the payoff for the cutoff at 0.3, which is $7,500.
The key insight is that by lowering the cutoff probability of first-time kids to 0.3, PriceTrimmer would market its products to many more customers—from 300 to 1,210. Of these, 310 are true first-time households, an increase of 210 from the 100 PriceTrimmer reached by setting the cutoff probability of first-time kids at 0.5. These 210 kids increase PriceTrimmer’s cash flow by $$ 6,300 (210 \times ($ 50 - $ 20))$ but the false positives increase by 700, from 200 to 900, resulting in additional marketing expenses with no benefit of $7,000. The net effect is a decrease in cash flow of $begin inline style $ 700 thin space left parenthesis $ text 6,300 end text minus $ text 7,000 end text right parenthesis end style$ . Lowering the cutoff probability of first-time kids would be better only if the cost of marketing and promotion were significantly lower or if the benefits from targeting were significantly higher.