Working With PeerLend Digital Data

Having understood the tools and methods data scientists use and their strengths and weaknesses, Paige is ready to train a model from the full PD dataset and construct a payoff matrix to quantify the value of its predictions on the new loans PD offers in Q1 2020. Paige’s team returns to the PD dataset described in Step 2: Obtain and Explore Relevant Data. There are 500,000 funded loans in the dataset, of which 100,000 are loans that default $(\text{loan\hspace{0.17em}status}=1)$ and 400,000 are loans that repay $(\text{loan\hspace{0.17em}status}=0)$. The overall default rate is .

Per Step 3: Prepare the Data, the team cleans the data to eliminate any features that would contribute to target leakage. It then separates the dataset into a training, validation, and holdout samples. The holdout sample is 20% of the total set $(\text{500,000}\times 0.20=\text{100,000})$, the training set is 60% of the total set $(\text{500,000}\times 0.60=\text{300,000})$, and the cross-validation set is 20% of the total set $(\text{500,000}\times 0.20=\text{100,000})$.

In Step 4: Build a Model, the team trains its models on the sample of 300,000 loans and cross-validates them with a sample of 100,000. It chooses the pruned decision tree that maximizes the likelihood value in the validation sample.^* The team then runs the model on the holdout sample and obtains a similar likelihood value.

Per Step 5: Evaluate the Model, Paige and the team consider some important metrics to understand how much they can rely on the model to give accurate predictions on a set of new loan data. The first metric, the model’s likelihood value computed on the validation set, is very good: it is 0.85.^* The model will provide reasonably accurate predictions. The second set of metrics is feature impact. Exhibit 11-22 presents the feature impacts. The feature impact is a value (0 to 100) assigned to each feature variable in the training set that indicates how important it is in partitioning the data to minimize Gini impurity, that is, how frequently a feature is used in making cuts at various nodes of the decision-tree model. The variable that is used most often (grade) is normalized to a value of 100. The next most important variable in making cuts is credit score with a value of 18, indicating that it is used 18% as frequently as grade is to make a cut.

Not surprisingly, the grade assigned to the loan by PeerLend Digital was by far the most important feature for distinguishing default loans from repay loans. Grades A and B had a high number of repay loans, while Grades E and F had many more default loans. It suggested, as Paige had expected, that PeerLend Digital was doing a good job of identifying default risk. Interestingly, however, credit score, debt-to-income, annual income, and loan amount were also relevant features (though much less important than grade) for separating repay loans from default loans in the decision-tree model. This meant that there was incremental information content in these variables for identifying default loans over and above what was available in the grade feature. These variables might help to identify more of the default loans (true positives) without classifying too many repay loans as default loans (false positives). If they could do so, Sierra would not invest in loans predicted to default and earn a return greater than what it would have earned by investing in all PD loans.

Paige is surprised by some features not being relevant to identifying default loans. For example, she had anticipated that homeownership would have a higher impact; she hypothesized that loan applicants who owned homes tended to have more stable incomes, savings, and lives, which were all characteristics that would make them more likely to repay a small loan. Instead, the model showed this feature had no discernable impact. Nevertheless, Paige is comfortable with the features used by the model to identify default loans. They make intuitive sense. She is ready to evaluate how well the model performs in classifying default and repay loans.

Paige turns to the ROC curve. Exhibit 11-23 presents the ROC curve. It moves nicely up along the y-axis far away from the diagonal line joining the origin (0, 0) to the upper-right corner (1, 1), indicating that the model does well in partitioning defaults from repays. The shape of the curve suggests that the model predictions will be useful for predicting defaults without the risk of too many false positives and false negatives.

Paige explores several confusion matrixes using different default probability cutoffs. Exhibit 11-24 presents the contents of the confusion matrixes vertically for easy comparison of the distribution of predictions across six selected default probability cutoffs for the validation sample of 100,000 loans. Sierra plans to invest $100 in each of the 100,000 loans; if Sierra does not invest in a PeerDigital loan, Sierra will invest the $100 earmarked for that loan in the alternate investment. Recall that Paige rejects all loans above the default probability cutoff (classified as positive with a high predicted probability of default) and accepts all loans below the default probability cutoff (classified as negative with a low predicted probability of default). For example, at the 0.35 cutoff, Sierra rejects all loans with a predicted default probability greater than 0.35—the true positives of 3,942 and the false positives of 5,072 for a total of $\text{9,014}\  (\text{3,942}+\text{5,072})$ loans—and invests in all loans with a predicted default probability below 0.35—the true negatives of 76,959 and the false negatives of 14,027 for a total of $\text{90,986}\  (\text{76,959}+\text{14,027})$ loans.

Consistent with the simple example, a higher default probability cutoff means that Sierra is rejecting fewer loans and accepting more loans. Therefore, as the default probability cutoff increases, Sierra rejects fewer default loans (true positives) and also has fewer repay loans misclassified as defaults (false positives). At a higher default probability cutoff, Sierra accepts more loans that repay (true negatives) but also has more default loans misclassified as repay (false negatives).

Paige uses values from her payoff matrix to determine which default probability cutoff is the most profitable investment. Recall that Paige assigns $45 to each true negative (repay loans) and $-\mathrm{\$}70$ to a false negative (default loan). She uses two values for the alternative investment opportunities, $6 over 3 years and $12 over 3 years, to model a more-conservative and a less-conservative payoff scenario. Exhibit 11-25 presents these two payoff scenarios and the cumulative and annual return on investment (ROI) on the $10,000,000 investment. Consider the payoff column for the 0.35 default probability cutoff and the alternate investment payoff of $6 over 3 years.

On the 76,959 true negative loans, Sierra earns $\$\text{3,463,155}\hspace{0.17em} (76,959\hspace{0.17em}\text{loans}\times \$45)$. On the 14,027 false negative loans, Sierra loses $\$\text{981,890}\hspace{0.17em} (\text{14,027}\times (-\$70))$. Sierra does not invest in the 3,942 loans it classifies as true positives. Instead it invests the available funds $\$\text{394,200}\hspace{0.17em} (\text{3,942}\hspace{0.17em}\text{loans}\times \$100\hspace{0.17em}\text{per loan})$ in the safe bond to earn 6% over 3 years equal to $\$\text{23,652\hspace{0.05em}}(6\%\times \$\text{394,200})$. Sierra also does not invest in the 5,072 loans it classifies as false positives and instead invests $507,200 in the safe bond to earn $\$\text{30,432} \left(6\%\times \$\text{507,200}\right)$. The 0.35 default probability cutoff yields the highest ROI of $\$\text{2,535,349}÷\$\text{10,000,000}=25.35\mathrm{\%}$ over 3 years or $8.45\mathrm{\%} (25.35\%÷3)$ per year.^*

From a technical perspective, Paige is pleased with the analysis. In the $6 alternative-payout scenario, the optimal default probability cutoff of 0.35 yields an annual return of 8.45%, which is 1% greater than what Sierra would have earned by investing in all 100,000 PD loans. Twenty percent of the loans would have defaulted and 80% would have repaid, for a payout of $\text{20,000}\times \$(-70)+\text{80,000}\times \$45=-\$\text{1,400,000}+\$\text{3,600,000}=\$\text{2,200,000}$. The return over 3 years, based on the $10,000,000 investment, would be $\$\text{2,200,000}÷\$\text{10,000,000}=22\mathrm{\%}\hspace{0.17em}\mathrm{or}\hspace{0.17em}22\mathrm{\%}÷3=7.33\mathrm{\%}$ per year. Data science allows her to make better investments on behalf of clients to increase their returns and increase the firm’s profit.