In this step, the management accountant determines how to organize and process the data. What additional data might be needed? How should different variables be measured, and what variables should be excluded? For example, Paige must probe how PD measures “annual income.” As the management accountant, she may draw different conclusions about the value of this measure depending on whether it refers to an average annual income over the past 5 years (indicating some stability of annual income) versus income from just the past year. To clarify this definition, she may have to refer to PD’s data dictionary or find more detailed documentation of the collection process. This is an example of the importance of substantive expertise and management accounting knowledge in doing data science, as shown in Exhibit 11-1.
Paige notices that the PD data contain “past” and “present” data as well as loan performance data. These features are shown in Exhibit 11-3 under the column “Phase 2.” PD collects these data after the loan is funded and the borrower has begun to repay the loan. PD does not have these data at the time it approves a borrower and places the loan on the platform. Neither does Sierra Investments. From the point of view of the model, these data are from the future. The model Sierra Investments builds with these performance data will perform much better than a model it builds without it, because these data “from the future” add significant additional signals about loan performance. But these are data Sierra’s managers must not use in building the model because the data are not available at the time of making the investment decision. If Sierra’s managers did include future data, it would be an example of target leakage. Data scientists use the term target leakage to refer to data that are not available at the time of the analysis and that therefore should be excluded. Paige must use substantive expertise to determine which features might contribute to target leakage. In preparing the data, the management accountant must advise the data science team that these features cannot be used for training the predictive models.
To build a model to predict defaults, Sierra Investments downloads the dataset of 500,000 records for the period Q1 2015 to Q4 2016 from the PD website. All these loans are scheduled to come due between Q1 2018 and Q4 2019. Sierra Investments plans to use this model to predict which of the new loans PD offers in Q1 2020 it should avoid because these loans are likely to default. Paige wonders whether she should obtain data from even earlier periods such as from 2005 onward. If she did, she would have access to more data to estimate the model. The core of data science is using data from the past to predict the future. The judgment she must make is whether the older data are representative of the economic conditions in 2020. Paige is concerned that the older data come from a period spanning the global financial crisis. The circumstances of that time differ from what she expects in 2020. She is much more comfortable using the most recent data to make predictions. She also believes that 500,000 records are more than enough to run her models.
Before releasing the historic loan data, PD scrubs the data of any information that would allow an investor to discover the individual identities of any borrowers. Knowledge of these identities might help the data science team make better-informed decisions. This would be particularly true if the team could integrate additional data about borrowers into its dataset. Such data might include preferences shared over social media, criminal records, driving records, etc. Any data that could indicate a leaning toward risky decision making and behavior might provide an additional “signal.”
Paige might be tempted to seek out such data. However, personal data privacy has become a highly contested issue in the last several years and the legal landscape is rapidly changing. Before acting, she would have to research what activity for collecting data was legal. For example, data scientists could “scrape” data from websites, but they should only do so under the terms of the website. Some websites allow scraping of data; others limit this activity. In addition, she would need to think carefully about what activity was, and was not, ethical from the perspective of the firm. For example, if she could buy third-party data on the borrowers at PD, should she? Or could data on individual borrowers be discriminatory if information on race, religion, gender, age, or marital status is included?