Credit Scoring, Response Modelling and Insurance Rating: A Practical Guide to Forecasting Consumer Behaviour Book

List of Tables xii

List of Figures xiii

Acknowledgements xiv

1 Introduction 1

1.1 Scope and content 3

1.2 Model applications 5

1.3 The nature and form of consumer behaviour models 8

1.3.1 Linear models 9

1.3.2 Classification and regression trees (CART) 12

1.3.3 Artificial neural networks 14

1.4 Model construction 18

1.5 Measures of performance 20

1.6 The stages of a model development project 22

1.7 Chapter summary 28

2 Project Planning 30

2.1 Roles and responsibilities 32

2.2 Business objectives and project scope 34

2.2.1 Project scope 36

2.2.2 Cheap, quick or optimal? 38

2.3 Modelling objectives 39

2.3.1 Modelling objectives for classification models 40

2.3.2 Roll rate analysis 43

2.3.3 Profit based good/bad definitions 45

2.3.4 Continuous modelling objectives 46

2.3.5 Product level or customer level forecasting? 48

2.4 Forecast horizon (outcome period) 50

2.4.1 Bad rate (emergence) curves 52

2.4.2 Revenue/loss/value curves 53

2.5 Legal and ethical issues 54

2.6 Data sources and predictor variables 55

2.7 Resource planning 58

2.7.1 Costs 59

2.7.2 Project plan 59

2.8 Risks and issues 61

2.9 Documentation and reporting 62

2.9.1 Project requirements document 62

2.9.2 Interim documentation 63

2.9.3 Final project documentation (documentation manual) 64

2.10 Chapter summary 64

3 Sample Selection 66

3.1 Sample window (sample period) 66

3.2 Sample size 68

3.2.1 Stratified random sampling 70

3.2.2 Adaptive sampling 71

3.3 Development and holdout samples 73

3.4 Out-of-time and recent samples 73

3.5 Multi-segment (sub-population) sampling 75

3.6 Balancing 77

3.7 Non-performance 83

3.8 Exclusions 83

3.9 Population flow (waterfall) diagram 85

3.10 Chapter summary 87

4 Gathering and Preparing Data 89

4.1 Gathering data 90

4.1.1 Mismatches 95

4.1.2 Sample first or gather first? 97

4.1.3 Basic data checks 98

4.2 Cleaning and preparing data 102

4.2.1 Dealing with missing, corrupt and invalid data 102

4.2.2 Creating derived variables 104

4.2.3 Outliers 106

4.2.4 Inconsistent coding schema 106

4.2.5 Coding of the dependent variable (modelling objective) 107

4.2.6 The final data set 108

4.3 Familiarization with the data 110

4.4 Chapter summary 111

5 Understanding Relationships in Data 113

5.1 Fine classed univariate (characteristic) analysis 114

5.2 Measures of association 123

5.2.1 Information value 123

5.2.2 Chi-squared statistic 125

5.2.3 Efficiency (GINI coefficient) 125

5.2.4 Correlation 126

5.3 Alternative methods for classing interval variables 129

5.3.1 Automated segmentation procedures 129

5.3.2 The application of expert opinion to interval definitions 129

5.4 Correlation between predictor variables 131

5.5 Interaction variables 134

5.6 Preliminary variable selection 138

5.7 Chapter summary 142

6 Data Pre-processing 144

6.1 Dummy variable transformed variables 145

6.2 Weights of evidence transformed variables 146

6.3 Coarse classing 146

6.3.1 Coarse classing categorical variables 148

6.3.2 Coarse classing ordinal and interval variables 150

6.3.3 How many coarse classed intervals should there be? 153

6.3.4 Balancing issues 154

6.3.5 Pre-processing holdout, out-of-time and recent samples 154

6.4 Which is best - weight of evidence or dummy variables? 155

6.4.1 Linear models 155

6.4.2 CART and neural network models 158

6.5 Chapter summary 159

7 Model Construction (Parameter Estimation) 160

7.1 Linear regression 162

7.1.1 Linear regression for regression 162

7.1.2 Linear regression for classification 164

7.1.3 Stepwise linear regression 165

7.1.4 Model generation 166

7.1.5 Interpreting the output of the modelling process 168

7.1.6 Measures of model fit 170

7.1.7 Are the assumptions for linear regression important? 173

7.1.8 Stakeholder expectations and business requirements 174

7.2 Logistic regression 175

7.2.1 Producing the model 177

7.2.2 Interpreting the output 177

7.3 Neural network models 180

7.3.1 Number of neurons in the hidden layer 181

7.3.2 Objective function 182

7.3.3 Combination and activation function 182

7.3.4 Training algorithm 183

7.3.5 Stopping criteria and model selection 184

7.4 Classification and regression trees (CART) 184

7.4.1 Growing and pruning the tree 185

7.5 Survival analysis 186

7.6 Computation issues 188

7.7 Calibration 190

7.8 Presenting linear models as scorecards 192

7.9 The prospects of further advances in model construction techniques 193

7.10 Chapter summary 195

8 Validation, Model Performance and Cut-off Strategy 197

8.1 Preparing for validation 198

8.2 Preliminary validation 201

8.2.1 Comparison of development and holdout samples 202

8.2.2 Score alignment 203

8.2.3 Attribute alignment 207

8.2.4 What if a model fails to validate? 210

8.3 Generic measures of performance 211

8.3.1 Percentage correctly classified (PCC) 212

8.3.2 ROC curves and the GINI coefficient 213

8.3.3 KS-statistic 216

8.3.4 Out-of time sample validation 216

8.4 Business measures of performance 218

8.4.1 Marginal odds based cut-off with average revenue/loss figures 219

8.4.2 Constraint based cut-offs 220

8.4.3 What-if analysis 221

8.4.4 Swap set analysis 222

8.5 Presenting models to stakeholders 223

8.6 Chapter summary 224

9 Sample Bias and Reject Inference 226

9.1 Data methods 231

9.1.1 Reject acceptance 231

9.1.2 Data surrogacy 232

9.2 Inference methods 235

9.2.1 Augmentation 235

9.2.2 Extrapolation 236

9.2.3 Iterative reclassification 240

9.3 Does reject inference work? 241

9.4 Chapter summary 242

10 Implementation and Monitoring 244

10.1 Implementation 245

10.1.1 Implementation platform 245

10.1.2 Scoring (coding) instructions 246

10.1.3 Test plan 247

10.1.4 Post implementation checks 248

10.2 Monitoring 248

10.2.1 Model performance 249

10.2.2 Policy rules and override analysis 250

10.2.3 Monitoring cases that would previously have been rejected 252

10.2.4 Portfolio monitoring 252

10.3 Chapter summary 253

11 Further Topics 254

11.1 Model development and evaluation with small samples 254

11.1.1 Leave-one-out cross validation 255

11.1.2 Bootstrapping 255

11.2 Multi-sample evaluation procedures for large populations 256

11.2.1 k-fold cross validation 256

11.2.2 kj-fold cross validation 257

11.3 Multi-model (fusion) systems 257

11.3.1 Static parallel systems 258

11.3.2 Multi-stage models 259

11.3.3 Dynamic model selection 262

11.4 Chapter summary 262

Notes 264

Bibliography 273

Index 278