Data Exploration Exercise¶

Using whichever methods and libraries you prefer, create a notebook with the following:

1. Data preparation and Data exploration
2. Identify the three most significant data features which drive the credit risk
3. Modeling the credit risk
4. Model validation and evaluation using the methods that you find correct for the problem

Your solution should have instructions and be self-contained. For instance, If your choice is a python notebook, your notebook should install all the required dependencies to run it.

Import and preparation ¶

Data Exploration¶

In order to visually inspect the data it’s necessary to convert the datatype of the categorical features from string to category. This will also be necessary to train the model. Therefore the data will be formatted before further exploration.

Outcome distribution¶

Ideally there would be an approximately equal number of outcome classes. In this dataset the split is 30% “bad” and 70% “good”. The low number of bad credit risk assessments may limit the models ability to accurately predict a bad credit assessment relative to good assessments because of the limited training examples. This could result in more False Positives than would typically be expected.

Data formating¶

Visual inspection¶

The plot above compares each feature to each of the other features. The plots along the diagonal show the density plot of that feature, grouped by the value of the “class” feature that the model will predict.

Inspecting the pairplot shows that each of the features has approximately the same distribution for each class value.

Of the continuous features, Credit Amount is most heavily left skewed. Age is also left skewed.

If may improve the model to log transform credit_amount and age, particularly when predicting credit class when credit_amount or age values are in the middle of their respective ranges.

This could be done as follows:

df['log_credit_amount'] = np.log10(df['credit_amount'])

Check for multicoliniarity¶

Multicolinearity emerges when three or more model variables are highly correlated. Multicolinearity can emerge even when isolated pairs of variables are not colinear.

The Variance Inflation Factor (VIF) is a measure of colinearity among predictor variables. It is calculated by dividing the variance of all model betas by the variane of a single beta.

The results show that there is a significant amount of multicoliniarity in the dataset.

We are about to identify the features that drive the credit risk decision the most, this will reduce colinearity, random noise and resource requirements.

Once we know which subset of features contribute most to the predictive value of the model we could check for colinearity within that subset.

Identify the three most significant features which drive credit risk¶

Feature selection is important because it removes non-informative or redundant predictors from the model. This improves model performance and reduces computational resource requirements. Various statistical tests could be used to find the most significant features, and can be placed into two braod categories - supervised (using domain knowledge and considering the relationship of the feature to the target variable) and unsupervised (which ignores the feature’s relevance to the target variable).

The ANOVA F-value method is appropriate for numerical inputs and categorical outputs.
The Chi^2 test is appropriate for categorical inputs with categorical outputs.

Feature significance is evaluated using both methods:

The three most significant features are:

• Duration (ANOVA: 48, Chi2: 321)
• Checking Status (ANOVA: 40, Chi2: 36)
• Credit Amount (ANOVA: 24, Chi2: 58262)

The most significant feature is Duration, followed by Checking Status and Credit Amount. This makes sense intuitively, because Duration and Credit Amount must be proportional to risk. Checking Status is unfamiliar and I don’t know what it means, in a real scenario I would speak to stakeholders or team members to learn more about this feature.

Using an arbitrary but common VIF threshold value of 5, we see that none of the 3 most significant features are colinear. Hoever duration is close with a score of 4.46 and credit_amount is second with 3.84. Intuitively, it seems reasonable that credit_amount and duration might have some colinearity, as they could be generally expected to be inversely correlated to each other. - You could borrow more for a short period of time than you could for a longer duration.

Optimize the number of features to use¶

After an initial demonstration of the metrics used to assess model performance, the optimal number of features to use will be investigated.

Modeling Credit Risk¶

We use logistic regression because this is a binary classification problem with multiple predictor variables that are mostly independent.

Logistic Regression is a reasonable first choice of model type to use, but in a longer project we could use multiple model types and compare their performance against one another, then select the classifier which performs the best. Other types could include k-nearest neighbors, decision trees, or Support Vector Machines (SVMs).

Logistic regression makes the following assumptions:

1. No multicollinearity among the independent variables. Multicollinearity can be tested using the Variance Inflation Factor (VIF).
2. No influential outliers
3. Independence of errors (residuals) or no significant autocorrelation. The residuals should not be correlated with each other. This can be tested using the Durbin-Watson test, but this is out of scope for this exercize.
4. The sample size should be large (at least 50 observations per independent variables are recommended) -> 20 $\times$ 50 = 1000

Splitting the data into training, validation and testing sets¶

A validation set is used in addition to training and testing. This is because I expect to optimize model parameters by comparing the performance of different models using a dataset they weren’t trained on. If this were the same dataset that the final performance metric was based on then the model would be implicitely overfitted.

Instead, the performance of optimizations will be quantified using the validation set, and the optimal models performance will be quantified using the testing set.

This gives better prediction of actual performance against new production data (outside the available dataset) than only using training and testing data sets.

The training set has over twice as many positive values as negative values. This is similar to the complete data set which has 700 “good” credit class examples and 300 “bad” credit class examples.

Training the logistic regression model¶

The model is trained on the training dataset.

In this simple first case, we evaluate performance (discussed below) using the test data set. After an initial discussion of the results, the performance of different models is evaluated using a validation data set.

Model evaluation¶

A model could be evaluated in many different ways, each with different strengths and weaknesses depending on the characteristics of the data and the relevant domains.

Where possible, its preferable to use only a single metric to quantify and compare model performance, because its simpler (and less error prone) than if there are multiple metrics to consider.

There are 4 basic categories of results that the model could produce:

1. True Positives (TP)
2. False Positives (FP)
3. True Negatives (TN)
4. False Negatives (FN)

These can be combined to calculate the model’s Precision and Recall:

• Optimizing Precision minimizes false positives but ignores false negatives.
• Optimizing Recall minimizes false negatives but ignores false positives.

P = True Positives / (True Positives + False Positives) -> if False Positives are 0, P=1
R = True Positives / (True Positives + False Negatives) -> if False Negatives are 0, R=1

Precision is a ratio of the number of true positives divided by the sum of the true positives and false positives. It describes how good a model is at predicting the positive class. It is the ability of the classifier not to label as positive a sample that is negative.

High precision implies fewer false positives.

Precision = True Positives / (True Positives + False Positives)

Recall is calculated as the ratio of the number of true positives divided by the sum of the true positives and the false negatives. It quanitifies the ability of the model to find all the positive samples.

High recall implies fewer false negatives.

Recall = True Positives / (True Positives + False Negatives)

An ideal model will have good precision and recall. The F-Score provides a way to combine precision and recall into a single measure that captures both properties.

F-Measure = (2 * Precision * Recall) / (Precision + Recall)

The influence of precision and recall relative to each other can be changed by adding a coefficient into the F-measure. Further discussion of this is outside the scope of this brief exercise except to say that F1 (above) has beta=1 and places equal weight on precision and recall. F2 (beta=2) places less weight on precision and more on recall, and F0.5 emphasizes precision over recall.

Other metrics to consider include Precision Recall curves and the Receiver Operating Characteristic curve (ROC).

ROC curves are more informative when evaluating datasets with equal proportion of positive and negative results. In a skewed dataset such as this, ROC can be overly optimistic and Precision-Recall curves are more reliable.

Therefore after briefly demonstrating the confusion matrix we consider the average precision and recall instead of comparing the area under the ROC curve for different models.

Summary of results with the 3 most significant features¶

Confusion Matrix¶

The confusion matrix simply shows the number of True Positives, True Negatives, False Positives and False Negatives that the model predicted.

These results are based on the testing data set. This model was trained on the training dataset.

Precision-Recall Curve¶

Reviewing both precision and recall is useful in cases where there is an imbalance in the observations between the two classes. This is because the large number of class 1 (“good”) examples means we are less interested in the skill of the model at predicting this correctly, e.g. high true positives, and instead are more concerned with how the model can predict true negatives (where credit class is “bad”). The confusion matrix above shows that the largest error group is False Positives, which is expected because the model has relatively few examples of negative examples compared to positive. This is why the precision is lower than the recall.

A no-skill classifier is one that cannot discriminate between the classes and would predict a random or constant class in all cases. This creates a minimum precision called the “no-skill” line. Its value is the ratio of positive cases in the dataset. For our training data set it is 0.687. (Note it is not 0.7, due to the random selection of the training set.)

The Precision Recall metric is a useful measure of success of prediction when the classes are very imbalanced, as they in this case. A precision-recall curve focuses on the performance of a classifier on the positive class.

The precision-recall curve shows the tradeoff between precision and recall for different probability thresholds, which we refer to but do not investigate in detail in this limited exercise.

The threshold is the probability at which a “good” or “bad” label is applied. The default threshold is 0.5, which means that if the model calculates the probability of an observation being “good” as more than 0.5 then it applies the label “good”, otherwise it applies the label “bad”. This threshold probability can be varied and the impact on model performance quantified.

A high area under the curve represents both high recall and high precision. High scores for both show that the classifier is returning accurate results (high precision), as well as returning a majority of all positive results (high recall).

A system with high recall but low precision returns many positive results, but most of its predicted labels are incorrect when compared to the training labels.

A system with high precision but low recall is the opposite, returning very few positive results but most of them are correct. An ideal system with high precision and high recall will identify almost all positive cases, with all results labeled correctly.

AP summarizes a precision-recall curve as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight.

Summary¶

Using the 3 most signficant features we have the following results:

F1: 0.845 Precision-Recall AUC: 0.87

The model could be improved by finding the optimal number of features to consider.

Improving the model by optimizing the number of features¶

The code below iterates through the available features from most significant to least, adding an additional factor in each loop.

Each loop generates a new logistic regression model and trains it on the same training data set. Its performance is evalulated using the same validation data set. The results are summarized in the “results” dataframe and discussed below.

Due to the very small amount of data, changing the random_state value can produce results with different trends. Therefore the first recommendation to improve the model is to get more data. This is also discussed below.

Optimization Results¶

The results show that the highest F1 score (which places equal emphasis on Precision and Recall) of 0.859 is obtained using a logistic regression model that uses the three most significant features (Credit Amount, Checking Status and Duration).

Due to the small data sets (700 examples in the training set, 150 in each of the validation and testing sets) the results can vary significantly when a different random number seed is used to generate the 3 datasets.

Whilst the table above shows that AUC increases with number of features, the increase is small (0.9% from 5 to 10 features) and sensitive to the random number seeds - if a different seed is used, the same trend is not observed. Therefore, whilst Recall-Precision curves are a useful tool in imbalanced datasets, the small size of the dataset being used in this analysis creates variation that wouldnt be observed in larger datasets.

Model performance using Test Data Set¶

We test the performance of the model chosen in the optimization stage. In this stage we use the testing dataset, because this provides a more reliable indication of how the model might perform with (new) production data compared to using the results from the validation dataset.

Recommendations¶

1. Obtain more training data

   This will reduce the influence of randomness on the model scores.

2. Use a credit score instead of just a “good” or “bad” decision.

   The “good” or “bad” status is probably the result of a threshold being applied to a metric. Accessing the underlying metric would provide more detail and help train a more reliable model.

3. Consider additional features

   This would require domain expertise and additional context.