{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## <center><font color=navy>Big Data Economics</font></center>\n",
"### <center>Logistic Regression: an essential BD tool</center>\n",
"#### <center>Ali Habibnia</center>\n",
" \n",
"<center> Assistant Professor, Department of Economics, </center>\n",
"<center> and Division of Computational Modeling & Data Analytics at Virginia Tech</center>\n",
"<center> habibnia@vt.edu </center> "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-block alert-info\">\n",
"\n",
"Recruiters in industry expect you to know at least two algorithms: Linear Regression and Logistic Regression. Due to their ease of interpretation, consultancy firms use these algorithms extensively.\n",
"\n",
"<img src=\"images/funny.png\" width=\"600\">\n",
"</div>\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Readings:\n",
"\n",
"- https://www.khanacademy.org for basic math and stats.\n",
"1. ***Chapter 4.4*** [The Elements of Statistical Learning: Data Mining, Inference, and Prediction](https://web.stanford.edu/~hastie/ElemStatLearn/printings/ESLII_print12.pdf). \n",
"2. For a quick review see: ***Chapter 9.3***, Understanding Machine Learning From Theory to Algorithms."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Overview\n",
"\n",
"Many a time, situations arise where the dependent variable isn't normally distributed; i.e., the assumption of normality is violated. For example, think of a problem when the dependent variable is binary. Will you still use Multiple Regression? Of course not!\n",
"\n",
"Often we have to resolve questions with binary or yes/no outcomes.\n",
"\n",
"For example:\n",
"\n",
"* _Does a patient have cancer?_\n",
"\n",
"* _Will a team win the next game?_\n",
"\n",
"* _Will the customer buy my product?_\n",
"\n",
"* _Will I get the loan?_\n",
"\n",
"* Can we get a loan, from the Lending Club, of 10,000 dollars at 12 per cent or less, with a FICO Score of 720?\n",
"\n",
"> Logistic regression is used to predict the outcome of a categorical variable. A categorical variable is a variable that can take only specific and limited values.\n",
"\n",
"\n",
"In high dimensions, it is often convenient to think binary.\n",
"\n",
"\n",
"\n",
"#### Types of Logistic Regression\n",
"\n",
"* Binary Logistic Regression: The target variable has only two possible outcomes such as Spam or Not Spam, Cancer or No Cancer.\n",
"* Multinomial Logistic Regression: The target variable has three or more nominal categories such as predicting the type of Wine.\n",
"* Ordinal Logistic Regression: the target variable has three or more ordinal categories such as restaurant or product rating from 1 to 5.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### How does Logistic Regression work?\n",
"\n",
"Linear regression is well suited for estimating values, but it isn’t the best tool for predicting the categorical variables. Logistic regression is a special case of linear regression where the target variable is categorical in nature. Logistic Regression predicts the probability of occurrence of a binary event utilizing a logit function.\n",
"\n",
">\n",
">Linear Regression Equation:\n",
">\n",
">$$ y = \\beta_0 + \\beta_1x_1 + \\beta_2x_2 + \\cdots + \\beta_px_p $$\n",
"\n",
"In Logistic Regression, we use the same equation but with some modifications made to $y$. Let's reiterate a fact about Logistic Regression: we calculate probabilities. And, probabilities always lie between 0 and 1. In other words, we can say:\n",
"\n",
"The response value must be positive.\n",
"It should be lower than 1.\n",
"\n",
"We know the exponential of any value is always a positive number. And, any number divided by number + 1 will always be lower than 1. Let's implement these two findings:\n",
"\n",
">Logistic Function:\n",
">$$ p = \\frac{e^{y}}{1+e^{y}} $$\n",
"\n",
"Sigmoid \"S\" shape functions have the same characteristics.\n",
"\n",
"$$ p = \\frac{1}{1+e^{-y}} $$\n",
"\n",
">Apply Sigmoid function on linear regression:\n",
">$$ p = \\frac{e^{(\\beta_0 + \\beta_1x_1 + \\beta_2x_2 + \\cdots + \\beta_px_p)}}{1+e^{(\\beta_0 + \\beta_1x_1 + \\beta_2x_2 + \\cdots + \\beta_px_p)}}$$\n",
">\n",
"\n",
"Now we are convinced that the probability value will always lie between 0 and 1.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### A familiar example\n",
"\n",
"We are going to start by plotting something we understand in the real world, although we may never actually have plotted it before.\n",
"\n",
"Let's say on the x-axis is the number of goals scored by an NFL team over a season and say the outcome on the y-axis is whether they lost or won the game indicated by a value of 0 or 1 respectively. \n",
"\n",
"Then a plot for these scores might look like this:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"files/images/b1fig1_nfloutcomes.png\" />"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So, how do we predict whether we have a win or a loss if we are given a score? \n",
"Clearly linear regression is not a good model. \n",
"If we plot a normal linear regression over our data points, it looks like this:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"files/images/b1fig2_nfloutcomes_withline.png\" />"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### How do we model this sort of data best?\n",
"\n",
"We need a better way to model our data. We build a linear model for binary response data.\n",
"\n",
"We will just pull a function out of the data science bag of tricks and show that it works reasonably well. We are going to understand how we came up with that function and how it is related to binary outcomes and odds.\n",
"\n",
"This function will need to have a value of 0 for the loss scores and 1 for the win scores.\n",
"To make sense it will need to be 0 for some score and all scores below it and be 1 for some other score and all scores above it. And it will need to smoothly increase from 0 to 1 in the intermediate range.\n",
"\n",
"Logistic Regression assumes a linear relationship between the independent variables and the link function (logit).\n",
"\n",
"It will need to look something like this:\n",
"<img src=\"files/images/ey x.png\" width=\"450\"/>\n",
"<img src=\"files/images/standardSigmoidFunction.png\" />"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Now for a spot of Math\n",
"\n",
"A function that has the above shape is:\n",
"\n",
"$$P(x) = \\frac{1}{1 + e^{\\beta_0 + \\beta_1x}}$$\n",
"\n",
"where $P(x)$ is the probability of a score of $x$ leading to a win. \n",
"$\\beta_0, \\beta_1$ are parameters that we will estimate, so the curve fits our data.\n",
"\n",
"\n",
"Notice that we have a familiar looking linear function, \n",
"$$\\beta_0 + \\beta_1x$$ \n",
"but it's plugged into a formula that generates the shape we want. \n",
"\n",
"From the shape we can see that if Score was less than 20 then $P(x)$ would predict a loss, if Score was greater than 30, $P(x)$ would predict a win. But in the middle things would be somewhat fuzzy - we would have even odds when the score was around 25.\n",
"\n",
"So this sort of function is what we use to model binary outcomes."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Threshold functions for logistic regression\n",
"\n",
"\n",
"#### Odds, mathematically speaking. \n",
"\n",
"We are going to take the notion of odds, put a simple mathematical framework around it and then use our previous knowledge of linear regression to create a model that predicts binary outcomes. \n",
"\n",
"Basically all we need to know is that Probability is a number between 0 and 1 and indicates the likelihood of an event occurring. We remind ourselves that: \n",
"\n",
"probability = 0 is as good as the event being impossible and \n",
"\n",
"probability = 1 is as good as it being certain. \n",
"\n",
"We should also remind ourselves that if \n",
"the probability of an event happening is $p$ \n",
"then the probability of it not happening is $1 - p$. \n",
"\n",
"That's all the probability we need.\n",
"\n",
"Having said that let's start talking about odds.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Odds and Odds Ratio\n",
"\n",
"When bettors say the odds of winning are 1:4 what is this in terms of probability? \n",
"\n",
"It means 1 part chance of winning to 4 parts chance of losing. Note that total # of parts = 5 and odds of winning is 1 out of 5. So p is 1/5 = 0.2 and 1-p is 0.8. Here p is small and 1-p is large.\n",
"\n",
"The odds might be 1:1 which means p = 1/2 and 1-p = 1/2 i.e. equal chances of an even happening or not = \"even odds\".\n",
"\n",
"The odds might be 3:2 which means p = 0.6 and 1-p = 0.4. Here p is greater and 1-p is smaller.\n",
"\n",
"So depending on the ratio of p to 1-p we have more or less confidence in a bet winning.\n",
"\n",
"This suggests we might want to look at: \n",
"\n",
"$$Odds Ratio (OR) = \\frac {p}{1-p}$$\n",
"\n",
"If OddsRatio is high say: \n",
"\n",
"$$OR > 4$$ \n",
"\n",
"then the event might be considered very likely and if: \n",
"\n",
"$$OR < 0.25$$ \n",
"\n",
"then very unlikely. \n",
"\n",
"\n",
"### The Logit Function\n",
"\n",
"Mathematicians like to work with a function derived from this called the Logit function. It's the Log of the OddsRatio\n",
"\n",
"$$logit(p) = log(\\frac{p}{1-p})$$ \n",
"\n",
"or the LogOdds function. \n",
"\n",
"\n",
"<img src=\"files/images/logodds.png\" width=\"400\"/>\n",
"\n",
"As you might recognize, the right side of the (immediate) equation above depicts the linear combination of independent variables. The left side is known as the log - odds or odds ratio or logit function and is the link function for Logistic Regression.\n",
"\n",
"We can interpret the above equation as, a unit increase in variable $x$ results in multiplying the odds ratio by $exp$ to power $\\beta$. In other words, the regression coefficients explain the change in log(odds) in the response for a unit change in predictor. However, since the relationship between $p(X)$ and $X$ is not straight line, a unit change in input feature doesn't really affect the model output directly but it affects the odds ratio.\n",
"\n",
"So we would say: \n",
"\n",
"$$logit(p) = log( \\frac{p}{1-p} ) = \\beta_0 + \\beta_1X$$ \n",
"\n",
"where $X$ is the \"value\" of the event. \n",
"\n",
"So here instead of $Y = \\beta_0 + b_1X$ we want to plot $logit(p)$ on the $Y$ axis and the event or the score on the $X$ axis. \n",
"\n",
"So this is how the linear model slips in - we want to express log odds as a linear function of score. \n",
"\n",
"Patience now, we are just one step away. \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### How to estimate coefficients?\n",
"\n",
"In Linear Regression, we use the Sum of Squared Errors and Ordinary Least Square (OLS) method to estimate the best coefficients to attain good model fit. In Logistic Regression, we use maximum likelihood method to determine the best coefficients and eventually a good model fit.\n",
"\n",
"Maximum likelihood works like this: It tries to find the value of coefficients $(\\beta_0,\\beta_1)$ such that the predicted probabilities are as close to the observed probabilities as possible. In other words, for a binary classification (1/0), maximum likelihood will try to find values of $(\\beta_0,\\beta_1)$ such that the resultant probabilities are closest to either 1 or 0. The likelihood function is written as\n",
"\n",
"\n",
"$$Log Loss = \\sum_{(x,y)\\in D} -y \\cdot log(y_{pred}) - (1 - y) \\cdot log(1 - y_{pred})$$\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### How can you evaluate Logistic Regression model fit and accuracy ?\n",
"\n",
"In Linear Regression, we check adjusted R², F Statistics, MAE, and RMSE to evaluate model fit and accuracy. But, Logistic Regression employs all different sets of metrics. Here, we deal with probabilities and categorical values. Following are the evaluation metrics used for Logistic Regression:\n",
"\n",
"1. Akaike Information Criteria (AIC)\n",
"2. Null Deviance and Residual Deviance\n",
"3. Confusion Matrix\n",
"4. Receiver Operator Characteristic (ROC)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Showcase"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We're going to look at the data set from Lending Club, the first US peer-to-peer lending company. Let's assume we have a FICO Score (credit scores) of 720 and we want to borrow 10,000 dollars.\n",
"We would like to get an Interest Rate less that 12 per cent.\n",
"\n",
"The question we pose is: \n",
"\n",
"> Can we get a loan, from the Lending Club, of 10,000 dollars at 12 per cent or less, with a FICO Score of 720?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"How do we use Logistic Regression here? Let's recast the problem as follows:\n",
"\n",
"> What is the probability of getting a Loan, from the Lending Club, of 10,000 dollars at 12 per cent or less with a FICO Score of 720? \n",
"\n",
"Then let us decide that if we get a probability of less than 0.67 we say it means we won't get the loan and if it is greater than 0.67 we will. I.e. we are not confident until we have a 2/3 chance of getting it.\n",
"\n",
"In reality we can set the threshold higher, say 0.8, if we want to be \"more certain\" that it will happen, but for this exercise we'll just say 0.67.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We start with a model of the form\n",
"\n",
"$Interest Rate = \\beta_0 + \\beta_1*FICOScore + \\beta_2*LoanAmount$\n",
"\n",
"And the derive a second equation of the form:\n",
"\n",
"$Z = Prob (InterestRate < 12\\%)$.\n",
"\n",
"We apply this to the loan dataset and create a Logistic Regression Mode."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Interest.Rate</th>\n",
" <th>FICO.Score</th>\n",
" <th>Loan.Length</th>\n",
" <th>Monthly.Income</th>\n",
" <th>Loan.Amount</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>15.31</td>\n",
" <td>670</td>\n",
" <td>36</td>\n",
" <td>4891.67</td>\n",
" <td>6000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>19.72</td>\n",
" <td>670</td>\n",
" <td>36</td>\n",
" <td>3575.00</td>\n",
" <td>2000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>14.27</td>\n",
" <td>665</td>\n",
" <td>36</td>\n",
" <td>4250.00</td>\n",
" <td>10625</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>21.67</td>\n",
" <td>670</td>\n",
" <td>60</td>\n",
" <td>14166.67</td>\n",
" <td>28000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>21.98</td>\n",
" <td>665</td>\n",
" <td>36</td>\n",
" <td>6666.67</td>\n",
" <td>22000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Interest.Rate FICO.Score Loan.Length Monthly.Income Loan.Amount\n",
"6 15.31 670 36 4891.67 6000\n",
"11 19.72 670 36 3575.00 2000\n",
"12 14.27 665 36 4250.00 10625\n",
"13 21.67 670 60 14166.67 28000\n",
"21 21.98 665 36 6666.67 22000"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"dfr = pd.read_csv('data/loanf.csv')\n",
"dfr.head()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Interest.Rate</th>\n",
" <th>FICO.Score</th>\n",
" <th>Loan.Length</th>\n",
" <th>Monthly.Income</th>\n",
" <th>Loan.Amount</th>\n",
" <th>intercept</th>\n",
" <th>TF</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>15.31</td>\n",
" <td>670</td>\n",
" <td>36</td>\n",
" <td>4891.67</td>\n",
" <td>6000</td>\n",
" <td>1.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>19.72</td>\n",
" <td>670</td>\n",
" <td>36</td>\n",
" <td>3575.00</td>\n",
" <td>2000</td>\n",
" <td>1.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>14.27</td>\n",
" <td>665</td>\n",
" <td>36</td>\n",
" <td>4250.00</td>\n",
" <td>10625</td>\n",
" <td>1.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>21.67</td>\n",
" <td>670</td>\n",
" <td>60</td>\n",
" <td>14166.67</td>\n",
" <td>28000</td>\n",
" <td>1.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>21.98</td>\n",
" <td>665</td>\n",
" <td>36</td>\n",
" <td>6666.67</td>\n",
" <td>22000</td>\n",
" <td>1.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>394</th>\n",
" <td>18.75</td>\n",
" <td>670</td>\n",
" <td>36</td>\n",
" <td>4500.00</td>\n",
" <td>21250</td>\n",
" <td>1.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>396</th>\n",
" <td>14.33</td>\n",
" <td>670</td>\n",
" <td>36</td>\n",
" <td>3333.33</td>\n",
" <td>12000</td>\n",
" <td>1.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>407</th>\n",
" <td>22.47</td>\n",
" <td>670</td>\n",
" <td>60</td>\n",
" <td>5083.33</td>\n",
" <td>22000</td>\n",
" <td>1.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>413</th>\n",
" <td>15.80</td>\n",
" <td>665</td>\n",
" <td>36</td>\n",
" <td>6612.00</td>\n",
" <td>5000</td>\n",
" <td>1.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>427</th>\n",
" <td>13.11</td>\n",
" <td>670</td>\n",
" <td>36</td>\n",
" <td>4166.67</td>\n",
" <td>6000</td>\n",
" <td>1.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>80 rows × 7 columns</p>\n",
"</div>"
],
"text/plain": [
" Interest.Rate FICO.Score Loan.Length Monthly.Income Loan.Amount \\\n",
"6 15.31 670 36 4891.67 6000 \n",
"11 19.72 670 36 3575.00 2000 \n",
"12 14.27 665 36 4250.00 10625 \n",
"13 21.67 670 60 14166.67 28000 \n",
"21 21.98 665 36 6666.67 22000 \n",
".. ... ... ... ... ... \n",
"394 18.75 670 36 4500.00 21250 \n",
"396 14.33 670 36 3333.33 12000 \n",
"407 22.47 670 60 5083.33 22000 \n",
"413 15.80 665 36 6612.00 5000 \n",
"427 13.11 670 36 4166.67 6000 \n",
"\n",
" intercept TF \n",
"6 1.0 False \n",
"11 1.0 False \n",
"12 1.0 False \n",
"13 1.0 False \n",
"21 1.0 False \n",
".. ... ... \n",
"394 1.0 False \n",
"396 1.0 False \n",
"407 1.0 False \n",
"413 1.0 False \n",
"427 1.0 False \n",
"\n",
"[80 rows x 7 columns]"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# we add a column which indicates (True/False) whether the interest rate is <= 12 \n",
"dfr['TF']=dfr['Interest.Rate']<=12\n",
"# inspect again\n",
"dfr.head(80)\n",
"# we see that the TF values are False as Interest.Rate is higher than 12 in all these cases"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Interest.Rate</th>\n",
" <th>FICO.Score</th>\n",
" <th>Loan.Length</th>\n",
" <th>Monthly.Income</th>\n",
" <th>Loan.Amount</th>\n",
" <th>TF</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>650</th>\n",
" <td>10.0</td>\n",
" <td>700</td>\n",
" <td>36</td>\n",
" <td>3250.00</td>\n",
" <td>2800</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>204</th>\n",
" <td>10.0</td>\n",
" <td>715</td>\n",
" <td>36</td>\n",
" <td>15416.67</td>\n",
" <td>6000</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>440</th>\n",
" <td>10.0</td>\n",
" <td>730</td>\n",
" <td>36</td>\n",
" <td>6250.00</td>\n",
" <td>21000</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>521</th>\n",
" <td>10.0</td>\n",
" <td>715</td>\n",
" <td>36</td>\n",
" <td>5000.00</td>\n",
" <td>12000</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1017</th>\n",
" <td>10.0</td>\n",
" <td>735</td>\n",
" <td>60</td>\n",
" <td>4000.00</td>\n",
" <td>5000</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Interest.Rate FICO.Score Loan.Length Monthly.Income Loan.Amount \\\n",
"650 10.0 700 36 3250.00 2800 \n",
"204 10.0 715 36 15416.67 6000 \n",
"440 10.0 730 36 6250.00 21000 \n",
"521 10.0 715 36 5000.00 12000 \n",
"1017 10.0 735 60 4000.00 5000 \n",
"\n",
" TF \n",
"650 True \n",
"204 True \n",
"440 True \n",
"521 True \n",
"1017 True "
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# now we check the rows that have interest rate == 10 (just some number < 12)\n",
"# this is just to confirm that the TF value is True where we expect it to be\n",
"d = dfr[dfr['Interest.Rate']==10]\n",
"d.head()\n",
"# all is well"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we use our Logistic Regression modeler to create Logit model using this data, with the 'TF' column as the dependent (or response) variable and 'FICO.Score' and 'Loan.Amount' as independent (or predictor) variables.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Optimization terminated successfully.\n",
" Current function value: 0.319503\n",
" Iterations 8\n"
]
},
{
"ename": "TypeError",
"evalue": "__init__() missing 2 required positional arguments: 'endog' and 'exog'",
"output_type": "error",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[1;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m<ipython-input-4-6b4334bfe005>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m 7\u001b[0m \u001b[0mresult\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mlogit\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 8\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 9\u001b[1;33m \u001b[0msm\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mLogit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[1;31mTypeError\u001b[0m: __init__() missing 2 required positional arguments: 'endog' and 'exog'"
]
}
],
"source": [
"import statsmodels.api as sm\n",
"# statsmodels requires us to add a constant column representing the intercept\n",
"dfr['intercept']=1.0\n",
"# identify the independent variables \n",
"ind_cols=['FICO.Score','Loan.Amount','intercept']\n",
"logit = sm.Logit(dfr['TF'], dfr[ind_cols])\n",
"result=logit.fit()\n",
"\n",
"sm.Logit()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We should see some soothing messages from our software re-assuring us that all went well \n",
"and giving us some numbers we may not find useful right now. \n",
"More importantly we want the results.\n",
"What are the fitted coefficients that the software has computed?"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"FICO.Score 0.087423\n",
"Loan.Amount -0.000174\n",
"intercept -60.125045\n",
"dtype: float64\n"
]
}
],
"source": [
"# get the fitted coefficients from the results\n",
"coeff = result.params\n",
"print(coeff)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The numbers above are the coefficients for the respective independent, i.e. predictor, variables in the linear expression."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So, using the above coefficients, the linear part of our predictor is \n",
"\n",
"$$z = -60.125 + 0.087423*FicoScore -0.000174*LoanAmount$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, the probability of our desired outcome, ie our getting a loan at 12% interest or less, is\n",
"\n",
"$$p(z) = \\frac{1}{1 + e^{\\beta_0 + \\beta_1*FicoScore + \\beta_2*LoanAmount}}$$ \n",
"\n",
"where $\\beta_0 = −60.125, \\beta_1 = 0.087423$ and $\\beta_2 = −0.000174$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We create a function in code that encapsulates all this.\n",
"\n",
"It takes as input, a borrowers FICO score, the desired loan amount and the coefficient vector from our model. It returns a probability of getting the loan, a number between 0 and 1."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"from math import exp\n",
"def pz(fico,amt,coeff):\n",
" # compute the linear expression by multipyling the inputs by their respective coefficients.\n",
" # note that the coefficient array has the intercept coefficient at the end\n",
" z = coeff[0]*fico + coeff[1]*amt + coeff[2]\n",
" return 1/(1+exp(-1*z))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we use our data FICO=720 and Loan Amount=10,000 to get a probability using the z value\n",
"and the logistic formula. "
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.7463785889515121"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pz(720,10000,coeff)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This value of 0.746 tells us we have a good chance of getting the loan we want, according to our criterion, where anything above 0.67 was considered a 'yes'."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we are going to try (fico, amt) pairs as follows:\n",
"\n",
"* 720,20000\n",
"* 720,30000\n",
"* 820,10000\n",
"* 820,20000 \n",
"* 820,30000 "
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Trying multiple FICO Loan Amount combinations: \n",
"----\n",
"fico=720, amt=10,000\n",
"0.7463785889515121\n",
"fico=720, amt=20,000\n",
"0.3405398576881749\n",
"fico=720, amt=30,000\n",
"0.08308359523703378\n",
"fico=820, amt=10,000\n",
"0.9999457423271543\n",
"fico=820, amt=20,000\n",
"0.9996908677522978\n",
"fico=820, amt=30,000\n",
"0.9982408301380601\n"
]
}
],
"source": [
"print(\"Trying multiple FICO Loan Amount combinations: \")\n",
"print('----')\n",
"print(\"fico=720, amt=10,000\")\n",
"print(pz(720,10000,coeff))\n",
"print(\"fico=720, amt=20,000\")\n",
"print(pz(720,20000,coeff))\n",
"print(\"fico=720, amt=30,000\")\n",
"print(pz(720,30000,coeff))\n",
"print(\"fico=820, amt=10,000\")\n",
"print(pz(820,10000,coeff))\n",
"print(\"fico=820, amt=20,000\")\n",
"print(pz(820,20000,coeff))\n",
"print(\"fico=820, amt=30,000\")\n",
"print(pz(820,30000,coeff))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We see as somewhat expected that the person with a 720 FICO Score will have decreasing probability of getting loans with higher amounts.\n",
"However, the person with the 820 FICO Score is very likely to get loans with those amounts, again as expected."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}