Table of Contents
Logistic regression is a type of statistical analysis used to identify the relationship between one or more independent variables and a binary dependent variable. To perform logistic regression using Statsmodels, the user should first import the necessary datasets and libraries, then create a linear model using the Statsmodels package, and finally evaluate the model using statistical measures such as AIC, BIC, and R-squared. Additionally, the user may also plot the model to visualize the results.
The module in Python offers a variety of functions and classes that allow you to fit various statistical models.
The following step-by-step example shows how to perform using functions from statsmodels.
Step 1: Create the Data
First, let’s create a pandas DataFrame that contains three variables:
- Hours Studied (Integer value)
- Study Method (Method A or B)
- Exam Result (Pass or Fail)
We’ll fit a logistic regression model using hours studied and study method to predict whether or not a student passes a given exam.
The following code shows how to create the pandas DataFrame:
import pandas as pd #create DataFrame df = pd.DataFrame({'result': [0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1], 'hours': [1, 2, 2, 2, 3, 2, 5, 4, 3, 6, 5, 8, 8, 7, 6, 7, 5, 4, 8, 9], 'method': ['A', 'A', 'A', 'B', 'B', 'B', 'B', 'B', 'B', 'A', 'B', 'A', 'B', 'B', 'A', 'A', 'B', 'A', 'B', 'A']}) #view first five rows of DataFrame df.head() result hours method 0 0 1 A 1 1 2 A 2 0 2 A 3 0 2 B 4 0 3 B
Step 2: Fit the Logistic Regression Model
Next, we’ll fit the logistic regression model using the logit() function:
import statsmodels.formula.api as smf
#fit logistic regression model
model = smf.logit('result ~ hours + method', data=df).fit()
#view model summary
print(model.summary())
Optimization terminated successfully.
Current function value: 0.557786
Iterations 5
Logit Regression Results
==============================================================================
Dep. Variable: result No. Observations: 20
Model: Logit Df Residuals: 17
Method: MLE Df Model: 2
Date: Mon, 22 Aug 2022 Pseudo R-squ.: 0.1894
Time: 09:53:35 Log-Likelihood: -11.156
converged: True LL-Null: -13.763
Covariance Type: nonrobust LLR p-value: 0.07375
===============================================================================
coef std err z P>|z| [0.025 0.975]
-------------------------------------------------------------------------------
Intercept -2.1569 1.416 -1.523 0.128 -4.932 0.618
method[T.B] 0.0875 1.051 0.083 0.934 -1.973 2.148
hours 0.4909 0.245 2.002 0.045 0.010 0.972
===============================================================================
The values in the coef column of the output tell us the average change in the log odds of passing the exam.
For example:
- Using study method B is associated with an average increase of .0875 in the log odds of passing the exam compared to using study method A.
- Each additional hour studied is associated with an average increase of .4909 in the log odds of passing the exam.
The values in the P>|z| column represent the p-values for each coefficient.
For example:
- Studying method has a p-value of .934. Since this value is not less than .05, it means there is not a statistically significant relationship between hours studied and whether or not a student passes the exam.
- Hours studied has a p-value of .045. Since this value is less than .05, it means there is a statistically significant relationship between hours studied and whether or not a student passes the exam.
Step 3: Evaluate Model Performance
To assess the quality of the logistic regression model, we can look at two metrics in the output:
1. Pseudo R-Squared
This value can be thought of as the substitute to the R-squared value for a linear regression model.
It is calculated as the ratio of the maximized log-likelihood function of the null model to the full model.
This value can range from 0 to 1, with higher values indicating a better model fit.
In this example, the pseudo R-squared value is .1894, which is quite low. This tells us that the predictor variables in the model don’t do a very good job of predicting the value of the response variable.
2. LLR p-value
This value can be thought of as the substitute to the p-value for the of a linear regression model.
If this value is below a certain threshold (e.g. α = .05) then we can conclude that the model overall is “useful” and is better at predicting the values of the response variable compared to a model with no predictor variables.
In this example, the LLR p-value is .07375. Depending on the significance level we choose (e.g. .01, .05, .1) we may or may not conclude that the model as a whole is useful.
The following tutorials explain how to perform other common tasks in Python: