Table of Contents
To compare two ROC curves, the difference in the areas under the curves (AUC) can be calculated. This can be done by subtracting the AUC of one curve from the AUC of the other curve. If the difference is greater than a certain threshold, it can be assumed that one curve is better than the other. Additionally, the curves can be compared visually, by plotting them on the same graph and comparing the shape and location of the curves.
One way to visualize the performance of in machine learning is by creating a ROC curve, which stands for “receiver operating characteristic” curve.
This type of curve displays the sensitivity and specificity of a classification model:
- Sensitivity: The probability that the model predicts a positive outcome for an observation when the outcome is indeed positive.
- Specificity: The probability that the model predicts a negative outcome for an observation when the outcome is indeed negative.
The x-axis of a ROC curve represents (1- Specificity) and the y-axis represents the Sensitivity:
The more that the ROC curve hugs the top left corner of the plot, the better the model does at classifying the data into categories.
To quantify this, we can calculate the AUC (area under the curve) which tells us how much of the plot is located under the curve.
The closer AUC is to 1, the better the model.
When comparing two ROC curves to determine which classification model is best, we often look at which ROC curve “hugs” the top left corner of the plot more and thus has a higher AUC value.
Example: How to Compare Two ROC Curves
Suppose we fit a logistic regression model and a gradient boosted model to a dataset to predict the outcome of some response variable.
Suppose we then create ROC curves to visualize the performance of each model:
The blue line shows the ROC curve for the logistic regression model and the orange line shows the ROC curve for the gradient boosted model.
From our plot we can see the following AUC values for each model:
- AUC of logistic regression model: 0.7902
- AUC of gradient boosted model: 0.9712
Note: In this example we only compared two ROC curves, but it’s possible to fit several different classification models to a dataset and compare even more ROC curves to determine the best model to use.
The following tutorials provide additional information about classification models and ROC curves: