Table of Contents
Cohen’s Kappa is a statistical measure used to evaluate the level of agreement between two raters or evaluators. In order to calculate Cohen’s Kappa in Python, the following steps can be followed:
1. Import the necessary libraries: The first step is to import the necessary libraries such as scikit-learn, numpy, and pandas.
2. Prepare the data: The data must be prepared in a format that can be used by the scikit-learn library. This includes converting the data into a numpy array or a pandas dataframe.
3. Calculate the observed agreement: Using the scikit-learn library, the observed agreement between the two raters can be calculated by passing the actual values and the predicted values.
4. Calculate the expected agreement: The expected agreement is calculated by creating a contingency table using the actual and predicted values and then computing the expected agreement using a formula.
5. Calculate Cohen’s Kappa: Finally, Cohen’s Kappa can be calculated by using the formula, which takes into account the observed and expected agreements.
By following these steps, one can easily calculate Cohen’s Kappa in Python, providing a reliable measure of agreement between two raters or evaluators.
Calculate Cohen’s Kappa in Python
In statistics, is used to measure the level of agreement between two raters or judges who each classify items into mutually exclusive categories.
The formula for Cohen’s kappa is calculated as:
k = (po – pe) / (1 – pe)
where:
- po: Relative observed agreement among raters
- pe: Hypothetical probability of chance agreement
Rather than just calculating the percentage of items that the raters agree on, Cohen’s Kappa attempts to account for the fact that the raters may happen to agree on some items purely by chance.
The value for Cohen’s Kappa always ranges between 0 and 1where:
- 0 indicates no agreement between the two raters
- 1 indicates perfect agreement between the two raters
The following table summarizes how to interpret different values for Cohen’s Kappa:
The following example shows how to calculate Cohen’s Kappa in Python.
Example: Calculating Cohen’s Kappa in Python
Suppose two art museum curators are asked to rate 15 paintings on whether they’re good enough to be shown in a new exhibit.
The following code shows how to use the cohen_kappa_score() function from the sklearn library to calculate Cohen’s Kappa for the two raters:
from sklearn.metricsimport cohen_kappa_score
#define array of ratings for both raters
rater1 = [0, 1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0]
rater2 = [0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0]
#calculate Cohen's Kappa
cohen_kappa_score(rater1, rater2)
0.33628318584070793
Cohen’s Kappa turns out to be 0.33628.
Based on the table from earlier, we would say that the two raters only had a “fair” level of agreement.
You can use the function from the statsmodels library to calculate this metric.
Note: You can find the complete documentation for the cohen_kappa_score() function .
Additional Resources
The following tutorials offer additional resources on Cohen’s Kappa:
Cite this article
stats writer (2024). How can Cohen’s Kappa be calculated in Python?. PSYCHOLOGICAL SCALES. Retrieved from https://scales.arabpsychology.com/stats/how-can-cohens-kappa-be-calculated-in-python/
stats writer. "How can Cohen’s Kappa be calculated in Python?." PSYCHOLOGICAL SCALES, 28 Jun. 2024, https://scales.arabpsychology.com/stats/how-can-cohens-kappa-be-calculated-in-python/.
stats writer. "How can Cohen’s Kappa be calculated in Python?." PSYCHOLOGICAL SCALES, 2024. https://scales.arabpsychology.com/stats/how-can-cohens-kappa-be-calculated-in-python/.
stats writer (2024) 'How can Cohen’s Kappa be calculated in Python?', PSYCHOLOGICAL SCALES. Available at: https://scales.arabpsychology.com/stats/how-can-cohens-kappa-be-calculated-in-python/.
[1] stats writer, "How can Cohen’s Kappa be calculated in Python?," PSYCHOLOGICAL SCALES, vol. X, no. Y, ص Z-Z, June, 2024.
stats writer. How can Cohen’s Kappa be calculated in Python?. PSYCHOLOGICAL SCALES. 2024;vol(issue):pages.