Table of Contents
Cohen’s Kappa is a statistical measure of inter-rater agreement, commonly used in the field of social sciences to assess the level of agreement between two or more raters or observers. In order to calculate Cohen’s Kappa in R, one must first have a dataset containing the ratings or observations of the raters. The next step is to use the “irr” package, which provides functions for calculating Cohen’s Kappa and other measures of agreement. The “kappa2” function in the “irr” package can be used to calculate Cohen’s Kappa, taking into account the number of raters and categories in the dataset. The output of this function will provide the value of Cohen’s Kappa, indicating the level of agreement between the raters. This measure can be useful in evaluating the reliability and consistency of ratings or observations in a study.
Calculate Cohen’s Kappa in R
In statistics, is used to measure the level of agreement between two raters or judges who each classify items into mutually exclusive categories.
The formula for Cohen’s kappa is calculated as:
k = (po – pe) / (1 – pe)
where:
- po: Relative observed agreement among raters
- pe: Hypothetical probability of chance agreement
Rather than just calculating the percentage of items that the raters agree on, Cohen’s Kappa attempts to account for the fact that the raters may happen to agree on some items purely by chance.
The value for Cohen’s Kappa always ranges between 0 and 1where:
- 0 indicates no agreement between the two raters
- 1 indicates perfect agreement between the two raters
The following table summarizes how to interpret different values for Cohen’s Kappa:
The easiest way to calculate Cohen’s Kappa in R is by using the cohen.kappa() function from the psych package.
The following example shows how to use this function in practice.
Example: Calculating Cohen’s Kappa in R
Suppose two art museum curators are asked to rate 15 paintings on whether they’re good enough to be shown in a new exhibit.
The following code shows how to use the cohen.kappa() function from the psych package to calculate Cohen’s Kappa for the two raters:
library(psych)#define vector of ratings for both raters
rater1 = [0, 1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0]
rater2 = [0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0]
#calculate Cohen's Kappa
cohen.kappa(x=cbind(rater1,rater2))
Cohen Kappa and Weighted Kappa correlation coefficients and confidence boundaries
lower estimate upper
unweighted kappa -0.14 0.34 0.81
weighted kappa -0.14 0.34 0.81
Number of subjects = 15
The estimate column displays the value for Cohen’s Kappa.
Based on the table from earlier, we would say that the two raters only had a “fair” level of agreement.
If you want to calculate the level of agreement between three or more raters, it’s recommended to use Fleiss’ Kappa instead.
The following tutorials offer additional resources on Cohen’s Kappa:
Cite this article
stats writer (2024). How can I calculate Cohen’s Kappa in R?. PSYCHOLOGICAL SCALES. Retrieved from https://scales.arabpsychology.com/stats/how-can-i-calculate-cohens-kappa-in-r/
stats writer. "How can I calculate Cohen’s Kappa in R?." PSYCHOLOGICAL SCALES, 27 Jun. 2024, https://scales.arabpsychology.com/stats/how-can-i-calculate-cohens-kappa-in-r/.
stats writer. "How can I calculate Cohen’s Kappa in R?." PSYCHOLOGICAL SCALES, 2024. https://scales.arabpsychology.com/stats/how-can-i-calculate-cohens-kappa-in-r/.
stats writer (2024) 'How can I calculate Cohen’s Kappa in R?', PSYCHOLOGICAL SCALES. Available at: https://scales.arabpsychology.com/stats/how-can-i-calculate-cohens-kappa-in-r/.
[1] stats writer, "How can I calculate Cohen’s Kappa in R?," PSYCHOLOGICAL SCALES, vol. X, no. Y, ص Z-Z, June, 2024.
stats writer. How can I calculate Cohen’s Kappa in R?. PSYCHOLOGICAL SCALES. 2024;vol(issue):pages.