Table of Contents
The Chi-Square test is commonly used to analyze the relationship between two categorical variables. In order to determine the strength of this relationship, effect size measures can be calculated. Effect size measures provide a standardized measure of the magnitude of the relationship between the variables being studied. There are three main methods for calculating effect size in a Chi-Square test: 1) Cramer’s V, which measures the strength of association between categorical variables, 2) Phi coefficient, which is used when both variables have only two categories, and 3) Odds ratio, which compares the odds of an event occurring in one group to the odds of the same event occurring in another group. Each of these methods provides a different perspective on the relationship between the variables and can be useful for interpreting the results of a Chi-Square test. By calculating effect size, researchers can better understand the practical significance of their findings and make more informed conclusions.
Three Ways to Calculate Effect Size for a Chi-Square Test
In statistics, there are two commonly used Chi-Square tests:
Chi-Square Test for Goodness of Fit: Used to determine whether or not a categorical variable follows a hypothesized distribution.
Chi-Square Test for Independence: Used to determine whether or not there is a significant association between two categorical variables from a single population.
For both of these tests, we end up with a p-value that tells us whether or not we should reject the null hypothesis of the test. The p-value tells us whether or not the results of the test are significant, but it doesn’t tell us the effect size of the test.
There are three ways to measure effect size: Phi (φ), Cramer’s V (V), and odds ratio (OR).
In this post we explain how to calculate each of these effect sizes along with when it’s appropriate to use each one.
Phi (φ)
How to Calculate
Phi is calculated as φ = √(X2 / n)
where:
X2 is the Chi-Square test statistic
n = total number of observations
When to Use
It’s appropriate to calculate φ only when you’re working with a 2 x 2 contingency table (i.e. a table with exactly two rows and two columns).
How to Interpret
A value of φ = 0.1 is considered to be a small effect, 0.3 a medium effect, and 0.5 a large effect.
Cramer’s V (V)
How to Calculate
Cramer’s V is calculated as V = √(X2 / n*df)
where:
X2 is the Chi-Square test statistic
n = total number of observations
df = (#rows-1) * (#columns-1)
When to Use
It’s appropriate to calculate V when you’re working with any table larger than a 2 x 2 contingency table.
How to Interpret
The following table shows how to interpret V based on the degrees of freedom:
| Degrees of freedom | Small | Medium | Large |
|---|---|---|---|
| 1 | 0.10 | 0.30 | 0.50 |
| 2 | 0.07 | 0.21 | 0.35 |
| 3 | 0.06 | 0.17 | 0.29 |
| 4 | 0.05 | 0.15 | 0.25 |
| 5 | 0.04 | 0.13 | 0.22 |
Odds Ratio (OR)
How to Calculate
Given the following 2 x2 table:
| Effect Size | # Successes | # Failures |
|---|---|---|
| Treatment Group | A | B |
| Control Group | C | D |
The odds ratio would be calculated as:
Odds ratio = (AD) / (BC)
When to Use
It’s appropriate to calculate the odds ratio only when you’re working with a 2 x 2 contingency table. Typically the odds ratio is calculated when you’re interested in studying the odds of success in a treatment group relative to the odds of success in a control group.
How to Interpret
There is no specific value at which we deem an odds ratio be a small, medium, or large effect, but the further away the odds ratio is from 1, the higher the likelihood that the treatment has an actual effect.
It’s best to use domain specific expertise to determine if a given odds ratio should be considered small, medium, or large.