How to Perform Fisher’s Exact Test for Categorical Data

What is Fisher’s Exact Test?

The Fisher’s Exact Test is a powerful and precise statistical analysis tool specifically designed to determine the significance of the relationship between two mutually exclusive categorical variables. Unlike approximation methods, this test provides an exact probability (p-value) of observing the data, or data more extreme, given that the two variables are independent. This makes it a cornerstone technique in situations where data precision is paramount, such as clinical trials and molecular biology.

The core utility of the Fisher’s Exact Test emerges when working with limited datasets. It is the preferred choice when the sample size is small, or when the expected cell counts in a contingency table are low—typically below five or ten. Traditional asymptotic methods, most notably the chi-squared test, rely on the assumption that the sampling distribution of the test statistic is approximately chi-squared. This approximation breaks down severely when data is sparse, leading to inaccurate and unreliable results.

In essence, the test evaluates whether the proportions of categories within two distinct groups differ significantly from each other. To conduct this analysis accurately, the researcher must possess two group variables, each having two or more discrete options, and, critically, the data should result in fewer than ten observations per cell in the resultant 2×2 contingency table. This constraint ensures the exact nature of the probability calculation remains valid, thereby offering a more robust and reliable measure of significance compared to tests that rely on distributional assumptions for large samples.

The Fisher’s Exact Test is also historically known as the Fisher Irwin Test, Fisher’s Exact Test of Independence, or simply Fisher’s Test.

Assumptions for Fisher’s Exact Test

Every statistical methodology is underpinned by a set of foundational assumptions. These assumptions dictate the necessary characteristics of the data for the statistical output to be mathematically sound and yield meaningful, accurate conclusions. Violating these critical assumptions can lead to Type I or Type II errors, compromising the integrity of the research findings. While the Fisher’s Exact Test is less restrictive than tests relying on distributional approximations (like normality), it still requires strict adherence to specific data collection and structure criteria.

Understanding and verifying these prerequisites is a crucial step before proceeding with the analysis. Failing to satisfy these conditions means that the resulting p-value may not correctly reflect the true probability of association between the variables under investigation. The test is robust, but its exact calculation relies heavily on fixed marginal totals, which is indirectly enforced by the following assumptions.

The primary assumptions that must be met for a valid application of the Fisher’s Exact Test include:

1. Random Sample: The data must be collected via a simple random sampling procedure.
2. Independence: Each observation must be independent of all other observations.
3. Mutually Exclusive Groups: The categories defining the groups must be discrete and non-overlapping.

Let us delve into a detailed exploration of what these specific requirements entail for data preparation and collection.

Random Sample

The foundation of reliable inferential statistics rests upon the concept of a simple random sample. This requirement mandates that every potential data point or unit of observation within the population of interest must have an equal and known chance of being selected for inclusion in the study sample. This rigorous sampling method ensures that the collected data accurately represents the broader population without systemic distortion.

If the data points for each group in the analysis are not derived from a genuinely simple random sample, the statistical inference drawn from the test becomes inherently flawed. The potential for systematic error, known statistically as bias, increases significantly. Bias represents a pervasive tendency to generate incorrect results because the data itself does not provide an unbiased snapshot of the population parameters. This selection bias invalidates the interpretation of the p-value, as the sample distribution no longer reflects the theoretical distribution assumed by the test.

Independence

The assumption of independence stipulates that the value associated with any single observation (data point) must not exert any causal or correlational influence on the value of any other observation within the dataset. In simpler terms, each measurement should stand alone, free from dependency on others. This is a critical statistical requirement, particularly when analyzing group differences.

This assumption is commonly violated in scenarios involving repeated measures or longitudinal studies where data is collected from the same units of observation over a period of time (e.g., tracking a subject’s response before and after an intervention, or measuring a store’s sales on consecutive days). In such cases, the observations collected from the same subject, customer, or store are highly likely to be related or affect one another, leading to what is known as autocorrelation or clustered data. When independence is violated, the effective sample size is overestimated, resulting in smaller standard errors and artificially inflated test statistics, potentially leading to incorrect rejection of the Null Hypothesis.

Mutually Exclusive Groups

For the Fisher’s Exact Test to be applicable, the levels or groups of the primary categorical variables must be mutually exclusive. Mutual exclusivity means that a single data point or unit of observation can belong to one and only one category or group at any given time. There must be no overlap between the defined categories.

Consider a categorical variable such as “Hunger Status” defined by the levels ‘Yes’ and ‘No’. These groups are inherently mutually exclusive because an individual cannot simultaneously be categorized as both hungry and not hungry. If the groups were defined in a way that overlap was possible (e.g., categorizing by ‘likes apples’ and ‘likes red fruit’), the validity of the 2×2 contingency table structure would be compromised, as the counts in the cells would be ambiguous or misleading, preventing the accurate calculation of marginal totals required by the hypergeometric distribution utilized by Fisher’s test.

When to use Fisher’s Exact Test?

Choosing the correct statistical test is paramount to ensuring research validity. The Fisher’s Exact Test is specifically tailored for scenarios where traditional large-sample tests are inappropriate due to low cell counts. Its use is recommended when the research objective centers on comparing proportions across two distinct, independent groups based on categorical outcomes.

The decision matrix for selecting Fisher’s Exact Test can be summarized by five key requirements that your data structure and research question must satisfy. These requirements collectively define the appropriate domain for this exact calculation method, distinguishing it from related tests like the Chi-Square test or the G-Test.

You should employ Fisher’s Exact Test only when the following rigorous conditions are met simultaneously:

1. You are testing for a difference or association between two variables.
2. The variable of interest is inherently proportional or categorical.
3. The key categorical variables possess only two options (binary outcomes).
4. The groups being compared constitute independent samples.
5. There are less than 10 observations in any single cell of the contingency table.

A detailed examination of each criterion clarifies precisely when the Fisher’s Exact Test offers the optimal analytical solution.

Testing for a Difference

The primary purpose of applying the Fisher’s Exact Test is to investigate whether a statistically significant difference exists in the distribution of outcomes between two defined groups. Researchers are typically looking to establish if the proportion of success (or failure, or any binary outcome) in Group A is significantly higher or lower than the proportion in Group B.

This focus on testing for a difference contrasts sharply with other statistical objectives. For instance, some analyses aim to quantify the strength and direction of a relationship between two continuous variables (correlation), while others focus on building predictive models where one variable forecasts the value of another (regression/prediction). If your research question is specifically framed around comparing proportions or verifying an association in a 2×2 contingency table, the Fisher’s Exact Test is the appropriate method, provided the other assumptions hold true.

Proportional or Categorical Data

The variables under scrutiny must be fundamentally categorical variables. A categorical variable is defined as one whose possible values belong to a finite set of discrete categories that possess no inherent numerical order. Classic examples include eye color (e.g., blue, brown, green), city of residence, or type of dog breed.

Proportional variables are intrinsically linked to categorical data, often representing summary statistics derived from counts within these categories. Examples include conversion rates on a website (10% vs 15%), the percentage of people who voted, or the proportion of plants that survived versus those that succumbed to an experimental treatment. Since the Fisher’s Exact Test operates on the raw counts (frequencies) that underlie these proportions, it is perfectly suited for analyzing proportional outcomes, as long as the underlying data can be summarized in a 2×2 frequency table.

If the variables you intend to compare are continuous (e.g., height, income, test scores), you would need to use a different procedure, such as an Independent Samples T-Test.

Binary Outcomes (Two Options)

A strict requirement for the traditional Fisher’s Exact Test is that both the grouping variable and the outcome variable must be binary, meaning they should each have only two possible categories or levels. This structure naturally forms the basis of the 2×2 contingency table.

Examples of variables suitable for this structure include: whether a purchase was made (yes/no), recovery from a disease (yes/no), or gender assigned at birth (male/female). While extensions of the Fisher’s Exact Test exist for larger R x C tables, the standard and most commonly applied version of the test—which guarantees the exact probability calculation—is strictly limited to the binary, two-option scenario. If your variables have three or more categories, alternatives like the Chi-Square test or specific multivariate exact tests should be considered.

Independent Samples

The condition of independent samples dictates that the two populations or groups from which the data is drawn must be completely unrelated. The selection or inclusion of a subject in one group must not influence, in any way, the selection or observations associated with a subject in the other group. This assumption is crucial for ensuring that the differences observed are due to the grouping variable itself (e.g., Treatment A vs. Treatment B) rather than inherent dependencies in the measurement process.

As previously noted, repeated measurements collected from the same individual or unit over time constitute dependent samples. For instance, if you measure the change in a response (e.g., symptom presence) in a single group before and after an intervention, the observations are correlated, rendering the samples dependent. Using Fisher’s Exact Test on dependent samples would violate its independence assumption and lead to incorrect inferences regarding the difference in proportions.

If your experimental design involves analyzing repeated measures (dependent samples) where the outcome is binary, you should consider employing the McNemar Test, which is specifically designed for paired categorical data.

Less than 10 in a Cell

The requirement that cell counts remain low is perhaps the single most defining characteristic necessitating the use of the Fisher’s Exact Test. The rule-of-thumb widely accepted in statistical practice suggests using this test when the expected frequency in any cell of the 2×2 contingency table drops below ten, though a cutoff of five is also frequently used.

A “cell” simply refers to the count of observations that fall into the intersection of one level of the grouping variable and one level of the outcome variable. For example, in a study comparing Treatment A vs. Treatment B on recovery (Yes/No), one cell would contain the count of subjects who received Treatment A and recovered (Yes). When these counts are small, the sampling distribution of the test statistic for the chi-squared test—which approximates the continuous chi-square distribution—does not hold true. The approximation is poor, leading to potentially inflated Type I error rates (falsely rejecting the Null Hypothesis). By contrast, Fisher’s test calculates the exact probability using the hypergeometric distribution, circumventing the need for approximation and ensuring accuracy regardless of how small the cell counts are.

Conversely, if you find that you have expected counts of more than 10 in every cell, the large-sample assumptions are met, and we recommend transitioning to the Two-Proportion Z-Test for computational efficiency. If all cell counts are greater than 10 and the total sample size exceeds 1000 observations, the G-Test might also be a powerful alternative.

Fisher’s Exact Test Example

To illustrate the practical application of this method, consider a clinical trial investigating the efficacy of two distinct experimental drugs designed to treat a rare infectious disease. Due to the rarity of the disease, the sample size is severely constrained, leading to low observation counts in the recovery categories.

Grouping Variable: Treatment Type (A or B)

Outcome Variable: Recovered from Disease (Yes or No)

In this scenario, we are primarily interested in examining whether the two treatment groups (A and B) differ significantly in their respective rates of recovery from the disease. Our formal scientific question translates into a statistical framework that defines the Null Hypothesis (H0) and the Alternative Hypothesis (Ha). The H0 states that there is no difference in recovery rates between Treatment A and Treatment B, meaning the variables are independent. The Ha states that a statistically significant difference does exist.

Given that the outcome variable is strictly binary (Yes/No recovery), the grouping variable is also binary, and the limited sample size ensures that the cell counts are small (likely less than ten), Fisher’s Exact Test is the demonstrably appropriate tool. The test leverages the counts in the 2×2 contingency table to calculate the probability of observing that specific distribution of recovery outcomes, assuming H0 is true.

The result of the analysis is a probability, commonly referred to as the p-value. This p-value quantifies the chance of observing the obtained results (or results even more extreme in favor of the alternative hypothesis) if, in reality, there was absolutely no association or difference in recovery rate between the two treatment types. By convention, a p-value less than or equal to the predetermined significance level (alpha, typically 0.05) is considered statistically significant. If this condition is met, we possess sufficient evidence to reject the Null Hypothesis, allowing us to confidently conclude that the observed difference in recovery rates is not merely due to random sampling chance but reflects a genuine effect of the treatment type.