This tutorial explains the difference between a t-test and an ANOVA, along with when to use each test.
T-test
A t-test is used to determine whether or not there is a statistically significant difference between the means of two groups. There are two types of t-tests:
1. Independent samples t-test. This is used when we wish to compare the difference between the means of two groups and the groups are completely independent of each other.
For example, researchers may want to know whether diet A or diet B helps people lose more weight. 100 randomly assigned people are assigned to diet A. Another 100 randomly assigned people are assigned to diet B. After three months, researchers record the total weight loss for each person. To determine if the mean weight loss between the two groups is significantly different, researchers can conduct an independent samples t-test.
2. Paired samples t-test. This is used when we wish to compare the difference between the means of two groups and where each observation in one group can be paired with one observation in the other group.
For example, suppose 20 students in a class take a test, then study a certain guide, then retake the test. To compare the difference between the scores in the first and second test, we use a paired t-test because for each student their first test score can be paired with their second test score.
For a t-test to produce valid results, the following assumptions should be met:
- Random: A random sample or random experiment should be used to collect data for both samples.
- Normal: The sampling distribution is normal or approximately normal.
If these assumptions are met, then it’s safe to use a t-test to test for the difference between the means of two groups.
ANOVA
An ANOVA (analysis of variance) is used to determine whether or not there is a statistically significant difference between the means of three or more groups. The most commonly used ANOVA tests in practice are the one-way ANOVA and the two-way ANOVA:
One-way ANOVA: Used to test whether or not there is a statistically significant difference between the means of three or more groups when the groups can be split on one factor.
Example: You randomly split up a class of 90 students into three groups of 30. Each group uses a different studying technique for one month to prepare for an exam. At the end of the month, all of the students take the same exam. You want to know whether or not the studying technique has an impact on exam scores so you conduct a one-way ANOVA to determine if there is a statistically significant difference between the mean scores of the three groups.
Two-way ANOVA: Used to test whether or not there is a statistically significant difference between the means of three or more groups when the groups can be split on two factors.
Example: You want to determine if level of exercise (no exercise, light exercise, intense exercise) and gender (male, female) impact weight loss. In this case, the two factors you’re studying are exercise and gender and your response variable is weight loss (measured in pounds). You can conduct a two-way ANOVA to determine if exercise and gender impact weight loss and to determine if there is an interaction between exercise and gender on weight loss.
For an ANOVA to produce valid results, the following assumptions should be met:
- Normality – all populations that we’re studying follow a normal distribution. So, for example, if we want to compare the exam scores of three different groups of students, the exam scores for the first group, second group, and third group all need to be normally distributed.
- Equal Variance – the population variances in each group are equal or approximately equal.
- Independence – the observations in each group need to be independent of each other. Usually a randomized design will take care of this.
If these assumptions are met, then it’s safe to use an ANOVA to test for the difference between the means of three or more groups.
Understanding the Differences Between Each Test
The main difference between a t-test and an ANOVA is in how the two tests calculate their test statistic to determine if there is a statistically significant difference between groups.
An independent samples t-test uses the following test statistic:
test statistic t = [ (x1 – x2) – d ] / (√s21 / n1 + s22 / n2)
where x1 and x2 are the sample means for groups 1 and 2, d is the hypothesized difference between the two means (often this is zero), s12 and s22 are the sample variances for groups 1 and 2, and n1 and n2 are the sample sizes for groups 1 and 2, respectively.
A paired samples t-test uses the following test statistic:
test statistic t = d / (sd / √n)
where d is the mean difference between the two groups, sd is the standard deviation of the differences, and n is the sample size for each group (note that both groups will have the same sample size).
An ANOVA uses the following test statistic:
test statistic F = s2b / s2w
where s2b is the between sample variance, and s2w is the within sample variance.
A t-test measures the ratio of the mean difference between two groups relative to the overall standard deviation of the differences. If this ratio is high enough, it provides sufficient evidence that there is a significant difference between the two groups.
An ANOVA, on the other hand, measures the ratio of variance between the groups relative to the variance within the groups. Similar to the t-test, if this ratio is high enough, it provides sufficient evidence that not all three groups have the same mean.
Another key difference between a t-test and an ANOVA is that the t-test can tell us whether or not two groups have the same mean. An ANOVA, on the other hand, tells us whether or not three groups all have the same mean, but it doesn’t explicitly tell us which groups have means that are different from one another.
To find out which groups differ from one another, we would have to perform post-hoc tests.
Understanding When to use Each Test
In practice, when we want to compare the means of two groups, we use a t-test. When we want to compare the means of three or more groups, we use an ANOVA.
The underlying reason we don’t simply use several t-tests to compare the means of three or more groups goes back to understanding the type I error rate. Suppose we have three groups we wish to compare the means between: group A, group B, and group C. You may be tempted to perform the following three t-tests:
- A t-test to compare the difference in means between group A and group B
- A t-test to compare the difference in means between group A and group C
- A t-test to compare the difference in means between group B and group C
For each t-test there is a chance that we will commit a type I error, which is the probability that we reject the null hypothesis when it is actually true. This probability is typically 5%. This means that when we perform multiple t-tests, this error rate increases. For example:
- The probability that we commit a type I error with one t-test is 1 – 0.95 = 0.05.
- The probability that we commit a type I error with two t-tests is 1 – (0.952) = 0.0975.
- The probability that we commit a type I error with two t-tests is 1 – (0.953) = 0.1427.
This error rate is unacceptably high. Fortunately, an ANOVA controls for these errors so that the Type I error remains at just 5%. This allows us to be more confident that a statistically significant test result is actually meaningful and not just a result that we got from performing a lot of tests.
Thus, when we want to understand whether there is a difference between the means of three or more groups, we must use an ANOVA so that our results are statistically valid and reliable.