Design
This module covers the types of designs and analyses involving more than one independent variable. This gets tricky because it’s difficult when you first begin this exercise to differentiate between adding an additional level to an existing independent variable, as compared to adding a new independent variable all together. For example, let’s say that we are interested in studying the effect of room temperature on test taking. To do this we compare test scores of students who take a test in a 90 degree room vs. those who take a test in a 50 degree room. This is a case of one independent variable (i.e., room temperature) and two levels (i.e., 50 and 90 degrees), and the appropriate analysis would be a between-subjects’ t-test (assuming the two groups are made up of two separate groups of students). Let’s extend this design by adding a third group, 70 degrees. We now have an experiment with one independent variable (i.e., room temperature) and three levels (50, 70, and 90 degrees). We have not added an additional independent variable; rather we have simply added a third level to the already existing independent variable. It is crucial that you understand the difference between a variable and a level in order to select and interpret the analysis for a given experiment.
So how would we go about adding a second independent variable? Remember that a variable, by definition, must have at least two levels (i.e., it must “vary”). To keep our discussion of the complex concept of multi-factor designs at the most basic level, we will consider the simplest type of situation where there are more than one independent variables; the 2 X 2 (“two by two”) between-subjects factorial design. In our example with room temperature, let’s go back to the original two groups (50 vs. 70 degrees), but let’s add a new independent variable with two levels, test difficulty (difficult vs. easy). Note that this new variable is qualitatively different than room temperature, and it has levels of it’s own, so that we’ve done more than simply add an additional level to the existing independent variable. Since this is a between-subjects design we will randomly assign subjects to four groups, as illustrated in Figure 1. Experiments that include multiple categorical independent variables and a continuous variable are often called “factorial” designs, and the independent variables are called “factors” (not to be confused with a “factor analysis”, which is a multivariate statistical technique, which involves the statistical grouping of multiple continuous variables.)
(IV) Room Temperature |
||
(IV) Test Difficulty |
(Level) 50 degrees |
(Level) 90 degrees |
(Level) Hard Test |
Hard Test in 50 degrees |
Hard Test in 90 degrees |
(Level) Easy Test |
Easy Test in 50 degrees |
Easy Test in 90 degrees |
Figure 1. Example of a 2 X 2 Design
Whenever we see numbers and the multiplication sign in describing designs like this (2 X 2 or “two by two”), each of the numbers represents an independent variable and the value of the number represents the number of levels of that independent variable. For example, 3 x 2 would mean two independent variables, one with three levels and one with 2. 3 X 4 X 2 would represent an experiment with three independent variables, one with three levels, one with 4, and one with 2. Researchers often use the term “way” to refer to the number of independent variables in an analysis, so a “one-way” ANOVA refers to an ANOVA with one independent variable, and a two-way ANOVA would be used to analyze an experiment with two independent variables. Mathematically, we can analyze data with as many independent variables as we want. However, due to the complexity of interpreting higher level ANOVAs, it’s rare to see anything beyond a three-way ANOVA.
Results and Analyses
An experiment that includes multiple categorical independent variables and one continuous dependent variable is appropriately analyzed using an analysis of variance (ANOVA). Statistically, in a two-way ANOVA there are three basic types of effects that are tested: main effect for independent variable A, main effect for independent variable B, and effect for the interaction of A and B. We will consider main effects first. A main effect for a given independent variable means that there is a significant difference between the levels of this independent variable across levels of the other independent variable. Another way of thinking of this is that there is an effect for one independent variable regardless of the the level of the other. For example, in the test experiment, students are probably going to do better on the easy test than the hard, regardless of the temperature of the room they are tested in (Figure 2 displays means that represent such an effect, and Figure 3 is a line graph of these means). The mean of the means for a given independent variable, collapsing across the levels of another independent variable, are referred to as “marginal means” and these means aid us in interpreting a main effect. So, if we look at the marginal means in this example we can see that there is clearly a dramatic difference between the marginal means associated with test difficulty (60 vs. 80) and no difference in the means associated with room temperature.
Room Temperature |
|||
Test Difficulty |
50 degrees |
90 degrees |
Marginal means |
hard |
60 |
60 |
60 |
easy |
80 |
80 |
80 |
Marginal means (room temperature) |
70 |
70 |
Figure 2. Means representing a main effect for Test Difficulty.
Figure 3. Line graph of means from Figure 2
We could, of course, find a main effect of Room Temperature, by reversing this hypothetical example, in which case we could, for example, change the means for both test difficulty groups in the 50 degree room to a score of 60, and for both test difficulty groups in the 90 degree room to a score of 80. The examples thus far might lead you to the conclusion that a main effect for one independent variable precludes the possibility of finding a main effect for the other. Actually this is not true as illustrated in the table and line-graph in figures 4 and 5. Notice that in this case, we would conclude from the results that students perform best in 90 degrees regardless of whether the test was hard or easy, and they also perform better on the easy test regardless of whether they take it in 50 degrees or 90. Note that the graphs of the means in both figures 3 and 5 represent parallel lines. Parallel lines in these types of graphs indicate that there are main effects in the results, but no interactions. If the lines are not parallel this is indicative of an interaction. (Note that it is possible to find both a significant main effect and a significant interaction with the same set of means, and in this case, the lines will not be parallel. In interpreting such a case, the main effect is usually ignored, in that it is misleading. We will address this further below.)
Room Temperature |
|||
Test Difficulty |
50 degrees |
90 degrees |
Marginal means |
hard |
60 |
70 |
65 |
easy |
70 |
80 |
75 |
Marginal means (room temperature) |
65 |
75 |
Figure 4. Means Representing a Main Effect for Both Room Temperature and Test Difficulty.
Figure 5. Line Graph of Means from Figure 4
Figure 6 and 7 represent the classic “crossing” interaction. This effect is an interaction because the effect of one independent variable depends on, the effect of the other. If we found these means, and we were asked a question about one independent variable, such as: “Did students do better with the hard test or easy test?”, our answer would be something like: “That depends, with room temperature of 50 they did better on the hard test, with room temperatures of 90 they did better on the easy test”. Conversely, if someone asked: “Did students do better in ninety degrees or 50 degrees?”, your answer would be: “That depends, with the hard test they did better in 50 degrees, with the easy test they did better in 90 degrees.” As you can see from the marginal means there is absolutely no main effects in this case. This illustrates the fundamental advantage of using a multi-way design and analysis. If we were to set up and experiment where we just compared hard vs. easy tests or 50 vs. 90 degree rooms, and students scored just as is illustrated below, we would never realize that the effect of one of these factors was dependent on the other. Likewise if we analyzed the present experiment, and students scored just as illustrated below, just using two t-tests, one for each of the independent variables, our conclusions would be quite different about the effect of these two independent variables, and our conclusions would be incorrect.
Room Temperature |
|||
Test Difficulty |
50 degrees |
90 degrees |
Marginal means |
hard |
80 |
60 |
70 |
easy |
60 |
80 |
70 |
Marginal means (room temperature) |
70 |
70 |
Figure 6. Means Representing “crossing” Room Temperature X Test Difficulty Interaction
Figure 7. Line Graph of Means from Figure 6
Although the classic “crossing” interaction in figures 6 and 7 is used most often to illustrate an interaction in a two-way analysis, it’s also possible to find an interaction in which the lines do not cross (though note they are still not-parallel), such as illustrated in figures 8 and 9. Note that, to explain the results, we would still have to describe the results of one independent variable in terms of the other. For example: “Do students do better on hard tests or easy tests?” “It depends, in a fifty degree room there is no difference, but in a ninety degree room they do much better on easy tests.” Note that this also represents the case that I referred to above in which the marginal means indicate that there are two main effects. When we average across effects for room temperature, the mean for the ninety-degree room is higher, and when we average across effects for test difficulty, the easy test scores are higher. However, clearly if we were to conclude from these results that students do better in ninety degree rooms, regardless of test difficulty, or that they do better on easy tests regardless of room temperature, we would clearly be incorrect. The correct conclusion to be drawn from the results below is that students do best when the test is easy and the temperature is 90 degrees. Otherwise temperature and difficulty level doesn’t matter. This is why researchers often disregard a main effect that occurs in the data analysis when there is also an interaction.
Room Temperature |
|||
Test Difficulty |
50 degrees |
90 degrees |
Marginal means |
hard |
60 |
60 |
60 |
easy |
60 |
80 |
70 |
Marginal means (room temperature) |
60 |
70 |
Figure 8. Means Representing a non-crossing Room Temperature X Test Difficulty Interaction
Figure 9. Line Graph of Means from Figure 8