Question Browser

Common student errors for parts (a) and (b) are discussed below. However, there were two general misunderstandings that affected student responses of both parts of this question.

Common Core standard S-IC.3 indicates that students are expected to recognize the purposes of and differences between observational studies and experiments and to understand the role of randomization in each. This means that students should be looking for random selection in a well-designed observational study (so that it is likely that a sample is representative of the population) and for random assignment in a well-designed experiment (so that cause-and-effect conclusions are reasonable). Many student responses did not demonstrate an understanding of these ideas. This conceptual error is illustrated in the following two student responses, which indicate a lack of understanding of the difference between an observational study and an experiment and which fail to distinguish between random selection and random assignment.

Some students did not understand that the wording “Would either of the studies…” meant that they could say yes for one of the Studies and not for the other. These students attempted to answer either yes for both studies or no for both studies for each part of the question. The following is typical of student responses who interpreted the questions posed in this way.

Part (a): Recognize when it is reasonable to draw a cause-and-effect conclusion based on the design of a statistical study.

Some students indicated an understanding that Study 2 was an experiment and that it is only reasonable to reach a cause-and-effect conclusion based on a well-designed experiment. However it was common for these students to provide a weak or incomplete conclusion. The most common mistake here was to not specifically comment on the random assignment of subjects in Study 2. The following two student responses were scored as partially correct for part (a) and illustrate this error.

A relatively common student misunderstanding was to confuse statistical significance with causation. Because the question wording used “significantly lower,” these students used statistical significance as the justification of a causal conclusion. This misunderstanding is illustrated in the following two student responses, which were both scored as incorrect for part (a).

Another misunderstanding is illustrated in the following student response. This response indicates that the student believes that it is random selection (rather than random assignment) that is needed in order to draw a causal conclusion. For this reason, the student incorrectly selects Study 1 in the response to part (a).  

A few students thought that the conditions that would allow a cause-and-effect conclusion to be drawn were the conditions needed in order for use of a formal inference procedure (such as a two-sample t-test). Students making this mistake focused on the inference conditions rather than the study design in their responses. 

Finally, some students did not recognize that Study 2 was an experiment. This is illustrated by the following student response, which was scored as incorrect for part (a).

Part (b): Recognize when it is reasonable to generalize from a sample to a larger population based on the design of a statistical study.

The most common student error in part (b) was failure to specifically appeal to random selection as the explanation for why results could be generalized from Study 1. For example, the following student response was scored as only partially correct because it does not mention that the samples were random samples. If the response had used “random samples” rather than “samples” in the last sentence of the response, it would have been scored as essentially correct.

The other two common student errors on part (b) were related to thinking that sample size is what determines whether or not a sample is representative of the population. Some students indicated that they believed that results could be generalized because the sample sizes were large. Others indicated that the results could not be generalized because the samples were not large enough. But in both cases, students making either of these two errors failed to recognize the importance of random selection.

Each of the following three student responses indicate that generalization is justified because the sample sizes are large, and fail to recognize that the reason that generalization can be justified is that the samples were randomly selected.

Each of the following three student responses indicate that generalization is not justified because the sample sizes are too small, and fail to recognize that generalization can be justified because the samples were randomly selected.