TL;DR: Giving a control group something to do does not rule out placebo effects if treatment and control groups expect different benefits. In two surveys, people's expectations about particular games and tasks resembled published training results, even though the respondents had not undergone the training.
Better scores don't tell us why
Suppose people finish a program intended to improve attention or memory and score better on a follow-up test. Before crediting the training, we need to know what else could explain the change.
They may have benefited from practice with the test, or contact with researchers may have changed their engagement. Believing that the training should work could also affect how they approach the final assessment. A comparison group helps us sort out these explanations, provided it accounts for the things that could have changed performance.
In our 2013 paper, we paired an argument about study design with two surveys of expectations about video game training. We questioned the common assumption that giving the comparison group another activity is enough to account for placebo effects. Researchers also need to check whether the groups expect comparable benefits on the outcomes being measured, a requirement often overlooked.
What an active control leaves open
An active control group does an alternative activity instead of waiting or receiving no intervention. One group in a game-training study might play an action game while another plays Tetris for the same amount of time. Both groups get practice with a game and contact with researchers.
Participants still know which game they played. A fast, visually demanding game may seem more likely to improve attention than a slower one. If that expectation changes effort or performance on the final test, it could help explain a difference between groups that researchers attribute to the training itself.
Figure 1 of the paper, reproduced above, shows the checks involved. For an intervention study, it asks whether there is a control group, whether that group is active, and whether expectations match for each outcome. Checking expectations addresses one possible explanation for improvement; the experiment still has to meet the other requirements for a sound design.
Two surveys of expected improvement
We recruited 200 U.S. adults for each of two online surveys through Amazon Mechanical Turk. We randomly assigned participants to watch a brief video of the action game Unreal Tournament or a game commonly used as a control. One survey used Tetris as the control, the other The Sims. Each person saw only one game, so they could form different expectations without ever comparing the alternatives directly.
Next, participants read descriptions and watched videos of cognitive and perceptual tasks used to evaluate training. They reported whether they thought training on the game would improve their own performance on each task. Both the game videos and the task videos lasted approximately 30 seconds, and we randomized the task order.
We asked about expected benefits; these respondents did not complete a training program and then take performance tests. Their answers showed whether games used as control activities seemed as promising as the experimental game to people considering their possible benefits.
Expectations depended on the game and the task
For the useful field of view, a measure of visual attention, 83% of respondents who watched Unreal Tournament expected improvement, compared with 45% who watched Tetris. In the separate survey comparing Unreal Tournament with The Sims, the percentages were 75% and 38%. Both differences were statistically significant (p < .01).
For mental rotation, the pattern went the other way. This task involves imagining an object turned to a different orientation. Among respondents who watched Tetris, 89% expected improvement, compared with 69% who watched Unreal Tournament (p < .01). The game designated as the control could inspire higher expectations when its demands seemed to fit the task.
Expectations for story memory also differed. Unreal Tournament and Tetris did not differ significantly, while The Sims elicited higher expectations than Unreal Tournament. People did not expect an action game to improve everything. A lack of advantage on one task therefore cannot rule out different expectations on another.
These expectations resembled selected results from earlier game-training studies, which reported action-game advantages on visual-attention measures and a Tetris advantage on mental rotation. That resemblance makes expectations a plausible alternative explanation for the training results. It does not show that placebo effects caused those results, or that game training cannot improve cognition. People can hold different expectations without those expectations changing their performance.
How to check expectations
Researchers can start before choosing their control activity. A separate pilot sample can learn about the candidate activities and rate their expected benefits on each outcome. Researchers can then use those ratings to choose controls and measures with comparable expectations. Equal time or similar engagement alone cannot establish that match.
Another option is to choose an outcome for which people expect the control activity to help more. If the experimental training produces greater gains despite that expectation, the expectation difference would have difficulty explaining its advantage. We also propose manipulating expectations separately within treatment groups, when ethically appropriate, to distinguish effects of the activity from beliefs about its effectiveness.
A brief preview may miss beliefs that develop over extended training. Asking people about expectations during a study may change those expectations. And if expected improvement correlates with measured improvement, people may be recognizing a real benefit rather than producing it through belief.
Our surveys concerned gaming expectations, but the design problem also applies to research on cognitive training, education, and psychological treatment. Placebo effects can produce meaningful benefits and deserve study in their own right. To find out which parts of an intervention produce improvement, though, researchers have to distinguish those effects from expectations about them. An active control has to be checked against the particular outcomes and causal claims of the study.
Full Citation
Boot, W. R., Simons, D. J., Stothart, C., & Stutts, C. (2013). The pervasive problem with placebos in psychology: Why active control groups are not sufficient to rule out placebo effects. Perspectives on Psychological Science, 8(4), 445-454. https://doi.org/10.1177/1745691613491271