A plan to replicate the Loftus and Palmer (1974) experiment on false memories

Rolf Zwaan writes:

Suppose you and a friend witness a car crash. After a while, the police arrive. One officer turns to you and asks, “How fast were the cars going when they smashed into each other?” Another officer speaks to your friend and asks, “How fast were the cars going when they hit each other?” Could the phrasing of those questions influence the speed estimates you and your friend provide? A classic psychology study suggests it can. . . . Loftus and Palmer (1974). Participants watched short videos of car crashes and were later asked to estimate the vehicles’ speed. Those who were asked how fast the cars were going when they smashed into each other gave higher speed estimates than those asked how fast they were going when they merely hit. A week later, participants in the “smashed” condition were also more likely to falsely recall having seen broken glass at the crash site. . . . This study helped launch an entire research tradition on the malleability of memory . . .

Now comes the plan:

Despite its iconic status (and considerable citation count), we still don’t know how large or reliable the effect is by today’s standards of research transparency, statistical power, and replication. That’s a gap worth addressing–not to question the legacy of the original study, but to better understand its robustness.

Where I disagree

I kinda see what Zwaan’s saying, but I have two concerns with how he frames the problem:

1. He writes, “we still don’t know how large or reliable the effect is by today’s standards of research transparency, statistical power, and replication.” I think he could say, “we still don’t know how large or reliable the effect is,” full stop. When it comes to publication practices, standards of research transparency, statistical power, and replication have changed a lot in the past fifty years. But an effect is an effect, and it’s a legitimate question: How large or reliable is the effect?

Also, I’d ask, “How large or variable is the effect?” because I find it more useful to think in terms of variability (which is defined in terms of the underlying process) rather than reliable (which is more of a statistical concepts), but that’s a minor point.

2. He writes that this is worth studying “not to question the legacy of the original study, but to better understand its robustness.” I don’t get this.

First, if it turns out that, under a variety of conditions, the result does not replicate, then, yeah, it should call into question the legacy of the original study! Just as, if the results do replicate, this project would support that legacy.

Second, if you say that the goal is “to better understand its robustness,” you’re already implicitly accepting that the original published result was correct, that it uncovered a real effect. And maybe it did! But, if it didn’t, then it’s not an issue of “robustness”; it’s a problem of analysis and interpretation of the original data.

Again, I’m not saying that I think the Loftus and Palmer (1974) study was no good, or that I don’t believe its results, or that I don’t think it will replicate, or whatever. I have no idea. My only point here is that, if you’re going to think about a replication, either a hypothetical replication or, in this case, a series of real replication studies, you should be open to the possibility that the original study was flawed and the original study was mistaken. I’m not saying you should assume this, just that you should allow it as a possibility. And I don’t think Zwaan is allowing that in his framing above. I think he’s being too deferential.

What’s happening next?

Zwaan and his colleagues are organizing a large-scale replication study, and the author of the original 1974 paper is still around and, according to Zwaan, “has expressed enthusiastic support . . . and is willing to help us reconstruct the stimuli as faithfully as possible.” That sounds great.