In 2012, media headlines trumpeted the findings of a study that suggested humans can do basic mathematics and reading with their unconscious minds — without consciously trying to understand or solve a problem. But now a follow-up study is casting doubt on the paper’s findings.

As part of the original study in the Proceedings of the National Academy of Sciences (PNAS), scientists flashed one of the participants’ eyes with distracting colorful shapes whilst presenting the other eye with math problems or words. Because people are so focused on the shapes, the technique — known as continuous flash suppression — is supposed to make the volunteer unconscious to what appears before the other eye.

In one experiment, Ran Hassin and colleagues at the Hebrew University of Jerusalem in Israel displayed mathematical sums followed by a number to participants (using the eye that was not already distracted by colorful shapes). The researchers found that people were quicker to announce that number if it was the correct answer to the sum, suggesting they worked it out subconsciously.

Similarly, researchers presented the distracted participants with short sentences, and as New Scientist reported, people were quicker to point out nonsensical phrases, suggesting, again, that they subconsciously processed the words.

However, writing in Psychonomic Bulletin Review earlier this month, researchers Pieter Moors and Guido Hesselmann question the original study’s plausibility, noting that its claims are not fully supported by their own study. Moors and Hesselmann — based at the University of Leuven in Belgium and Charité-Universitätsmedizin in Berlin, Germany, respectively — used their own analysis tools to evaluate the information from Hassin’s study. They told us:

In our study, we question the plausibility of the claims made in the original paper on several different grounds. First, based on the paradigm used in this study (continuous flash suppression) one could already question whether such extensive processing is plausible. That is, research has shown that invisible stimuli are suppressed at very early stages of visual processing, stages too early for arithmetic to be possible. Thus, solely based on the paradigm used, we already thought that the findings were somewhat implausible.

They added that they had found statistical flaws in Hassin and his colleagues’ paper:

Nevertheless, if this finding would be observed over and over, we would have to adjust our thinking about how this suppression method works. Second, we evaluated the evidence presented in the paper at a statistical and theoretical level. Statistically, the paper relied on an analysis procedure that was not necessarily optimized for the experimental design.

Moors and Hesselmann concluded that their reanalyses made the idea that we can do math unconsciously “much less convincing as was presented in the original paper.” However, they say the study’s authors used methods that are more or less standard in their field, and therefore did the right thing. The original paper was also successfully been replicated by other researchers last year in a study published in the European Journal of Social Psychology using the same materials.  

So Moors and Hesselmann decided to take a closer look at the replication, which reported evidence for unconscious addition (but not subtraction), and found a critical coding error. After fixing that error, the effect for subconscious adding or subtracting no longer stood. As Retraction Watch reported, the repeat study has since been retracted by its authors.

For Moors and Hesselmann, this was further evidence that there’s inconclusive evidence for the unconscious arithmetic effect.

We contacted Hassin for his take on Moors and Hesselmann’s paper. He told us:

My colleagues and I have been paying very careful attention to this paper. We gave our full data to the authors, and we thank them for investing their precious time in re-analyzing our data. We are happy to encourage and facilitate every effort at examining our findings – after all, science is a social, cumulative endeavor.

As we see it, the more central analyses in the paper provide evidence for our conclusions, not against them (in the words of the authors: “although Bayes factors revealed evidence for the presence of a priming effect, it was generally weak”). Some analyses reported here are based on predictions we would not make, and are hence no-informative, and the relevance of other analyses is in question.

We continue to stay engaged with this work and are ready to potentially adjust our conclusions as more data and analyses become available.

Hassin seems to pointing to the fact that Moors and Hesselmann are not actually refuting his results, but only saying they are “weak.”

Overall, Moors and Hesselmann say they weren’t surprised that PNAS accepted the original paper, noting that they think the peer review system (which has received much criticism in recent years) worked adequately in this case. They added:

…we think our study is part of some kind of self-correction process in which high-profile findings are scrutinized and evaluated, and alternative analyses and methods for evaluating these claims are proposed. If these yield different results (as in our paper), this is just a “part of the puzzle” based on which researchers themselves have to evaluate how strong the evidence is for the claim that arithmetic can happen unconsciously.