Where evidence runs out

The most common reading of a brain-imaging study is also the one the method does not allow: region X activated during the task, therefore the mental process associated with X was happening. The reasoning inverts the arrow. One region takes part in many different functions, the insula shows up in disgust, pain, empathy and decision under risk, so its activation does not identify which of them occurred. This is reverse inference, and it only holds when the region is highly specific to that process, which is almost never the case, as PoldrackRussell PoldrackNamed the problem of reverse inference: seeing an area light up does not allow concluding which mental process is under way.One reference in this work:2006Can cognitive processes be inferred from neuroimaging dataSee on Wikipedia ↗See in the bibliography → showed in 2006 in the paper that became the reference on the subject. In practice this changes the sentence you are allowed to write: “the reward circuitnucleus accumbensThe centre of the reward circuit. When something goes right — a button that responds, a task that closes — it is what records “worth it, do it again”.See it in Emotion →See in the glossary → was engaged” is defensible; “the person felt pleasure” is something else.

MarekMarekShowed that studies linking brain to behaviour need thousands of participants to be reliable, and almost none has them.One reference in this work:2022Reproducible brain-wide association studies require thousands of individualsSee in the bibliography → and colleagues analysed, in 2022, datasets with thousands of participants and showed that one specific kind of result only stabilises with samples in the thousands: correlating a brain measure with a behavioural trait across different people. Below that, the effects that appear tend to be inflated and not to repeat in the next study. Comparing two conditions inside the same person needs far fewer people, and that is the design of a good part of the studies cited here. What still has nothing holding it up is the next jump, the one from a few dozen participants in a laboratory to a claim about people. The studies of neuroscience applied to interfaces that I read almost all run with a few dozen people, and that does not make them useless. It makes each finding a first piece of evidence, not a constant of nature. The honest reading of a study that size is “this happened in this group, in this task”, and the design decision that leans on it should be reversible.

Deena WeisbergWeisbergShowed that a bad psychological explanation becomes more convincing when a mention of the brain that explains nothing is added.One reference in this work:2008The seductive allure of neuroscience explanationsSee in the bibliography → and colleagues gave, in 2008, psychological explanations to lay readers, some with a piece of logically irrelevant neuroscientific jargon grafted into the middle, and the bad explanations became considerably more convincing with the graft. With brain images the equivalent finding is McCabeMcCabeShowed that a brain image makes an argument look more scientific, even when it adds nothing. The finding did not replicate later.One reference in this work:2008Seeing is believing: the effect of brain images on judgments of scientific reasoningSee in the bibliography → and Castel’s, from the same year, and it deserves the caveat I demand of others: it did not replicate in later attempts, as MichaelMichaelTried to replicate the brain image effect in ten experiments and did not find it. It is the correction of McCabe’s finding.One reference in this work:2013On the (non)persuasive power of a brain imageSee in the bibliography → and colleagues showed in 2013, while the jargon effect did. That makes the warning more precise, not less, because what fools you is not the picture, it is the word. It applies to me as much as to anyone else. There is a brain turning on the first screen here, and it is an illustration, not evidence of anything. If any argument on these pages convinced you because of it, the argument was not ready.

The three limitations hold for the whole field, and I turned them into a mark you find on each of the 38 laws: a PS attached to the block about the brain, saying one of two things. Either the effect and the structure were measured in the same task, which is the case for three laws and the hardest level to reach. Or the structures are real and well established but were measured in another phenomenon, and the bridge between the two is mine; there are twenty-eight of those, and the PS comes with an asterisk. The other seven have no brain block at all, and in its place comes a section called “Why there is no neuroscience here”, because a replicated behavioural effect does not become truer with an anatomical structure hanging off it, and it becomes less defensible. Whoever checks the structure and does not find it will discount the rest along with it. That is why there are seven of those, and they are the part of these 38 I like most.

There is a limit none of the three reaches, because it is not about method but about sample: knowing how much an effect was measured does not say in whom. They are two different questions, and for a long time I only asked the first. Fitts’s LawFitts’s lawAn important target is a big target, close to where the hand already is.Open the law → is the best-measured law here and it held for two of the six adults with cerebral palsy who were tested on it; one of the three laws where the effect and the structure were measured in the same task rests on a study run in the Netherlands. None of those studies lied, they all said who they measured. What gets lost is the road between the study and the drawing board, and that is where the population disappears from the statement. Where this second question leads is in The body, and its map is the matrix of 38 laws by six dimensions. The six studies that measure famous laws outside the body they were born in are discussed in References.

None of these limitations recommends abandoning evidence; all of them recommend declaring its reach. BrouwerBrouwerWrote the six recommendations for not going wrong when using a physiological signal as a measure of mental state. It is the method yardstick of the Limits page.One reference in this work:2015Using neurophysiological signals that reflect cognitive or affective state: six recommenda…See in the bibliography → and colleagues summarised that caution, in 2015, in recommendations that hold for any physiological signal: say what the measure captures and what it does not distinguish; compare conditions against each other instead of treating the absolute value as a diagnosis; use more than one measure, because each one’s weaknesses are different; and do not treat self-report and physiology as if one validated the other, because when they diverge, the divergence is usually the finding. It is the same discipline I ask for on the ethics page, applied now to reading the data instead of using it. And if you are still wondering how far to trust what is written here, the answer is the one I demand of any law: on what was measured, a lot; on what is my bridge, as much as I declared. When you cite one of these laws in your next meeting, which side of the PS does it come from?