Senses and attention

Vision is the door that costs the most. It’s the sense that takes up the most brain: something around 30% of the human cortex is visual territory, in a map that Van EssenVan EssenMapped the visual areas of the monkey and human cortex, and his is the count of how much of the cortex is vision’s territory.One reference in this work:2001Mapping visual cortex in monkeys and humans using surface-based atlasesSee in the bibliography → and colleagues drew in 2001 comparing monkeys and people. Territory, however, isn’t expenditure, and the energy bill is even tighter: firing a neuron costs so much that the cortex’s metabolic budget only allows fewer than 1% of them to be substantially active at the same time, as LennieLennieCalculated the energy cost of firing a neuron and concluded that the cortex’s budget only covers a tiny fraction of them active at the same time.One reference in this work:2003The cost of cortical computationSee in the bibliography → calculated in 2003. That’s thirty per cent of the territory, with less than one per cent switched on at any one time. That is the real ceiling behind “you can’t call attention to five things”. The limit is what the organ can pay for, not laziness on the part of whoever uses it.

To understand why we see only a fraction of what reaches the retina, it’s worth following the path of light. From the retina, the signals travel along the optic nerve to the lateral geniculate nucleusthalamusThe brain’s distribution hub: almost everything that comes from the senses passes through it before reaching the cortex. Smell is the odd one out, with a direct route to the areas of emotion and memory, and that is where the feeling-laden memory a scent pulls up comes from.See it in Senses and attention →See in the glossary →, in the thalamus, and from there on to the primary visual cortexV1The first area of the brain, right at the back of the head, to receive what the eye sent. It is where the image becomes contrast, edge and orientation, before you recognise what you are looking at.See it in Senses and attention →See in the glossary →, area V1, in the occipital lobe, where every pixel of what we see takes its first cortical form. Right after comes area V2V2The area right after V1. It joins the edges V1 detected, separates figure from ground and decides which elements form a group, all of it before you notice you looked.See it in Senses and attention →See in the glossary →, which expands the work of V1: it integrates edges, contours and spatial patterns, separates figure from ground and begins to recognise relations between the parts. It’s where the parts are assembled, before any conscious decision, and it’s there that it starts being settled which elements of your screen form a group and which don’t. Much of what Gestalt describes is decided at this stage and the following ones, V4 included, with feedback from the higher areas.

The executive attention network, anchored in the anterior cingulate and the lateral prefrontal cortexprefrontal cortexThe region behind the forehead, where the person decides, plans and sustains attention. It keeps one thing in focus at a time and inhibits the rest: it is where the bottleneck of conscious decision usually tightens.See it in Memory →See in the glossary →, is the one that resolves the conflict between the candidates and decides which of them stays, in the three-network model PosnerPosnerDescribed how attention works in a social world, and his is the base of much of what the site says about focus.2 references in this work:2012Attention in a social world1980Orienting of attentionSee in the bibliography → consolidated in 2012. To save energy along this route, evolution shaped filters that block almost all the excess, and only the most relevant signals actually reach consciousness. This bottleneck is neither recent nor negotiable: the basic organisation of the visual pathway, from the retina to the thalamus and to V1, is conserved across the whole primate lineage, and the skull of Homo sapiens was already within the current volume range three hundred thousand years ago, as NeubauerNeubauerCompared fossil skulls and showed that brain size reached today’s long before its shape did, globularity is recent.One reference in this work:2018The evolution of modern human brain shapeSee in the bibliography →, Hublin and Gunz measured in 2018. However much screens display feeds at high frequency, our visual resolution only responds to what this biological filter allows. It is exactly this limit that eye tracking makes visible, by showing what the person actually fixated and what they skipped without ever having seen it.

Sound takes a similar path. It enters through the ears, goes down the cochlea and activates the hair cells; the impulse continues to the cochlear nucleus, in the brainstembrainstemThe oldest part of the brain, at the junction with the spinal cord. It handles what is non-negotiable — breathing, waking, reacting to a fright — and it is where almost all the signals from the senses pass through.See it in Senses and attention →See in the glossary →, then to the inferior colliculussuperior colliculusA station in the brainstem that pulls the gaze towards what moves or contrasts, before the person decides to look. It is that reflex that every visual highlight hijacks.See in the glossary →, in the midbrain, to the medial geniculatethalamusThe brain’s distribution hub: almost everything that comes from the senses passes through it before reaching the cortex. Smell is the odd one out, with a direct route to the areas of emotion and memory, and that is where the feeling-laden memory a scent pulls up comes from.See it in Senses and attention →See in the glossary → body, in the thalamus, and finally to the primary auditory cortexA1The first area to receive sound, in the temporal lobe. It receives everything that comes in through the ears, but only one stream at a time gets priority and reaches consciousness.See it in Senses and attention →See in the glossary →, area A1, in the temporal lobe.

What happens when several sounds arrive at the same time was measured in 1953, in the experiment that became known as the cocktail party effect: CherryColin CherryRan the cocktail party experiment: how the ear picks one voice among several and what happens to the ones left out.One reference in this work:1953Some experiments on the recognition of speech, with one and with two earsSee on Wikipedia ↗See in the bibliography → showed that, among several voices, we choose only one to hear consciously. Magnetoencephalography later confirmed that attention already amplifies the chosen stream inside the auditory cortex itself, between 20 and 50 milliseconds after the sound, before any conscious decision, as WoldorffWoldorffMeasured that auditory attention amplifies the signal as early as the first 20 to 50 milliseconds, before any conscious decision.One reference in this work:1993Modulation of early sensory processing in human auditory cortex during auditory selective…See in the bibliography → and colleagues recorded in 1993. It’s this resolution in milliseconds that magnetoencephalography and EEG deliver, and that no other method delivers. The range the ear hears is old too. From the fossilised ossicles of five individuals from Atapuerca, MartínezMartínezMeasured the inner ear of 350,000-year-old fossils and showed that the human hearing range was already ours.One reference in this work:2004Auditory capacities in Middle Pleistocene humans from the Sierra de Atapuerca in SpainSee in the bibliography → and colleagues reconstructed in 2004 the transmission of sound through the outer and middle ear, and it already followed the human pattern in the middle range of speech, 2–4 kHz, and not the chimpanzee’s. In the 3.5–5 kHz range the picture is different: there the ones from Atapuerca were not yet modern, and the ones who get there are the Neanderthals, as Conde-ValverdeConde-ValverdeOne reference in this work:2021Neanderthals and Homo sapiens had similar auditory and speech capacitiesSee in the bibliography → and colleagues showed in 2021. The fossils were dated again after publication and are at least 430,000 years old, older than our own species. However many audio channels an app offers, we remain limited to a single conscious channel.

Touch enters through the skin and climbs. When we touch something, mechanical receptors generate impulses that reach the dorsal root ganglion and travel, through the spinal cord, to the thalamusthalamusThe brain’s distribution hub: almost everything that comes from the senses passes through it before reaching the cortex. Smell is the odd one out, with a direct route to the areas of emotion and memory, and that is where the feeling-laden memory a scent pulls up comes from.See it in Senses and attention →See in the glossary → and to the primary somatosensory cortexS1The map of the body drawn over the cortex. Fingers, lips and tongue take up enormous areas; back and legs, almost nothing: that is why the fingertip tells apart what the elbow cannot.See it in Senses and attention →See in the glossary →, S1, in the postcentral gyrus. There are four types of receptor, Merkel discs, Meissner corpuscles, Ruffini endings and Pacinian corpuscles, and the map they form in S1 is a mirror of the body’s surface, only a distorted mirror: hand and lips take up a disproportionate slice of the cortex, while the whole trunk fits into a small patch. That’s how PenfieldWilder PenfieldStimulated the cortex of awake patients during surgery and drew the homunculus, the body as the brain represents it.One reference in this work:1937Somatic motor and sensory representation in the cerebral cortex of manSee on Wikipedia ↗See in the bibliography → and Boldrey drew the homunculus, in 1937, by directly stimulating the cortex of awake patients.

The same disproportion shows up in the skin, when you count. The hairless hand, palm and fingers, has about 17,000 mechanoreceptive units, and they aren’t evenly distributed: at the fingertip the density reaches about 240 per square centimetre, against about 60 on the palm, in the count JohanssonJohanssonCounted, with Vallbo, the mechanoreceptive units of the human hand one by one, and measured the density of each type in each region of the glabrous skin.One reference in this work:1979Tactile sensibility in the human hand: relative and absolute densities of four types of me…See in the bibliography → and Vallbo made in 1979 recording fibre by fibre. Pacinian corpuscles, the type that responds to vibration, have their peak sensitivity around 250 Hz, and that’s why phone haptic motors are built to resonate between 175 and 235 Hz. The range didn’t come from an engineering choice. It’s the frequency the body feels best.

Smell and flavour enter through narrower doors. Smells are picked up by receptor cells in the olfactory mucosa and sent directly to the olfactory bulb, which projects to the piriform cortex and to limbic structures such as the amygdalaamygdalaTwo small structures in the temporal lobe that decide, in milliseconds, whether something is a threat or an opportunity. They respond to your screen before any conscious judgement about it.See it in Emotion →See in the glossary → and the hippocampushippocampusThe structure that turns recent experience into lasting memory. Everything your interface asks the person to remember tomorrow passes through it today.See it in Memory →See in the glossary →. That’s why certain aromas immediately evoke an emotion or a memory: smell is the only sense that reaches the limbic systemlimbic systemAn inherited name for a group of structures — amygdala, hippocampus, neighbours — that the design literature still treats as a single block. Neuroscience does not treat it that way, because there is no anatomical boundary and no common function, and each of these structures does different things.See it in Emotion →See in the glossary → at the very first stop, without passing through the thalamusthalamusThe brain’s distribution hub: almost everything that comes from the senses passes through it before reaching the cortex. Smell is the odd one out, with a direct route to the areas of emotion and memory, and that is where the feeling-laden memory a scent pulls up comes from.See it in Senses and attention →See in the glossary → first. Even with about half of our olfactory receptor genes inactive, we keep something around 400 functional ones, in the count MalnicMalnicOne reference in this work:2004The human olfactory receptor gene familySee in the bibliography →, Godfrey and Buck published in 2004.

Taste depends on the taste buds, which transmit sweet, salty, sour, bitter and umami to the solitary tract, in the brainstembrainstemThe oldest part of the brain, at the junction with the spinal cord. It handles what is non-negotiable — breathing, waking, reacting to a fright — and it is where almost all the signals from the senses pass through.See it in Senses and attention →See in the glossary →, from where the signals rise to the thalamus and reach the gustatory cortex, in the insular lobe. The resolution of smells and flavours remains low, and not even the structure of a molecule predicts well what it will smell like, as KellerKellerMeasured the perception of chemically diverse molecules and showed how much the human sense of smell discriminates, and how much it fails to describe.One reference in this work:2016Olfactory perception of chemically diverse moleculesSee in the bibliography → and Vosshall measured in 2016 with hundreds of compounds. Describing a smell in detail is nearly impossible, and what fails there is naming, not memory, as OlofssonOlofssonExplained why it is so hard to name a smell: the sense of smell has a weak link to the language system.One reference in this work:2015The muted sense: neurocognitive limitations of olfactory languageSee in the bibliography → and Gottfried showed in 2015. Any attempt to digitise odours runs into a bandwidth far smaller than that of video or audio.

To keep our balance and perceive movement, we rely on the semicircular canalssemicircular canalsThree rings of fluid in the inner ear, one for each axis of rotation. When the head turns, the fluid takes a moment to follow, and it is that delay the body reads as “I turned”.See it in Senses and attention →See in the glossary → and the otolithsotolithsTwo pouches with calcium crystals on a membrane, also in the inner ear. They register acceleration in a straight line and the direction of gravity, which way is down.See it in Senses and attention →See in the glossary → in the labyrinth of the inner ear. The first register rotation; the second, straight-line displacement and the direction of gravity. Both send signals to the vestibular nucleus in the brainstembrainstemThe oldest part of the brain, at the junction with the spinal cord. It handles what is non-negotiable — breathing, waking, reacting to a fright — and it is where almost all the signals from the senses pass through.See it in Senses and attention →See in the glossary → and, from there, to the cerebellumcerebellumThe structure low at the back of the skull that automates movement and sequence. The more someone uses your product, the more the operation migrates here, and the costlier it gets to change what has already become automatic.See it in Memory →See in the glossary → and the vestibular cortex.

And there’s something here with no parallel in the other senses: they fire all the time. Even with the head still, a semicircular canal fibre keeps up about ninety impulses per second. There is no off position. Movement doesn’t make the signal appear. It makes that rate rise on one side of the head and fall on the other.

The brain is constantly comparing what the eye reports with what the inner ear reports, and it counts on the two saying the same thing. When they disagree, you get sick. And the two ways of making them disagree are symmetrical. In the car, looking at your phone, the ear says you’re moving and the eye sees a page standing still. In a virtual reality headset, sitting down, the eye sees you walking down a corridor and the ear says nobody has moved. It’s the same disagreement, with the roles swapped, and that’s why looking out of the side window helps in one case and closing your eyes helps in the other.

On a screen, what produces the disagreement is delay: you turn your head, the inner ear registers it instantly, and the image arrives a few moments later. Note that the delay isn’t between one sense and another. It’s between the movement and the drawing, and it’s the late arrival of the drawing that makes the two senses tell different stories.

It’s worth noticing where the numbers that get repeated about this come from, because they are a textbook case of a number changing owners along the way. LaViolaLaviola Jr.Reviewed what is known about sickness in virtual environments, the mismatch between what the eye sees and what the inner ear feels, and what it costs whoever uses them.One reference in this work:2000A discussion of cybersickness in virtual environmentsSee in the bibliography → describes delay as a cause, in 2000, and gives no threshold at all: in the section on delay there isn’t a single millisecond, and the display number he does give is 30 Hz, about flicker, which is another problem. Thirty hertz is the common video rate, and flicker is that shimmer a fluorescent lamp makes in the corner of your eye.

The famous “under 20 milliseconds” is a design recommendation John Carmack made in 2013, and it became physiological fact along the way. For a ruler: at 60 frames per second, each frame lasts 16.7 ms. In other words, the recommendation is “delay shorter than one frame”, something anyone who has ever tuned an animation knows by the name of dropped frame.

What was actually measured is something else. People detect delay below 17 ms, and one participant detected 3.2 ms; the studies that induced sickness on purpose, meanwhile, worked with delays from a few tens of milliseconds up to 340 ms, in the review StauffertStauffertReviewed what was actually measured about latency and sickness in virtual reality, and separated what is a measured threshold from what is a design recommendation turned into fact.One reference in this work:2020Latency and cybersickness: impact, causes, and measuresSee in the bibliography → and colleagues published in 2020. That is, between two and twenty frames. Detecting isn’t getting sick, and the two numbers keep being swapped for one another, even though the two ranges touch at the bottom end.

This seemed to be the rare case where permanence can be measured in fossils, because the bony labyrinth is bone and bone survives. I went and checked, and the finding says otherwise: the labyrinth of the Atapuerca hominins has a derived pattern of semicircular canal proportions, characteristic of the Neanderthal lineage, and not the modern geometry, as QuamQuamReconstructed the bony labyrinth of the Atapuerca hominins and found a derived pattern, of the Neanderthal lineage, not the modern human geometry.One reference in this work:2016The bony labyrinth of the middle Pleistocene Sima de los Huesos hominins (Sierra de Atapue…See in the bibliography → and colleagues described in 2016. So this sensor doesn’t serve as an example of “the same for hundreds of thousands of years”. The one that does is the ear, whose fossil ossicles already transmitted sound the way ours do.