III. ASSEMBLING THE VISUAL IMAGE
29. INTERPRETING THE RETINAL IMAGE
So far, we have looked at how the visual system analyzes the optical image to produce brightness, color, and directional detail using receptive fields of cortical neurons. Yet what we see is more than these effects projected to different positions on some internal screen for us to view. Our visual system is the viewer. It recognizes objects at various places relative to each other within a scene. That requires substantial further organization of the information extracted from the retinal image.
We have only an inkling of how the visual system assembles that view from shape-specific single neurons, clusters of line-specific neurons, and combinations of spatial-frequency neurons. This is the Binding Problem. (More about that later.) While those neural networks have yet to be determined, experiments on visual perception have uncovered many principles used to do the organizing.
We know the pathways that connect the various areas in the occipital lobe with the temporal, parietal, and frontal lobes. They have been mapped anatomically with the aid of stains and dissection. EEG (electroencephalograph) can follow the rapid changes in neural activity, but this method can not locate them precisely enough to determine their exact location. fMRI sensing can highlight where such activity occurs, but it is too slow to follow how that activity changes (further explanation coming).
Since the 1850s, perceptual experiments have produced a massive literature of measurements connecting what people are shown with what they report seeing. The results reveal many operating principles used in assembling the visual image. They provide clues to the physiological basis of how that assembly is performed.
Some of these principles reflect the system’s evolution in a world where physics determines how surfaces around us are located, shaped, and reflect light: 1. Daylight comes from above. 2. Life forms tend to be symmetric wholes compared to non-life forms. 3. At eye level, curvature of the earth places the horizon about 5 kilometers away. 4. Most stuff retains its shape and position unless acted upon, etc. The following pages describe some operating principles that use these properties of the world.
Pictorial Cues to Distance
Why are distant shapes small? Because the eye focuses more distant objects into smaller retinal images:
That this question seems silly indicates how totally we take for granted that the picture in our head is “out there”.
At around 5 months, we begin to relate change in the visual size of a familiar object to the physiological changes that accompany the eyes’ accommodation, vergence and binocular disparity responses.[1]
Thereupon, we infer that a smaller image size is due to the distance of that object.
The couple down this road are small, but they do not look like small people, even though there are no physiological cues to their distance. That is because there are numerous pictorial cues in the visual image itself that are also associated with the couple’s distance. The following describes some of these.
1. Consistent with the image size of distant objects, the road gets smaller towards the middle of the photo. Since it is the same road, its decreasing size indicates the road leads away from our view. Another example is the popular photo of railroad tracks coming together near the horizon. This cue is called linear perspective. The design of the Parthenon shows that the ancient Greeks understood this cue.
2. Another cue also depends on this size-distance relationship. Note that gravel and stones along the road become smaller further towards the middle. This texture gradient cue indicates distance because we assume them to be of similar size along the road. Therefore, the smaller ones must be more distant. Here is another example:
3. A totally different distance cue in the above photos is so obvious that it is often overlooked as a distance cue. The road clearly goes downhill. Yet is it not paradoxical that its end is higher in the photo than where it begins in front? It is simply the case that higher objects in our view of a landscape or any scene are usually further away.

Here is a simple example of this horizon–relative cue. Involving both maturation and experience, it develops later than the distance-size cues at about 7 months.

Note how horizon closeness tends to override the effect of size on distance. That makes sense because objects can have any size while located at any distance. Closeness to the horizon depends only on distance’
4. Note also the blueish color and indistinctness of the distant mountains in these photos. This aerial perspective is a distance cue that arises from the air scattering photons in the atmosphere. (See Ch 6.)
However, the most powerful distance cue of all is only suggested in the above photos.
5. Interposition. A shape that overlaps another shape is seen as closer:

This cue is stronger than both the size and relative height cues, though they weaken the distance effect when they are contrary.
6. Shading on surfaces and shadows cast by shapes are cues that produce a three-dimensional effect. These depth cues give shapes the appearance of solid objects.

The visual system extracts some 20 pictorial cues to the distance and the 3-dimensional properties of the flat shapes in the retinal image. That these visual processes effectively convey distance effects from flat images on paper and your computer screen reflects their effectiveness for producing them in your head from the “flat” optical image on your retina.
Size Perception
Inferences of distance are based on known physiological processes plus many pictorial cues. Geometry determines that photons from larger shapes get focused into larger retinal images. However, the perception of a shape’s size is another matter. Inferences of size can’t be based on the size of a retinal image because that varies with distance. Therefore, size perception must depend on pictorial cues in the retinal image – plus cognition.
As you probably suspected, these two squirrels are the same size. Yet distance cues indicate that one is further away. Even when not backed up by physio-logical cues, the pictorial distance cues override our beliefs about giant squirrels. It is impossible to see that the squirrels are the same size.
The moon illusion is the most widely known example of the visual system adjusting size perception based on a shape’s apparent distance. The moon’s retinal image is the retinal image is the same size on the horizon and over-retinal image is the same size whether it is on the horizon or overhead. (See this yourself by comparing its size with the size of your pinky nail.) [2]
Yet, the small size of the couple’s image just down the road does not make them look small. Evidently, something else is also happening in this picture:
Size constancy refers to the visual system taking into account a shape’s distance together with knowledge about its size when it is nearby. Two pictorial cues support this constancy in addition to cognition: 1. The couple’s relative width compared to the road is the similar to that of the nearby person, 2. Their relative height compared to grass along the road is similar to that of the nearby walker. That is not so for the giant squirrel.
This simple drawing further shows that size is a relative perception.
The center circles are the same size. Though you probably suspected this too, the larger appearance of the one surrounded by small circles is difficult to overcome.
In some circumstances, size relativity can create an illusion. Yet this is no more surprising than the local adaptation effect noted in brightness perception (p. 66). The exaggeration is generally beneficial because the process aids size discrimination. The slight inaccuracy is rarely a problem in natural situations.
The relative nature of size perception is also illustrated by our tendency to see vertical distances as longer than horizontal distances.

Close one eye and look closely at this figure. The lines are now equal. Our binocular visual field is about 50% wider than it is high. Relative to this field, the vertical line extends across a larger portion than the horizontal line:

The subjectiveness of size is evident in many other ways, including a tendency for bright shapes to look larger than dark shapes – the irradiation illusion.
Shape Constancy
Distant objects produce retinal images whose exact shape is analyzed into specific lines and spatial frequencies in cortical maps. Comparing the response of these particular neurons with neural responses from memory enables recognition. As we move the same object produces a slightly different image in a different retinal location. Yet, we instantly see these new neural responses as the same object.
Shape constancy helps us quickly recognize objects we have seen before in different circumstances.

For example, we continue to see this sign as a rotated rectangle:

We see the windows and doors facing the street as square rectangles, not oblique rectangles.

Having seen this chair once in one position, we easily recognize it in any other position. A computer could have made this composite from a single photo. It is not surprising that our visual system does the opposite.
Organizing Principles

In poor light, survival may depend on more than acuity. The visual system continually seeks shapes in the retinal image. This is evident in random scenes: As you view this figure, some white or black lines temporarily seem to connect and produce jagged lines or enclose irregular shapes. (Depending on your screen size, some distances may work better.)
The following figures demonstrate some methods used to organize the visual image to make it easier to understand.
1.Features close together get lumped together:

Even slight differences in spacing is enough to produce the appearance of rows or columns.
2. Similarities brightness, color, orientation, or size get lumped together:

The visual system continually uses such principles in assembling shapes from receptive fields into whole images.
To avoid randomness, the system uses proximity and similarity of the pieces in this image to enable recognition of this common shape:

Expected lines are imagined, even when there are no actual lines.
These abilities could determine whether you spot your next meal amongst the trees:

(Upper right is a cube; above a young moose at near left.)
Noticing unusual shapes can be critical for survival.
Can you spot the hiker in this photo?:
Because Symmetry is rare in non-life forms, we search for it every-where. Did it take longer to find all 5 Zs in the lower figure?
Our visual system also tends to have directional preferences in making images available. Who hasn’t see the gruesome appearance of a face at night when illuminated from beneath with a flashlight?
Almost all natural light comes from above, so the system organizes visual images with that in mind.
In unusual conditions, this can lead to misconceptions.
Traveling across the Arctic, it is essential to distinguish approaching a crater from a distant pingo:

While not fooled if your snowmobile flips, in a microscope, goose bumps may be mistaken for dimples:

Gradual discoveries of these cues and effects by artists through the ages provide interesting examples in Art History courses.
The above examples show some of the principles followed by the system to assemble a visual image from how cells in the visual cortex respond to the retinal image. As noted earlier regarding the retinal image being upside-down, “vision is not constrained by neuroanatomy” (Ch 10). These principles describe yet-to-be-identified neural processes that make the information in the retinal image more effective than a merely replicating that image in guiding our thoughts and behavior.
Here is a picture that seems meaningless at first to many people:
Let your eyes wander over it, seeking possibilities.
? See note [3].
Before and after you learned what this represents, the responses of the cells in your retina and primary visual cortex were the same, yet what you saw became amazingly different.
This illustrates the difference between the retinal image, the receptive fields of cortical neurons, and the “picture-in-your-head”.
More examples of principles that the brain uses to interpret the retinal image can found in text books for sensation and perception courses. For many years I used Wolfe JM, et al’s Sensation and Perception, now in its 7th edition, 2024, published by Sinauer Associates of Oxford University Press.
- Hemker, et al (2010) The relative contribution of relative height, linear perspective and texture gradient to pictorial depth perception in 7-month-old infants. Perception, 39, 1476-1490. ↵
- On the horizon, various cues (proximity to the horizon, linear perspective, size gradient, interposition, atmospheric coloring, and blur) indicate that the moon’s distance is at or beyond the horizon. Overhead, there are no cues to its distance. Measurements of the moon’s apparent size indicate that it is perceived to be 1/3 to 1/4 closer than the horizon when overhead. This effect is referred to as the vault of heaven - a perceptual sense of how high the sky is. ↵
- It is a cow, from Dallenbach KM (1951) A puzzle-picture with a new principle of concealment. American Journal of Psychology, 64, 431-433. If you need more help, go to Appendix H. ↵