Abstract: This paper presents an efficient metric for quantifying the visual fidelity of natural images based on near-threshold and suprathreshold properties of human vision. The proposed metric, the visual signal-to-noise ratio (VSNR), operates via a two-stage approach. In the first stage, contrast thresholds for detection of distortions in the presence of natural images are computed via wavelet-based models of visual masking and visual summation in order to determine whether the distortions in the distorted image are visible. If the distortions are below the threshold of detection, the distorted image is deemed to be of perfect visual fidelity (VSNR = infin)and no further analysis is required. If the distortions are suprathreshold, a second stage is applied which operates based on the low-level visual property of perceived contrast, and the mid-level visual property of global precedence. These two properties are modeled as Euclidean distances in distortion-contrast space of a multiscale wavelet decomposition, and VSNR is computed based on a simple linear sum of these distances. The proposed VSNR metric is generally competitive with current metrics of visual fidelity; it is efficient both in terms of its low computational complexity and in terms of its low memory requirements; and it operates based on physical luminances and visual angle (rather than on digital pixel values and pixel-based dimensions) to accommodate different viewing conditions.

Abstract: The authors model the neural mechanisms underlying spatial cognition, integrating neuronal systems and behavioral data, and address the relationships between long-term memory, short-term memory, and imagery, and between egocentric and allocentric and visual and ideothetic representations. Long-term spatial memory is modeled as attractor dynamics within medial-temporal allocentric representations, and short-term memory is modeled as egocentric parietal representations driven by perception, retrieval, and imagery and modulated by directed attention. Both encoding and retrieval/imagery require translation between egocentric and allocentric representations, which are mediated by posterior parietal and retrosplenial areas and the use of head direction representations in Papez's circuit. Thus, the hippocampus effectively indexes information by real or imagined location, whereas Papez's circuit translates to imagery or from perception according to the direction of view. Modulation of this translation by motor efference allows spatial updating of representations, whereas prefrontal simulated motor efference allows mental exploration. The alternating temporal-parietal flows of information are organized by the theta rhythm. Simulations demonstrate the retrieval and updating of familiar spatial scenes, hemispatial neglect in memory, and the effects on hippocampal place cell firing of lesioned head direction representations and of conflicting visual and ideothetic inputs.

Abstract: Humans, being highly social creatures, rely heavily on the ability to perceive what others are doing and to infer from gestures and expressions what others may be intending to do. These perceptual skills are easily mastered by most, but not all, people, in large part because human action readily communicates intentions and feelings. In recent years, remarkable advances have been made in our understanding of the visual, motoric, and affective influences on perception of human action, as well as in the elucidation of the neural concomitants of perception of human action. This article reviews those advances and, where possible, draws links among those findings.

Abstract: Playing action video games enhances several different aspects of visual processing; however, the mechanisms underlying this improvement remain unclear. Here we show that playing action video games can alter fundamental characteristics of the visual system, such as the spatial resolution of visual processing across the visual field. To determine the spatial resolution of visual processing, we measured the smallest distance a distractor could be from a target without compromising target identification. This approach exploits the fact that visual processing is hindered as distractors are brought close to the target, a phenomenon known as crowding. Compared with nonplayers, action-video-game players could tolerate smaller target-distractor distances. Thus, the spatial resolution of visual processing is enhanced in this population. Critically, similar effects were observed in non-video-game players who were trained on an action video game; this result verifies a causative relationship between video-game play and augmented spatial resolution.

Abstract: Visual clutter concerns designers of user interfaces and information visualizations. This should not surprise visual perception researchers because excess and/or disorganized display items can cause crowding, masking, decreased recognition performance due to occlusion, greater difficulty at both segmenting a scene and performing visual search, and so on. Given a reliable measure of the visual clutter in a display, designers could optimize display clutter. Furthermore, a measure of visual clutter could help generalize models like Guided Search (J. M. Wolfe, 1994) by providing a substitute for "set size" more easily computable on more complex and natural imagery. In this article, we present and test several measures of visual clutter, which operate on arbitrary images as input. The first is a new version of the Feature Congestion measure of visual clutter presented in R. Rosenholtz, Y. Li, S. Mansfield, and Z. Jin (2005). This Feature Congestion measure of visual clutter is based on the analogy that the more cluttered a display or scene is, the more difficult it would be to add a new item that would reliably draw attention. A second measure of visual clutter, Subband Entropy, is based on the notion that clutter is related to the visual information in the display. Finally, we test a third measure, Edge Density, used by M. L. Mack and A. Oliva (2004) as a measure of subjective visual complexity. We explore the use of these measures as stand-ins for set size in visual search models and demonstrate that they correlate well with search performance in complex imagery. This includes the search-in-clutter displays of J. M. Wolfe, A. Oliva, T. S. Horowitz, S. Butcher, and A. Bompas (2002) and Bravo and Farid (2004), as well as new search experiments. An additional experiment suggests that color variability, accounted for by Feature Congestion but not the Edge Density measure or the Subband Entropy measure, does matter for visual clutter.

Abstract: Peelen and Downing review recent evidence for body-selective neural mechanisms in the visual cortex and discuss how body-selective brain regions might relate to action perception and the 'mirror' system, perception of the self and the 'body schema', and understanding the emotions of others.

Abstract: In mammalian visual system, spikes evoked by visual stimuli have millisecond-scale timing even though the relevant timescales of visual processing themselves are much slower. It has therefore long been debated whether spike timing itself carries some form of the neural code. Now experiments in the lateral geniculate nucleus of cats, the part of the brain that is the primary processor of visual information, show that spike timing precision is not absolute for all classes of visual stimuli. Rather, the degree of precision is relative to the timescale of the stimulus, and this relatively high level of precision is required to construct an accurate representation of the stimulus. In the mammalian visual system, spikes evoked by visual stimuli have millisecond-scale timing, even though the relevant time scales of visual processing themselves are much slower. In cat lateral geniculate nucleus, spike timing precision is not absolute for all classes of visual stimuli, but is relative to the time scale of the stimulus. Further, it is demonstrated that this relatively high level of precision is required to construct an accurate representation of the stimulus. The timing of action potentials relative to sensory stimuli can be precise down to milliseconds in the visual system1,2,3,4,5,6,7, even though the relevant timescales of natural vision are much slower. The existence of such precision contributes to a fundamental debate over the basis of the neural code and, specifically, what timescales are important for neural computation8,9,10. Using recordings in the lateral geniculate nucleus, here we demonstrate that the relevant timescale of neuronal spike trains depends on the frequency content of the visual stimulus, and that ‘relative’, not absolute, precision is maintained both during spatially uniform white-noise visual stimuli and naturalistic movies. Using information-theoretic techniques, we demonstrate a clear role of relative precision, and show that the experimentally observed temporal structure in the neuronal response is necessary to represent accurately the more slowly changing visual world. By establishing a functional role of precision, we link visual neuron function on slow timescales to temporal structure in the response at faster timescales, and uncover a straightforward purpose of fine-timescale features of neuronal spike trains.

Abstract: We compared two tasks that are widely used in research on mentalizing---false belief stories and animations of rigid geometric shapes that depict social interactions---to investigate whether the neural systems that mediate the representation of others' mental states are consistent across these tasks. Whereas false belief stories activated primarily the anterior paracingulate cortex (APC), the posterior cingulate cortex/precuneus (PCC/PC), and the temporo-parietal junction (TPJ)---components of the distributed neural system for theory of mind (ToM)---the social animations activated an extensive region along nearly the full extent of the superior temporal sulcus, including a locus in the posterior superior temporal sulcus (pSTS), as well as the frontal operculum and inferior parietal lobule (IPL)---components of the distributed neural system for action understanding---and the fusiform gyrus. These results suggest that the representation of covert mental states that may predict behavior and the representation of intentions that are implied by perceived actions involve distinct neural systems. These results show that the TPJ and the pSTS play dissociable roles in mentalizing and are parts of different distributed neural systems. Because the social animations do not depict articulated body movements, these results also highlight that the perception of the kinematics of actions is not necessary to activate the mirror neuron system, suggesting that this system plays a general role in the representation of intentions and goals of actions. Furthermore, these results suggest that the fusiform gyrus plays a general role in the representation of visual stimuli that signify agency, independent of visual form.

Abstract: Visual processing is not determined solely by retinal inputs. Attentional modulation can arise when the internal attentional state (current task) of the observer alters visual processing of the same stimuli. This can influence visual cortex, boosting neural responses to an attended stimulus. Emotional modulation can also arise, when affective properties (emotional significance) of stimuli, rather than their strictly visual properties, influence processing. This too can boost responses in visual cortex, as for fear-associated stimuli. Both attentional and emotional modulation of visual processing may reflect distant influences upon visual cortex, exerted by brain structures outside the visual system per se. Hence, these modulations may provide windows onto causal interactions between distant but interconnected brain regions. We review recent evidence, noting both similarities and differences between attentional and emotional modulation. Both can affect visual cortex, but can reflect influences from different regions, such as fronto-parietal circuits versus the amygdala. Recent work on this has developed new approaches for studying causal influences between human brain regions that may be useful in other cognitive domains. The new methods include application of functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) measures in brain-damaged patients to study distant functional impacts of their focal lesions, and use of transcranial magnetic stimulation concurrently with fMRI or EEG in the normal brain. Cognitive neuroscience is now moving beyond considering the putative functions of particular brain regions, as if each operated in isolation, to consider, instead, how distinct brain regions (such as visual cortex, parietal or frontal regions, or amygdala) may mutually influence each other in a causal manner.

Abstract: In visual search tasks, observers look for targets in displays containing distractors. Likelihood that targets will be missed varies with target prevalence, the frequency with which targets are presented across trials. Miss error rates are much higher at low target prevalence (1%-2%) than at high prevalence (50%). Unfortunately, low prevalence is characteristic of important search tasks such as airport security and medical screening where miss errors are dangerous. A series of experiments show this prevalence effect is very robust. In signal detection terms, the prevalence effect can be explained as a criterion shift and not a change in sensitivity. Several efforts to induce observers to adopt a better criterion fail. However, a regime of brief retraining periods with high prevalence and full feedback allows observers to hold a good criterion during periods of low prevalence with no feedback.

Abstract: Our ability to coordinate the use of our left and right eyes and to make use of subtle differences between the images received by each eye allows us to perceive stereoscopic depth, which is important for the visual perception of three-dimensional space. Binocular neurons in the visual cortex combine signals from the left and right eyes. Probing the roles of binocular neurons in different perceptual tasks has advanced our understanding of the stages within the visual cortex that lead to binocular depth perception.

Abstract: Recent studies, conducted almost exclusively in primates, have shown that several cortical areas usually associated with modality-specific sensory processing are subject to influences from other senses. Here we demonstrate using single-unit recordings and estimates of mutual information that visual stimuli can influence the activity of units in the auditory cortex of anesthetized ferrets. In many cases, these units were also acoustically responsive and frequently transmitted more information in their spike discharge patterns in response to paired visual-auditory stimulation than when either modality was presented by itself. For each stimulus, this information was conveyed by a combination of spike count and spike timing. Even in primary auditory areas (primary auditory cortex [A1] and anterior auditory field [AAF]), approximately 15% of recorded units were found to have nonauditory input. This proportion increased in the higher level fields that lie ventral to A1/AAF and was highest in the anterior ventral field, where nearly 50% of the units were found to be responsive to visual stimuli only and a further quarter to both visual and auditory stimuli. Within each field, the pure-tone response properties of neurons sensitive to visual stimuli did not differ in any systematic way from those of visually unresponsive neurons. Neural tracer injections revealed direct inputs from visual cortex into auditory cortex, indicating a potential source of origin for the visual responses. Primary visual cortex projects sparsely to A1, whereas higher visual areas innervate auditory areas in a field-specific manner. These data indicate that multisensory convergence and integration are features common to all auditory cortical areas but are especially prevalent in higher areas.

Abstract: This article describes the impact of visual attention on consumers' in-store buying behaviour. Through an eye-track experiment, it demonstrates the advantage of a behaviour model that addresses visual attention and an increase in visual stimuli during the process. It reveals that consumers exhibit a muddled search strategy where packaging design influences the decision process in several phases. Five phases were found in an in-store decision process, and the post-purchase phase seems to be essential for even low-level in-store decision processes. Further knowledge on packaging design elements is needed for a broader understanding of visual influence during in-store purchase decisions.

Abstract: The brain should integrate related but not unrelated information from different senses. Temporal patterning of inputs to different modalities may provide critical information about whether those inputs are related or not. We studied effects of temporal correspondence between auditory and visual streams on human brain activity with functional magnetic resonance imaging (fMRI). Streams of visual flashes with irregularly jittered, arrhythmic timing could appear on right or left, with or without a stream of auditory tones that coincided perfectly when present (highly unlikely by chance), were noncoincident with vision (different erratic, arrhythmic pattern with same temporal statistics), or an auditory stream appeared alone. fMRI revealed blood oxygenation level-dependent (BOLD) increases in multisensory superior temporal sulcus (mSTS), contralateral to a visual stream when coincident with an auditory stream, and BOLD decreases for noncoincidence relative to unisensory baselines. Contralateral primary visual cortex and auditory cortex were also affected by audiovisual temporal correspondence or noncorrespondence, as confirmed in individuals. Connectivity analyses indicated enhanced influence from mSTS on primary sensory areas, rather than vice versa, during audiovisual correspondence. Temporal correspondence between auditory and visual streams affects a network of both multisensory (mSTS) and sensory-specific areas in humans, including even primary visual and auditory cortex, with stronger responses for corresponding and thus related audiovisual inputs.

Abstract: Events can sometimes appear longer or shorter in duration than other events of equal length. For example, in a repeated presentation of auditory or visual stimuli, an unexpected object of equivalent duration appears to last longer. Illusions of duration distortion beg an important question of time representation: when durations dilate or contract, does time in general slow down or speed up during that moment? In other words, what entailments do duration distortions have with respect to other timing judgments? We here show that when a sound or visual flicker is presented in conjunction with an unexpected visual stimulus, neither the pitch of the sound nor the frequency of the flicker is affected by the apparent duration dilation. This demonstrates that subjective time in general is not slowed; instead, duration judgments can be manipulated with no concurrent impact on other temporal judgments. Like spatial vision, time perception appears to be underpinned by a collaboration of separate neural mechanisms that usually work in concert but are separable. We further show that the duration dilation of an unexpected stimulus is not enhanced by increasing its saliency, suggesting that the effect is more closely related to prediction violation than enhanced attention. Finally, duration distortions induced by violations of progressive number sequences implicate the involvement of high-level predictability, suggesting the involvement of areas higher than primary visual cortex. We suggest that duration distortions can be understood in terms of repetition suppression, in which neural responses to repeated stimuli are diminished.

Abstract: When a single flash of light is presented interposed between two brief auditory stimuli separated by 60-100 ms, subjects typically report perceiving two flashes (Shams et al., 2000, 2002). We investigated the timing and localization of the cortical processes that underlie this illusory flash effect in 34 subjects by means of 64-channel recordings of event-related potentials (ERPs). A difference ERP calculated to isolate neural activity associated with the illusory second flash revealed an early modulation of visual cortex activity at 30-60 ms after the second sound, which was larger in amplitude in subjects who saw the illusory flash more frequently. These subjects also showed this early modulation in response to other combinations of auditory and visual stimuli, thus pointing to consistent individual differences in the neural connectivity that underlies cross-modal integration. The overall pattern of cortical activity associated with the cross-modally induced illusory flash, however, differed markedly from that evoked by a real second flash. A trial-by-trial analysis showed that short-latency ERP activity localized to auditory cortex and polymodal cortex of the temporal lobe, concurrent with gamma bursts in visual cortex, were associated with perception of the double-flash illusion. These results provide evidence that perception of the illusory second flash is based on a very rapid dynamic interplay between auditory and visual cortical areas that is triggered by the second sound.

Abstract: Older adults are often slower and less accurate than are younger adults in performing visual-search tasks, suggesting an age-related decline in attentional functioning. Age-related decline in attention, however, is not entirely pervasive. Visual search that is based on the observer's expectations (i.e., top-down attention) is relatively preserved as a function of adult age. Neuroimaging research suggests that age-related decline occurs in the structure and function of brain regions mediating the visual sensory input, whereas activation of regions in the frontal and parietal lobes is often greater for older adults than for younger adults. This increased activation may represent an age-related increase in the role of top-down attention during visual tasks. To obtain a more complete account of age-related decline and preservation of visual attention, current research is beginning to explore the relation of neuroimaging measures of brain structure and function to behavioral measures of visual attention.

Abstract: Attention is thought to enhance perceptual performance at attended locations through top-down attention signals that modulate activity in visual cortex. Here, we show that activity in early visual cortex is sustained during maintenance of attention in the absence of visual stimulation. We used functional magnetic resonance imaging (fMRI) to measure activity in visual cortex while human subjects performed a visual detection task in which a variable-duration delay period preceded target presentation. Portions of cortical areas V1, V2, and V3 representing the attended part of the visual field exhibited sustained increases in activity throughout the delay period. Portions of these cortical areas representing peripheral, unattended parts of the visual field displayed sustained decreases in activity. The data were well fit by a model that assumed the sustained neural activity was constant in amplitude over a time period equal to that of the actual delay period for each trial. These results demonstrate that sustained attention responses are present in early visual cortex (including primary visual cortex), in the absence of a visual stimulus, and that these responses correlate with the allocation of visuospatial attention in both the spatial and temporal domains.

Abstract: Emotional stimuli tend to capture and hold attention more than non-emotional stimuli do. Aversive pictures have been found to impair perception of visual targets even after the emotional informatio...

Abstract: We investigated the role of object-based attention in modulating the maintenance of faces and scenes held online in working memory (WM). Participants had to remember a face and a scene, while cues presented during the delay instructed them to orient their attention to one or the other item. Event-related functional magnetic resonance imaging revealed that orienting attention in WM modulated the activity in fusiform and parahippocampal gyri, involved in maintaining representations of faces and scenes respectively. Measures from complementary behavioral studies indicated that this increase in activity corresponded to improved WM performance. The results show that directed attention can modulate maintenance of specific representations in WM, and help define the interplay between the domains of attention and WM.

Abstract: If working memory is limited by central capacity (e.g., the focus of attention; N. Cowan, 2001), then storage limits for information in a single modality should apply also to the simultaneous storage of information from different modalities. The authors investigated this by combining a visual-array comparison task with a novel auditory-array comparison task in 5 experiments. Participants were to remember only the visual, only the auditory (unimodal memory conditions), or both arrays (bimodal memory conditions). Experiments 1 and 2 showed significant dual-task tradeoffs for visual but not for auditory capacity. In Experiments 3-5, the authors eliminated modality-specific memory by using postperceptual masks. Dual-task costs occurred for both modalities, and the number of auditory and visual items remembered together was no more than the higher of the unimodal capacities (visual: 3-4 items). The findings suggest a central capacity supplemented by modality- or code-specific storage and point to avenues for further research on the role of processing in central storage.

Abstract: In 4 experiments, the authors explored the role of visual layout in rule-based syntactic judgments. Participants judged the validity of a set of algebraic equations that tested their ability to apply the order of operations. In each experiment, a nonmathematical grouping pressure was manipulated to support or interfere with the mathematical convention. Despite the formal irrelevance of these grouping manipulations, accuracy in all experiments was highest when the nonmathematical pressure supported the mathematical grouping. The increase was significantly greater when the correct judgment depended on the order of operator precedence. The result that visual perception impacts rule application in mathematics has broad implications for relational reasoning in general. The authors conclude that formally symbolic reasoning is more visual than is usually proposed.

Abstract: Over the last 50 years or so, a large body of empirical research has demonstrated the importance of a variety of low-level spatiotemporal factors in the multisensory integration of auditory and visual stimuli (as, for example, indexed by research on the ventriloquism effect). Here, the evidence highlighting the contribution of both spatial and temporal factors to multisensory integration is briefly reviewed. The role played by the temporal correlation between auditory and visual signals, stimulus motion, intramodal versus crossmodal perceptual grouping, semantic congruency, and the unity assumption in modulating multisensory integration is also discussed. Taken together, the evidence now supports the view that a number of different factors, both structural and cognitive, conjointly contribute to the multisensory integration (or binding) of auditory and visual information.