Using multimodal attention to design sensory substitution devices: Basic research and application

Abstract

Perception of the world around us can never be limited to only one stream. The perception of the world is multimodal. Understanding how humans perceive and integrate information from multiple sensory modalities is essential for developing effective sensory substitution devices (SSDs). This thesis explores the complex interplay of sensory modalities and attention in perception and how it relates to the design of SSDs. The research investigates tactile perception and its implications for SSDs, highlighting the importance of understanding tactile illusions and anisotropies. In paper I the intensity order illusion (IOI) was studied. This illusion can be described as the mislocalization of the second stimuli following the first one. If the second stimulus on the lower back has higher intensity than the first stimulus then participants tend to perceive it as an upward movement. This also works vice versa. Two experiments were conducted to understand what causes the IOI and to find out whether it works in horizontal deimension or only in vertical dimension. The findings from Paper I reveal insights into the IOI and the role of amplitude for the IOI. Additionally, the study validates the effectiveness of Lofelt 5 actuators for tactile stimulation, crucial for SSD development involving tactile stimulation. Examining multimodal attention, Papers II and III delve into the effects of synchrony and cross-modal cueing on foraging performance. In Paper II the focus was on studying the influence of visual and auditory synchrony on foraging. Three experiments were run to study the the influence of synchrony. The results show that when targets move in visual synchrony, this guides attention during multiple target visual search. These results can possibly be explained by the grouping of the targets. There was no evidence that participans have any search benefit from non-spatial sound even when the task was more complicated (Experiment 3). Paper III was made as an extension of the second paper. The main goal was to understand whether cues from different modalities can improve visual search. Three cues from different modalities (visual, haptic and sound) were compared to the condition where no cues were presented. The results clearly showed that cross-modal cueing (haptic cue and sound cue) is more effective compared to uni-modal cueing (visual cue). These findings can benefit in developing SSD and provide some insights about how information should be conveyed using SSD’s. Overall, the research underscores the importance of considering multimodal integration and attentional mechanisms in the development of SSDs, but also more generlly, for understanding perception. By leveraging insights from tactile perception and multimodal attention, future SSD designs can optimize sensory feedback delivery, enhancing usability and effectiveness for users with sensory impairments.