Accessible XR
SocialCueSwitch: Towards Customizable Accessibility by Representing Social Cues in Multiple Senses
A toolkit and design principle — "social cue switching" — that lets blind/low-vision and d/Deaf/hard-of-hearing users choose how social VR cues like gaze, gesture, and proximity are rendered.
Jonathan Isaac Segal, Samuel Rodriguez, Akshaya Raghavan, Heysil Baez, Crescentia Jung, Jazmin Collins, Shiri Azenkot, Andrea Stevenson Won
The research at a glance
What if people could choose which senses carry social cues in virtual reality?
Approach
We identified social cues that are often inaccessible to blind and low-vision or d/Deaf and hard-of-hearing users, then built tools to represent those cues through alternative sensory modalities.
Takeaway
Social cue switching makes accessibility customizable: people can choose visual, audio, or haptic representations, supported by a toolkit and a shared distribution approach.
A toolkit and design principle presented in the CHI 2024 Extended Abstracts.
Abstract
In virtual environments, many social cues (e.g. gestures, eye contact, and proximity) are currently conveyed visually or auditorily. Indicating social cues in other modalities, such as haptic cues to complement visual or audio signals, will help to increase VR’s accessibility and take advantage of the platform’s inherent flexibility. However, accessibility implementations in social VR are often siloed by single sensory modalities. To broaden the accessibility of social virtual reality beyond replacing one sensory modality with another, we identified a subset of social cues and built tools to enhance them allowing users to switch between modalities to choose how these cues are represented. Because consumer VR uses primarily visual and auditory stimuli, we started with social cues that were not accessible for blind and low vision (BLV) and d/Deaf and hard of hearing (DHH) people, and expanded how they could be represented to accommodate a number of needs. We describe how these tools were designed around the principle of social cue switching, and a standard distribution method to amplify reach.
Contributions
- Identifies social VR cues — gesture, eye contact, proximity, who is speaking to whom — that are typically only conveyed visually or aurally.
- Introduces 'social cue switching': letting users choose whether a cue is rendered as visual, audio, or haptic feedback.
- Ships a sample toolkit and a standard distribution approach so accessibility tools built across labs can reach a common audience.
Citation
@inproceedings{segal2024socialcueswitch,
title={SocialCueSwitch: Towards Customizable Accessibility by Representing Social Cues in Multiple Senses},
author={Segal, Jonathan Isaac and Rodriguez, Samuel and Raghavan, Akshaya and Baez, Heysil and Jung, Crescentia and Collins, Jazmin and Azenkot, Shiri and Won, Andrea Stevenson},
booktitle={Extended Abstracts of the CHI Conference on Human Factors in Computing Systems},
series={CHI EA '24},
articleno={295},
pages={1--7},
year={2024},
publisher={ACM},
doi={10.1145/3613905.3651109}
}