Automatic Conversion and Rendering of Graphics for BVIs
Main Features
- Real-time conversion and rendering
- Vibrotactile display
- Low vision display
- Robotic assistance as needed
- Speech/braille choice
Project Summary
The project intends to build and deliver a low-cost system for the real-time automatic conversion of visual diagrams into accessible dual vibrotactile and enhanced visual forms rendered on aligned, refreshable tactile and visual displays. It will also provide haptic assistive support, to the degree desired by the user, for exploring the accessible diagram. This will provide individuals who are blind or visually impaired independent on-demand access to the information in visual graphics, which are ubiquitously used in education, employment and daily living.
BVI and TVI Involvement Requested
Focus Groups
We are interested in finding out when and how BVIs currently use or would like to use graphical information, and what features would be desirable in an automated system. We are interested in talking to both BVIs and TVIs who have experience and/or interest in graphics access. We will be holding focus groups both in person (around Virginia) and online during the months of October-November 2025. Participants will be compensated for their time. For more information contact: Prof. Dianne Pawluk, [email protected], 804-525-8793 or Andrew Tuck, [email protected], 703-215-6107.
User Studies at VCU
Soon we will be holding studies to examine how BVIs with experience accessing charts and graphs, whether through tactile graphics or low vision representations, do so. We will also hold studies to examine how TVIs with experience teaching BVIs to access charts and graphs do so. For more information contact: Prof. Dianne Pawluk, [email protected], 804-525-8793.
Extended Summary
Visual diagrams are playing an increasingly dominant role in providing information in education, employment and daily living. Unfortunately, individuals who are blind or visually impaired (BVIs) experience enormous obstacles in independently accessing this information, impacting their success in all aspects of their lives. The proposed project intends to addresses this critical need through the development of a low-cost system for the real-time automatic conversion of visual diagrams into accessible dual vibrotactile and enhanced visual forms rendered on aligned, refreshable tactile and visual displays, allowing BVIs to use touch, their residual vision or a combination of both to access graphical information. To ease the significant time and cognitive load required by users to interpret accessible diagrams, automated haptic guidance will be available to help BVIs explore a diagram through shared control with the user. Basic alt-text descriptions will also be available, both in Braille and in spoken form, to also support the effective exploration of the diagram. A participatory design approach will be used for the system’s development, involving BVIs and other stakeholders in: focus groups, low technology prototyping and user testing both in the laboratory and in the field.
The display system will consist of a low-cost refreshable haptic display, previously developed in one of the PI’s laboratories, combining multi-fingered tactile feedback with shared control of movement between the user and haptic guidance fixtures (with user adjustable weightings) to which an enhanced vision display will be added. Machine learning will be used to convert visual diagrams to an accessible form useful for both lower level and higher-level thinking. To obtain a more effective model for conversion, the key component to our approach is to take advantage of the underlying structure of known diagram types, with separate models for graphs and charts (simple and highly structured), illustrations and photos (complex and least structured), and maps and engineering diagrams (focused on the connectivity of elements and using a symbol set). The R61 Phase of the project will focus on providing a beta version of the complete system for simple, high impact graphics (graphs and charts) through multiple design iterations with BVI user testing in the laboratory. The R33 Phase of the project will deploy the beta version in the field with BVI users to further assess and improve the design for real world use. This phase will also work on the development of conversion methods for illustrations and photos, and indoor/outdoor maps and engineer diagrams, again through multiple design iterations with BVI user testing the laboratory. Dissemination will also occur during this stage.