Sight GPT

Abstract

Sight GPT is a web-based application designed to assist visually impaired users in obtaining detailed information about images captured with their smartphones. The platform enables interactive exploration of photographs and the surrounding visual environment through a multimodal interface that combines speech input, haptic feedback, and visual cues. Primarily, Sight GPT aims to help visually impaired users mentally visualize content they cannot perceive, while also accommodating users with partial vision. The system integrates advanced Natural Language Processing (NLP) and Computer Vision techniques to provide precise answers to user queries about on-screen images. Image analysis is further enhanced by background removal and segmentation models, which isolate and emphasize salient elements of the scene. Preliminary evaluations demonstrate the effectiveness of the application in improving accessibility and user experience, marking an initial step toward leveraging AI-powered multimodal systems to support visually impaired individuals.