Researchers at Penn State have developed NaviSense, a new smartphone application designed to assist people who are visually impaired in navigating their environments. While technology for navigation has improved recently, the developers noticed that existing options often rely on connecting users to a live human assistant remotely. This approach can be slow and often raises privacy concerns for the user. Other automated options exist, but they typically require digital models of objects to be loaded into the system memory beforehand, which limits what the device can recognize effectively.
To solve these problems, the group utilized artificial intelligence (AI), specifically implementing Large Language Models (LLMs) and Vision-Language Models. A Vision-Language Model is an LLM-like technology that processes both images and text to answer questions about visual surroundings. These tools allow NaviSense to identify items in real-time based solely on a user’s spoken command. The app connects to an external server to process information, meaning users do not need to prepare the device with specific object data ahead of time.
Improving navigation through user feedback
Before building the final software, the lead student investigator conducted interviews to understand the specific challenges the visually impaired community faces. The group found that users wanted help not just finding an object visually, but physically reaching for it. Consequently, NaviSense monitors the movement of the user's phone to track their hand location. It provides feedback through sounds and vibrations to guide the user's hand directly toward the target object.
In controlled tests with twelve participants, the application located objects faster and more accurately than commercial alternatives. Participants praised the specific directional cues that told them to move left, right, up, or down to reach the item. The work was recognized with an audience choice award at a recent computing conference. Moving forward, the developers are working to optimize the power consumption of the artificial intelligence models, as high energy use can drain smartphone batteries quickly. They aim to solve this efficiency issue before making the tool available commercially.