Connected Wearables for People with Blindness and Low-Vision

Overview

Tens of millions of people worldwide live with blindness, and many more with low vision. Wearable assistants with cameras, microphones and inertial sensors can help them navigate, avoid obstacles and understand their surroundings. But the most capable perception and reasoning models, including large vision-language models, are far too heavy to run on a discreet, battery-powered wearable.

This project, a collaboration with Rizzo Labs at NYU Langone, connects the wearable to AI services at the network edge. The wearable runs fast, local perception for time-critical alerts and streams video, sensor data and voice queries over a 5G link to an edge server. There, a vision-language model agent draws on a spatial memory of what the user has seen and on maps of the environment, building on the UNav vision-based navigation system, to answer questions, give directions and describe the scene. Responses come back as audio and haptic feedback.

The wireless link is central to the design. Camera video needs high uplink bandwidth, while collision avoidance and conversation need low latency, and both must hold up as the user moves through buildings and city streets. The server continuously estimates the link rate and feeds it back so the wearable can adapt its video to what the channel can carry.

Background on the project’s first phase is on the original project page.