New York City records massive amounts of video from thousands of traffic cameras every day, but city agencies lack the resources to manually watch this footage to identify safety issues. To solve this problem, researchers at the NYU Tandon School of Engineering have developed a new artificial intelligence (AI) system called SeeUnsafe. This computer program automatically reviews existing video files to detect collisions and dangerous driving behaviors. The project recently received the Vision Zero Research Award for its potential to help the city reduce traffic fatalities.
The system uses multimodal large language models capable of understanding and processing different forms of information, such as images and text, simultaneously. Because the software is pre-trained to interpret these visual and textual inputs, transportation officials can use the tool without needing to be computer vision experts. They also avoid the typically expensive process of labeling their own data to train the computer.
Detecting danger before accidents happen
In testing, the system successfully categorized video clips as crashes, normal traffic, or near-misses with an accuracy rate of roughly 77 percent. A near-miss is defined as a specific incident where vehicles come dangerously close to hitting pedestrians or other cars but manage to avoid actual contact. Traditionally, city planners implement safety changes only after accidents occur. This new technology allows agencies to be proactive. By analyzing patterns of near-misses, officials can identify hazardous intersections and improve signage or signal timing before a serious injury takes place.
Beyond simply flagging a video, the software generates road safety reports. These are written explanations describing the weather, traffic volume, and specific vehicle movements that caused the dangerous event. While the researchers noted that the system still faces challenges in low-light conditions, it offers a way to utilize the vast network of cameras already installed in cities. By automatically pinpointing high-risk locations, the group hopes to transform how transportation agencies approach road safety interventions.
The researchers have described the methods and results of this study in a paper published in Accident Analysis & Prevention.