Researchers at Cornell University and KAIST have created WatchHand, a new system that turns ordinary smartwatches into tools for tracking hand movements. The technology lets users control devices with simple finger gestures, such as tapping the thumb and index finger twice to skip a song or moving a cursor without a mouse. It works on existing smartwatches without adding any extra hardware.
WatchHand uses the smartwatch's built-in speaker and microphone. The speaker sends out inaudible sound waves, which are too high-pitched for humans to hear. These waves bounce off the hand and return to the microphone, forming an echo pattern. A machine learning algorithm, a type of artificial intelligence that learns from data, then analyzes this pattern to estimate the exact 3D position of the hand and fingers in real time. All processing happens directly on the watch, so personal movement data stays private.
Breakthrough in everyday wearable sensing
This approach achieves continuous 3D hand pose tracking on standard smartwatches using only acoustic sensing. Previous systems often needed bulky extra sensors, making them impractical for daily wear. WatchHand avoids this by relying on software updates alone, potentially adding new abilities to millions of existing devices.
Tests involved 40 people and about 36 hours of gesture data across different smartwatch models, hands, and noisy environments. The system reliably tracked finger movements and wrist rotations. It performed well even with background noise, though accuracy dropped when users walked.
WatchHand could support people with limited mobility or speech by enabling gesture-based control. It may also serve as a controller for augmented reality and virtual reality, where digital elements overlay the real world or create fully simulated environments. Future uses might include tracking typing motions directly from the wrist.
The development reflects a shift toward turning common wearables into intelligent sensors that understand human movements with low energy use.
The paper “WatchHand: Enabling Continuous Hand Pose Tracking On Off-the-Shelf Smartwatches,” will be presented at the ACM CHI conference on Human Factors in Computing Systems.