From surface signals to subsurface intelligence in wearable hand tracking

A SHIFT FROM SURFACE SENSING TO BIOMECHANICAL INTELLIGENCE

The ability to accurately and continuously track human hand motion remains a central challenge for robotics, human-computer interaction, and emerging spatial computing systems. For decades, dominant approaches have relied on surface-level sensing modalities, including vision-based tracking systems^[1], electromyography (EMG)^[2], and alternative wearable modalities such as strain^[3] and inertial sensors^[4] to infer hand kinematics. Fundamentally, they depend on indirect proxies of motion, inferring intent from surface electrical or kinematic signals rather than directly observing biomechanical actuation^[5]. This often limits robustness, precision, and generalizability across environments and users^[6,7].

In this context, the recent work by Lu et al. represents a new paradigm in subsurface intelligent sensing, enabling continuous reconstruction of all 22 degrees of freedom (DOFs) of the hand with high accuracy and low latency^[8]. By integrating wearable ultrasound imaging with machine learning, the authors introduce a wrist-mounted system capable of directly observing subsurface anatomical structures, including tendons and muscles, in real time, as illustrated in Figure 1. Subsurface intelligence in this sense is defined as the direct inference of human intent from dynamically imaged internal anatomical structures, rather than from externally observable electrical, optical, or kinematic proxies. Unlike traditional sensors, subsurface intelligent sensing does not rely on external cameras or restrictive instrumentation and bypasses the inherent ambiguity of surface measurements. This aligns closely with the broader paradigm of embodied intelligence, where sensing, control, and physical structure are tightly integrated^[9].

Figure 1. From surface sensing to subsurface intelligence for wearable hand tracking. Conventional methods infer hand motion from surface-level electrical or kinematic proxies (left), while contemporary subsurface sensing directly images deep tissue dynamics (center). This progression drives real-time reconstruction of continuous 22-DOF hand kinematics for VR control, rehabilitation, and teleoperation (right). DOF: Degree of freedom; VR: virtual reality; EMG: electromyography; AR: augmented reality.

Importantly, this approach also addresses persistent limitations in existing hand-tracking systems [Table 1]. Vision-based methods, while powerful, are susceptible to occlusions, lighting variability, and viewpoint constraints^[10,11]. EMG-based systems, though wearable, often suffer from signal variability due to electrode placement, skin impedance, and user-specific physiology^[7]; these systems are typically limited to recognizing predefined discrete gestures rather than continuously reconstructing arbitrary hand configurations. By contrast, Lu et al. showed that ultrasound imaging provides a stable and anatomically grounded sensing modality, capable of capturing deep tissue motion with high fidelity across diverse conditions^[8].

IMPLICATIONS FOR SOFT ROBOTICS AND WEARABLE REHABILITATION

The implications of this work are especially compelling for soft robotics and wearable rehabilitation, which aim to translate human intent into assistive actuation. Soft robotic gloves have emerged as promising devices for restoring hand function after neurological trauma by using compliant materials for safe and adaptive interaction with the human body^[19,20]. However, most devices rely on EMG signals or simple motion sensors, which provide limited resolution and require user-specific calibration. A biomechanical intelligent system could hence enable more precise and responsive control of assistive devices.

Recent magnetic resonance imaging (MRI)-compatible^[21] and fully fabric-based soft robotic gloves^[22] illustrate the growing maturity of wearable rehabilitation platforms. The work of Lu et al. complements these systems by providing a high-fidelity biomechanical sensing layer in closed-loop assistive control^[8]. Compared with state-of-the-art systems^[23], a model inference time of less than 9 ms shows promise for real-time closed-loop feedback. Additionally, unlike prior ultrasound-based sensing research^[24,25], this framework enables continuous, high-dimensional estimation of hand kinematics that respond in real time to the user’s biomechanical state.

For instance, a controller would compute the error between estimated kinematics and a desired trajectory and modulate actuator assistance accordingly. A proportional assistive force will be provided by a soft robotic actuation system, either tendon- or pneumatic-driven, to the deficit between the patient’s voluntary movement and a desired trajectory, thereby implementing the assist-as-needed paradigm for motor rehabilitation robotics.

CHALLENGES AND THE ROAD TOWARD EMBODIED INTELLIGENCE

Despite its promise, several challenges must be addressed before wearable ultrasound-based hand tracking can achieve widespread adoption.

First, ensuring cross-user applicability remains a key concern, particularly given the current reliance on user-specific model training. As Lu et al. pointed out, variations in anatomy, tissue composition, and sensor placement can affect signal quality and model performance^[8]. Ultrasound imaging is inherently sensitive to acoustic coupling between the transducer and skin. Probe pressure directly affects the degree to which the soft interface maintains conformal contact with the wrist surface, where moderate changes in strap tightness may alter acoustic impedance and introduce spatial artifacts. Moreover, small shifts in sensor position caused by natural wrist movement may further alter imaging planes, producing temporal inconsistencies that challenge model robustness during unconstrained daily activities. These changes the apparent morphology of the muscles that the AI model was trained on. Addressing these challenges will likely require the development of large-scale datasets, of which there is much scarcity^[26] and few standardized data augmentation methods^[27]. More generalizable learning frameworks are needed considering data scarcity, potentially drawing on recent advances in foundation models for sensorimotor intelligence.

Second, while ultrasound provides rich biomechanical information, translating these signals into accurate and interpretable motion representations is nontrivial, particularly due to entanglement between joint configuration and muscle force^[28]. In real-world applications, distinguishing between these two sources of morphological change is essential for reliable interpretation. The proposed system focuses on decoding joint angles under controlled conditions without accounting for variations in muscle strength at fixed joint postures. This means there may be different predictions for the same hand configuration depending on grip force, which is a critical limitation for safety-critical applications in assistive and rehabilitation robotics where distinguishing voluntary intent from isometric co-contraction is essential. The relationship between internal tissue dynamics and external kinematics is complex and context-dependent, necessitating sophisticated modeling approaches. In this regard, integrating physics-informed models with data-driven learning may offer a promising path forward.

Third, practical considerations related to hardware miniaturization, power consumption (~5 W), and long-term wearability constraints must be addressed. Advances in flexible electronics, low-power ultrasound transducers, and integrated system design will be critical in this regard. Broader literature on biomechanical energy harvesting offers a conceptually relevant connection. For instance, Zou et al. propose a wire-driven mechanism integrated into a vest-like garment that allows joint motion to excite an onboard generator without direct mechanical loading of joints^[29]. Admittedly, the 5W budget is beyond what biomechanical harvesters can currently supply, so this energy-harvesting connection should be framed as a medium-to-long-term prospect rather than a near-term engineering solution.

Looking ahead, three directions will be pivotal: (1) cross-user generalization via large-scale learning; (2) multimodal sensing to disentangle force and motion; and (3) miniaturized, low-power wearable ultrasound systems. Together, these developments will determine whether such systems evolve from compelling laboratory demonstrations into ubiquitous human-machine interfaces.

Ultimately, the work of Lu et al. points toward a future in which human motion is no longer inferred but reconstructed directly from subsurface biomechanical dynamics^[8]. By bridging the gap between sensing and embodiment, wearable ultrasound technologies may redefine how humans interact with machines, thereby enabling more natural, adaptive, and high-fidelity human-machine interfaces.