跳到正文
BeingBeyond:具身智能研究·· 3 天前AI 評分46

TactoHand:從影片學習基於幾何的稠密手部表面互動場

TactoHand: Learning Geometry-Based Dense Hand-Surface Interaction Fields from Video

AI 導讀

TactoHand 從單目 RGB 影片預測手部表面的接觸概率、接近程度及局部 3D 力場,並在每個 MANO 手部網格的 778 個頂點上輸出結果。訓練資料彙集 24 個來源、共 49.02 million 個跨相機視角的影格;在未參與訓練的 HOCap 上,接觸 Macro-F1 為 50.5%,接近距離平均絕對誤差為 6.40 mm。

正文

Whether a hand interacts with an object, another hand, or a surface in the scene, the hand itself provides a common reference for describing the interaction. TactoHand uses that reference to predict three quantities from monocular RGB video. At each of the 778 vertices on each MANO hand mesh, it estimates the probability of contact, proximity to the interacting surface, and the local components of a 3D force. Contact and proximity describe the geometry of an interaction. For hand–object interactions, force describes the hand’s mechanical effect on the object.

Overview of the TactoHand method
Fig. 1 TactoHand decodes three fields on a shared hand mesh. The physics layer connects vertex forces to the observed motion of the object.

Interactive predictions

Choose a sequence to compare the RGB frame with its posed hand meshes. Step through the predictions at 10 Hz and switch between contact, distance and force. Select a vertex to inspect its values, or turn on the wrench overlay to see net force and torque.

Learning from geometry and object motion

Collecting tactile measurements at scale requires specialized hardware. We instead draw on existing interaction datasets that record hand geometry and, for some objects, motion. Our collection brings together 24 sources spanning hand–object, hand–hand, and hand–scene interactions, with 49.02 million captured frames counted across camera views.

Labeled examples from the 24 source datasets, including hand–object, hand–hand and hand–scene interactions.
Fig. 2 The 24 sources span different camera views, capture settings and types of interaction. HOCap contributes to the collection and is held out of training.

To label proximity, we compute the distance from each hand vertex to the interacting surface. The proximity target is highest at the surface and decays with increasing separation. To label contact, we also consider the orientation of the hand surface relative to the nearest point and, for closed meshes, whether the vertex lies inside the interacting mesh. The quality of both labels depends on the alignment of the source geometry.

The distinction matters where two surfaces nearly meet. For a vertex outside an object but within the dataset’s contact threshold, the label depends on whether the hand surface faces the nearest object point. Within 3 mm, the nearest-point direction becomes unstable, so distance alone suffices. For an open scene surface, where an interior is difficult to define, we use unsigned distance.

Surface-normal filtering and inside-mesh inclusion
Fig. 3 Left, the hand-surface normal helps decide which nearby vertices qualify as contact. Right, vertices inside a closed object mesh are included to accommodate small pose errors.
Palm and dorsal views of HACO and TactoHand contact annotations for a mug and a tool.
Fig. 4 For the mug and tool grasps, compare the two sets of labels on the palm and back of the hand with the corresponding RGB crops.

Object motion constrains the total force and torque acting on an object, but many distributions of force across the fingers can produce the same motion. We use geometry, friction, environmental support, and consistency over time to estimate one plausible distribution. We derive these force labels for 15 hand–object datasets. Training excludes GigaHands force targets and all HOCap data.

To construct a force target, we first use the object’s linear and angular motion to estimate the net force and torque required under a reference mass of 1 kg. The solver then allocates that load between the hand and its environment. Hand vertices with signed distance below 20 mm are candidates. The solver favors vertices closer to the object, limits tangential force through friction, and penalizes abrupt changes between successive frames.

Friction cone, signed distances, and force allocation weights
Fig. 5 The friction cone limits the force direction at each candidate vertex. The lower-right plot shows how the penalty for assigning force increases as the vertex moves away from the object.

Predicting hand-surface fields from video

A pretrained DINOv3 ViT-L encodes each frame. A temporal transformer combines information across the clip, and three learned queries decode contact, proximity, and force at each 10 Hz video timestamp. The contact prediction combines the probability that the hand is engaged with the probability of contact at each vertex given engagement.

RGB inputs alongside contact, proximity and vertex-force fields on corresponding hand meshes.
Fig. 6 The three outputs share the same hand surface. Contact marks likely touching vertices, proximity varies with separation, and the arrows show the local force vectors.

The model learns engagement from whether any vertex on the hand touches the interacting surface. On engaged frames, a class-balanced loss supervises the spatial pattern of contact. The same engagement probability scales the force vectors across that hand, encouraging small forces when contact is unlikely.

The force prediction has one component along the surface normal and two along the tangent directions. To express those components in camera coordinates, we use the ground-truth posed hand mesh. On eligible rigid-object sequences, we sum the vertex forces and their moments about the object’s center of mass. A differentiable dynamics layer uses the resulting force and torque to advance the object state. The difference between the predicted and recorded poses provides an additional training signal for the vertex forces.

Force inputs and pose supervision during a rigid-body rollout
Fig. 7 Each interval between video frames contains three physics steps. The first uses the predicted forces and the next two use motion-derived references. The resulting object pose is compared with the recorded pose.

Contact, proximity, and force evaluation

On HOCap, which we held out of training, the temporal model achieves a contact Macro-F1 of 50.5%, compared with 18.0% for HACO. For proximity, the same model has a mean absolute distance error of 6.40 mm, compared with 40.10 mm for InterField-LSTM when predictions and targets are both clipped at 100 mm. On MOW, TactoHand and HACO achieve the same contact Macro-F1 of 34.5%.

The two metrics capture different aspects of the prediction. Contact Macro-F1 gives each frame–hand pair with predicted or reference contact equal weight. The distance metric also covers vertices away from contact. Clipping at 100 mm gives the methods a common evaluation range because InterField predicts distances up to 100 mm and TactoHand receives proximity supervision up to 250 mm.

Contact and proximity by dataset

Contact accuracy Macro-F1 / Micro-F1 · % ↑
DatasetTactoHand
Temporal
HACOEgoPHIContactPrompt
HOCap50.5 / 64.518.0 / 19.511.4 / 16.2—
ARCTIC (s05, ego)51.0 / 63.842.2 / 55.831.8 / 38.5—
DexYCB59.5 / 67.634.3 / 41.52.7 / 4.3—
MOW34.5 / 44.334.5 / 41.512.4 / 28.729.7 / 36.5
Proximity error ADE / ADE-100 · mm ↓
DatasetTactoHand
Temporal
TactoHand
Single frame
InterField-SFInterField-LSTM
HOCap42.51 / 6.4046.09 / 6.35132.51 / 38.15134.69 / 40.10
ARCTIC (s05, ego)16.23 / 9.3220.02 / 10.5829.65 / 9.8129.68 / 9.77
DexYCB23.29 / 3.4226.14 / 3.92128.99 / 32.83127.43 / 31.05
MOW—46.86 / 12.21108.13 / 33.15110.08 / 35.00

Higher F1 and lower distance error are better. A dash marks an unreported result. On MOW, the paper reports proximity for the single-frame model only. Paper, Table 1.

Contact predictions compared with geometry-derived reference annotations
Fig. 8 Compare each predicted contact map with the geometry-derived reference for the same hand crop to see where contact is missed or extends beyond the reference region.

On DexYCB, adding physics supervision to the temporal model reduces mean position error from 5.7 cm to 3.3 cm and rotation error from 45.2° to 29.4° in a 0.4-second rollout. This evaluation uses the model’s force prediction at each video timestamp and motion-derived reference forces at the two intermediate simulation steps. It excludes divergent windows. The result measures short-horizon motion consistency under this protocol. Individual vertex forces remain estimates that depend on assumptions about inertia, friction, and support.

Refining hand–object reconstruction

We use the predicted contact and proximity fields to refine reconstructions from EasyHOI and ArtHOI. In each method, we replace the original contact objective while keeping the rest of the optimization fixed. These experiments use metric object geometry, and EasyHOI also uses the ground-truth object orientation. Across DexYCB, HOCap, and MOW, the predicted fields reduce aligned hand-vertex error for both methods.

During refinement, the predicted fields stay fixed while the reconstruction changes. Proximity specifies a target separation for each vertex. Contact probability weights a second term that encourages the hand and object surface normals to face one another at likely contacts. On DexYCB, these terms reduce aligned hand-vertex error from 9.94 to 9.08 mm for EasyHOI and from 7.32 to 4.63 mm for ArtHOI.

How the refinement constraints act

On this ARCTIC bottle example, the animation first aligns opposing surface normals at likely contacts, then adjusts the hand to match the predicted gaps. It illustrates the two objectives on annotated geometry.

HOCap reconstructions with contact artifacts highlighted in the HACO variants
Fig. 9 The red circles highlight contact artifacts in the HACO variants on HOCap. The corresponding TactoHand reconstructions use both contact and proximity during refinement.
Reconstruction accuracy Lower is better ↓
DatasetMethodPA-VPE (mm)Hand–object CD (10⁻³ m²)
Base+ HACO+ TactoHandBase+ HACO+ TactoHand
DexYCBEasyHOI9.9379.5169.07724.14012.5522.004
DexYCBArtHOI7.3176.0274.62611.58713.6271.431
HOCapEasyHOI9.1799.0517.59259.41668.94311.461
HOCapArtHOI6.9846.6846.0735.2345.1991.104
MOWEasyHOI14.83214.27613.93969.08072.63815.266
MOWArtHOI13.57213.26113.10073.99373.14732.413

Lower values are better for both aligned hand-vertex error (PA-VPE) and bidirectional mean squared hand–object Chamfer distance (CD). Paper, Table 3.

Citation

@article{xi2026tactohand,
  title   = {TactoHand: Learning Geometry-Based Dense Hand-Surface Interaction Fields from Video},
  author  = {Xi, Ziheng and Li, Weibo and Yu, Jiayi and Zheng, Sipeng and
           Zhang, Wanpeng and Xu, Haiweng and Luo, Hao and Mao, Chuan and Lu, Zongqing},
  journal = {arXiv preprint arXiv:2610.39585},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.39585}
}

來源:BeingBeyond:具身智能研究 · research.beingbeyond.com