Egocentric head motion prediction

HAP

A Hand-Driven Active Perception Framework

Anticipating future human head motion from hand dynamics, interaction intent, and target-conditioned occlusion.

Anonymous

View code Documentation

Motivation

Head motion is part of perception.

During manipulation, the head redirects perception toward an interaction target while remaining coordinated with upper-body and hand motion. Treating future head motion as passive trajectory continuation misses both effects.

HAP predicts future 6-DoF human head motion from an observed interaction history. It infers target confidence from hand motion and object geometry, reasons over dynamic target-conditioned occlusion, and combines learned motion with a constant-velocity prior at each prediction horizon.

Method

From hand-driven intent to future head motion

HAP connects target inference, dynamic occlusion reasoning, and motion prediction in a single target-conditioned forecasting pipeline.

Overview of the HAP framework, from RGB-D observations and hand motion to target confidence, dynamic P-TAOG encoding, and future head motion prediction.
HAP estimates target confidence, constructs a dynamic P-TAOG, and fuses its temporal representation with observed hand-head motion.
01

Target intention

Observed 5D hand states and 8D hand-object geometry produce soft target probabilities over candidate objects.

02

Dynamic P-TAOG

A Predictive Target-centric Amodal Occlusion Graph represents current occlusion and potential risk among candidates.

03

Gated prediction

Directed PNA layers and a causal GRU encode graph evolution before a horizon-wise gate blends learned and constant-velocity trajectories.

EgoPAT3Dv2 + Bottle

Goal-directed RGB-D interactions

We evaluate HAP on EgoPAT3Dv2 and Bottle. Bottle contains 12 recording sessions and 382 reach-to-target clips with coordinated hand motion, viewpoint changes, and changing target visibility; its experiments use a session-disjoint 10/1/1 split. EgoPAT3Dv2 uses a subject-disjoint 773/84/118 train/validation/test split.

Bottle dataset examples showing reach-to-target phases, synchronized RGB-D observations, and recording diversity.

Held-out test sets

Head-motion prediction

Each cell reports EgoPAT3Dv2 / Bottle; translation is in millimeters and rotation in degrees.

Method tADE ↓ tFDE ↓ rADE ↓ rFDE ↓
CVH10.685 / 26.65623.600 / 57.3212.145 / 1.8774.469 / 3.946
UniHand24.934 / 58.75439.603 / 75.6854.929 / 3.9056.133 / 4.848
DKF20.934 / 36.35138.403 / 60.4192.922 / 2.9755.037 / 4.643
RVAE12.597 / 26.08824.338 / 42.5872.013 / 1.9773.683 / 3.125
DASE15.275 / 25.59629.510 / 43.5762.094 / 2.0673.781 / 3.218
VRNN22.195 / 38.71439.660 / 63.6833.105 / 3.0775.184 / 4.651
SRNN20.599 / 36.42339.028 / 60.0753.139 / 3.2105.200 / 4.727
AGF19.012 / 40.82133.933 / 65.9732.615 / 3.0944.540 / 4.914
OCT14.764 / 35.85827.462 / 57.1692.386 / 2.4914.053 / 3.876
USST19.665 / 40.23333.319 / 61.6883.023 / 3.1174.918 / 4.625
Diff-IP2D28.515 / 53.84944.141 / 76.8284.778 / 3.8836.159 / 4.926
MADiff24.601 / 38.29244.312 / 63.5093.774 / 3.0486.190 / 5.020
HAP (DA3)10.563 / 18.56322.251 / 36.6192.091 / 1.4954.123 / 2.760
HAP (ORB-SLAM3)8.897 / 19.08718.098 / 36.2831.538 / 1.7313.178 / 3.115

32.3%

HAP achieves the strongest overall results among the evaluated methods. EgoPAT3Dv2 has substantially lower translational errors, while Bottle tests prediction under frequent, evolving target occlusion.

The fixed target-intention estimator reaches 92.79% top-1 accuracy (206/222) on the curated EgoPAT3Dv2 validation split. On Bottle, Full HAP reports 18.563 / 36.619 mm tADE/tFDE and 1.495 / 2.760 deg rADE/rFDE; the horizon-wise gate improves both motion-only and P-TAOG-conditioned predictors.

Qualitative comparison

Following future direction and magnitude

Across independent test clips, target-conditioned context helps HAP reduce endpoint drift and better match changes in future head motion.

Qualitative 3D trajectory predictions from HAP, UniHand, MADiff, CVH, and RVAE across eight test clips.

Visualizations

Egocentric reach-to-target sequences

Short observations from the Bottle interaction set.

Sequence 01
Sequence 02
Sequence 03
Sequence 04
Sequence 05
Sequence 06

Citation

BibTeX

@misc{anonymous2026hap,
  title  = {HAP: A Hand-Driven Active Perception Framework for
            Egocentric Head Motion Prediction},
  author = {Anonymous},
  year   = {2026}
}