HSI Reference Data Generation

日期：2026-05-28

0. 目标

本文件只讨论 reference data 如何生成，不讨论最终 RL policy 训练。

Reference data 的目标是把已有 SAGE task plan / constraints 变成可供 physics tracking/refinement 使用的 kinematic human-scene interaction reference：

SAGE task_plan.json / task_validation.json / constraints.json
  -> Kimodo and/or HOI-FHLI generation
  -> HSIReference
  -> kinematic filtering
  -> physics tracking / residual refinement

这里统一叫 HSIReference。不要用 paired-HOI 这类命名，因为 reference 不一定都有 active object，也不一定只表示 human-object pair；它可以是 navigation、sit、lie、 push、pull、carry 或这些技能的 long-horizon 组合。

1. Reference 不是最终控制器

Kimodo 和 HOI-FHLI 的输出都只作为 training/reference data：

• 用于产生 full-body kinematic motion。
• 用于提供 object trajectory、contact window、hand pose 等交互线索。
• 用于 Stage 3 的 tracking policy / residual refinement。
• 不作为 test-time final policy 的完整未来轨迹输入。

最终 policy 仍然应该是 state-based policy：

proprioception + task command + object state + scene/interaction fields -> action

2. 应覆盖的 Interaction Primitives

2.1 MVP 必须覆盖

MVP skill set 应该覆盖 TokenHSI / UniHSI 常见技能，同时突出我们自己的 crowded interaction setting：

navigate
stand / idle
sit / stand_up
lie_down / get_up
lift / pick_up
carry / move_object
place / put_down
push
pull
carry_through_clutter
push_or_pull_through_clutter

推荐第一版任务：

1. navigate + sit
2. navigate + lie_down + get_up
3. pick/lift + carry/move + place/put_down
4. push
5. pull
6. pick/lift + carry/move + place/put_down through clutter

其中第 6 项是最重要的主线任务：它同时要求 active object interaction 和 crowded obstacle avoidance。

2.2 HOI-FHLI Release 明确支持的 Primitive

本地 submodules/hoifhli_release 的 sampling pipeline 里，interaction action 主要是三类：

lift
push
pull

lift 在 prompt 里对应：

Lift the {object}, move the {object}, and put down the {object}.

因此在本项目里映射为：

lift = pick_up + carry/move + place/put_down

HOI-FHLI 还支持：

navigation segment
interaction segment
finger/grasp generation
long sequence = navigation and interaction alternation

本地 release 中可见的测试对象包括：

monitor
trashcan
largebox
smallbox
plasticbox
suitcase
woodchair
whitechair
smalltable
largetable
floorlamp
floorlamp1
clothesstand
tripod
bottle
right_shoes1
toy1
toy2
laundrybasket
vase1_big
vase1_big2
box_a
box_b
box_c
newobject

对我们最有用的对象子集：

box / plasticbox / suitcase / laundrybasket:
  lift, carry, place

woodchair / whitechair:
  push, pull, carry-if-physics-allows

smalltable / largetable:
  push, pull, place support target

trashcan / monitor:
  lift/carry/place or push depending on size/mass proxy

2.3 不建议放进 3 个月 MVP 的 Primitive

以下任务不要作为 MVP 主线：

open washing machine
open drawer
open door
open fridge
close articulated object

原因：

• HOI-FHLI release 里没有看到 articulated appliance open/close 的直接支持。
• 这类任务需要 object joint state、handle contact、joint-limit reward、articulated asset。
• 它们更接近 VLM-RMD 的 setting，而不是现成 reference generator 可以直接解决的问题。

可以在 paper discussion / future work 里提到：

VLM-RMD-style reward/program synthesis is a possible extension for articulated objects.

但 3 个月 MVP 不应被 articulated object 任务拖住。

3. 什么时候用 Kimodo

Kimodo 负责 scene/path/body constrained human reference，尤其适合没有 active moving object，或者 active object 不是 reference generation 的核心时使用。

使用 Kimodo 的情况：

navigate
walk along SAGE route
approach target object
sit on chair/sofa/bed
stand_up
lie_down
get_up
reach / body keyframe
body pose constrained transition
human-only baseline reference

Kimodo 的主要输入：

root path / waypoints
heading target
body keyframes
pose constraints
timeline constraints

Kimodo 的主要输出：

human root trajectory
human local/global joint rotations
global joints or markers
optional SMPL-X AMASS-style output
optional SOMA output

Kimodo 不负责：

object COM trajectory
stable grasp
hand-object contact labels
physical object dynamics
held-object-scene collision

因此 Kimodo 对 sit/lie/navigate 是主 generator；对 carry 只能提供 root/path/body scaffold，不能单独作为 object interaction reference。

4. 什么时候用 HOI-FHLI

HOI-FHLI 负责 active object interaction reference。

使用 HOI-FHLI 的情况：

lift / pick_up
carry / move object
place / put_down
push
pull
navigation + interaction alternation
hand/finger contact generation
object trajectory generation

HOI-FHLI 的主要输入：

object name / object mesh proxy
action name: lift | push | pull
language text
planned path waypoints
initial object pose
final object pose
previous navigation end pose

HOI-FHLI 的主要输出：

SMPL-X body pose
finger pose
human root translation
global joints
object COM trajectory
object rotation trajectory
left/right hand contact window
left/right wrist pose relative to object

HOI-FHLI 不负责：

SAGE-scale scene collision guarantee
strict A* path following
full physics success
articulated object open/close in the current release

因此 HOI-FHLI 对 lift/carry/place/push/pull 是主 generator；但它的结果还需要 kinematic collision filtering 和 physics refinement。

5. 能否用 A* 轨迹控制 HOI-FHLI Carry

可以。HOI-FHLI 的 long-sequence pipeline 支持 planned path conditioning。

代码语义是：

navigation model:
  planned path = human root XY waypoints

interaction model:
  planned path = object COM XY waypoints

因此 carry-through-clutter 可以这样做：

SAGE occupancy / navmesh
  -> inflate obstacles by human radius + held object clearance
  -> A* from pickup position to place position
  -> smooth and downsample path
  -> convert to N x 3 waypoints
  -> HOI-FHLI lift interaction planned_obj_path
  -> HSIReference

注意：

• 这是 diffusion waypoint conditioning，不是 hard constraint。
• 它主要约束 object COM / human root 的 XY，不保证全身和 held object 都不撞。
• A* 需要先用 human body 和 held object bbox 膨胀障碍物。
• 生成后必须做 kinematic collision filtering。
• 最终物理可行性由 tracking/refinement 负责。

6. 什么时候两个都用

6.1 Carry Through Clutter

推荐同时使用 Kimodo 和 HOI-FHLI：

Kimodo:
  scene-aware approach path
  body/root motion scaffold
  optional pre-grasp and post-place transition

HOI-FHLI:
  lift/carry/place interaction segment
  object COM trajectory
  hand contact and finger pose

Physics refinement:
  merge, track, and repair contact/collision

具体做法：

1. 用 SAGE/A* 得到 pickup 到 place 的 collision-aware route。
2. 给 HOI-FHLI 生成 lift interaction，planned path 是 object COM path。
3. 用 Kimodo 生成 approach/retreat 或 sit/lie 等非 active-object 段。
4. 把所有 segment 导成 HSIReference。
5. 用 scorer 选出 collision 少、接触合理、路径偏差小的 candidate。
6. 送入 physics tracking/refinement。

6.2 Navigate + Sit / Lie

只用 Kimodo：

navigate path
heading target
pelvis/torso keyframe
sit/lie final pose

HOI-FHLI 不适合作为 sit/lie 主 generator。

6.3 Push / Pull Through Clutter

优先用 HOI-FHLI：

planned path = object COM path
action = push or pull

Kimodo 可以作为对照或补 approach/retreat segment。

6.4 Pure Navigation / Avoidance

只用 Kimodo 或 HumanoidPF/CAT-style traversal baseline。HOI-FHLI 不需要参与。

7. Generator Selection Rule

实现时可以用一个简单 router：

if task has no active moving object:
    use Kimodo

elif task type in {sit, lie_down, stand_up, get_up}:
    use Kimodo

elif task type in {lift, pick_up, carry, place, push, pull}:
    use HOI-FHLI

elif task type is carry_through_clutter:
    use HOI-FHLI for interaction segment
    optionally use Kimodo for approach/retreat and alternative root scaffold

elif task type is articulated_open_close:
    defer from MVP

Candidate count:

Kimodo-only tasks:
  generate 4-8 candidates

HOI-FHLI interaction tasks:
  generate 2-4 candidates per A* route

carry-through-clutter:
  generate 2-3 A* route variants
  generate 2-4 HOI-FHLI candidates per route
  optionally generate Kimodo approach/retreat candidates

8. HSIReference Format

Kimodo 和 HOI-FHLI 不需要直接格式兼容。它们只需要分别导出到同一个 HSIReference。

推荐 schema：

reference_id
scene_id
task_id
source: kimodo | hoifhli | hybrid | procedural
task_type
fps
duration
world_frame
human_model_type: smplx22 | smplx52 | soma30 | soma77 | g1 | markers_only

human:
  root_trans[T, 3]
  root_quat[T, 4] optional
  body_local_rot[T, J, ...] optional
  global_joints[T, K, 3]
  body_markers[T, M, 3] optional
  foot_contacts[T, ...] optional

objects:
  object_id:
    category
    role: active | passive | support | goal
    mesh_path optional
    bbox optional
    pos[T, 3] optional
    quat[T, 4] optional
    static_pose optional

contacts:
  body_part
  object_id
  start_frame
  end_frame
  type: grasp | support | push | pull | sit | lie | touch
  confidence

path:
  root_waypoints optional
  object_waypoints optional
  planned_by: sage_astar | hoifhli_random | kimodo_constraint | manual

metadata:
  generator_config
  source_files
  scores

Required fields by task:

navigate:
  human.root_trans
  human.global_joints
  path.root_waypoints

sit / lie:
  human.root_trans
  human.global_joints
  support object static_pose
  contact window if available

lift / carry / place:
  human.root_trans
  human.global_joints
  active object pos/quat
  hand-object contact window
  object_waypoints or object goal

push / pull:
  human.root_trans
  human.global_joints
  active object pos/quat
  hand-object contact window
  object_waypoints

9. Format Compatibility

Kimodo 和 HOI-FHLI 不需要共享 native format。兼容性分两层：

reference-level compatibility:
  both export HSIReference

skeleton-level compatibility:
  preferably use SMPL-X end-to-end if physics tracking supports it

推荐优先级：

preferred:
  HOI-FHLI-SMPLX + Kimodo-SMPLX + PHC-X/SMPL-X tracking

fallback:
  keep HSIReference skeleton-neutral
  retarget SMPL-X references to SOMA only if ProtoMotions/SOMA is the stable physics path

也就是说，能全用 SMPL-X 就优先全用 SMPL-X；不要优先选择 SMPL-X -> SOMA， 除非 physics stack 逼迫我们必须用 SOMA。

9.1 Kimodo to HSIReference

Kimodo outputs may be SOMA or SMPL-X depending on model/checkpoint.

Adapter behavior:

Kimodo output
  -> read root and local joints
  -> compute global joints / markers
  -> write HSIReference.human
  -> attach static scene objects from SAGE metadata
  -> attach path/root constraints from existing constraints.json

For Kimodo-only tasks, objects may contain only static support/passive objects.

For sit/lie:

support object = chair / sofa / bed / floor region
contact type = sit or lie
object trajectory = none

9.2 HOI-FHLI to HSIReference

HOI-FHLI is SMPL-X based and exports human/object quantities:

root_trans
human_jnts_local_rot_aa
finger_jnts_local_rot_aa
object COM position
object rotation matrix
left/right contact windows

Adapter behavior:

HOI-FHLI result
  -> human.root_trans
  -> human.body_local_rot
  -> human.global_joints
  -> objects[active].pos / quat
  -> contacts from left/right contact windows
  -> object_waypoints from planned_obj_path

Finger details should be optional in MVP. If the physics skeleton cannot use fingers, store wrist/palm contact and ignore detailed finger joints during tracking.

9.3 Hybrid Reference

Hybrid references should not splice raw joint rotations blindly. Prefer segment-level composition:

Kimodo approach segment
HOI-FHLI interaction segment
Kimodo retreat or transition segment

Then export:

source = hybrid
segments:
  [{source: kimodo, frames: ...}, {source: hoifhli, frames: ...}]

Before physics tracking, run:

root continuity check
heading continuity check
joint velocity spike check
hand-object contact continuity check
object pose continuity check

If continuity fails, keep segments separate and let physics teacher handle transitions with reset/phase conditioning first.

9.4 Can We Use SMPL-X for Everything?

Yes, if these components are stable enough:

HOI-FHLI:
  already SMPL-X based

Kimodo:
  use Kimodo-SMPLX-RP rather than Kimodo-SOMA

physics:
  use PHC-X / SMPL-X humanoid tracking rather than ProtoMotions/SOMA

This is the cleanest route because:

• HOI-FHLI already outputs SMPL-X body and finger motion.
• Kimodo has an SMPL-X model and AMASS-style SMPL-X export.
• No lossy cross-skeleton retargeting is needed.
• Hand/wrist contact from HOI-FHLI is easier to preserve.

The tradeoff is that Kimodo-SMPLX may be less robust than Kimodo-SOMA, and SMPL-X physics tracking may be harder to stabilize than a simpler SOMA humanoid. The first smoke test should compare:

Kimodo-SMPLX -> PHC-X/SMPL-X tracking
HOI-FHLI-SMPLX -> PHC-X/SMPL-X tracking
Kimodo-SOMA -> ProtoMotions/SOMA tracking
HOI-FHLI-SMPLX -> SOMA retarget -> ProtoMotions/SOMA tracking

If the first two work, use SMPL-X end-to-end.

9.5 How to Convert SMPL-X to SOMA

There is no clean parameter-level conversion:

SMPL-X pose parameters != SOMA joint rotations
SMPL-X skeleton topology != SOMA skeleton topology
SMPL-X body shape != Kimodo/ProtoMotions fixed SOMA body shape

So SMPL-X -> SOMA should be implemented as retargeting, not direct conversion.

Recommended retargeting pipeline:

SMPL-X reference
  -> run SMPL-X forward kinematics
  -> extract global joints / markers
  -> choose corresponding SOMA joints / markers
  -> solve SOMA IK per frame
  -> add temporal smoothness and contact constraints
  -> export SOMA HSIReference / Kimodo-SOMA NPZ / SOMA BVH

Minimal marker correspondence:

pelvis/root
spine / chest / neck / head
left/right shoulder
left/right elbow
left/right wrist or palm
left/right hip
left/right knee
left/right ankle
left/right toe / heel

IK objective:

E = w_marker * ||M_soma(q_t) - M_smplx(t)||^2
  + w_root * ||root_soma - root_smplx||^2
  + w_smooth * ||q_t - q_{t-1}||^2
  + w_vel * ||dq_t - dq_{t-1}||^2
  + w_foot * foot_contact_constraint
  + w_hand * hand_object_contact_constraint

For carry/push/pull, preserve interaction by adding:

hand-object relative pose during contact
object COM trajectory
contact start/end frames
held-object clearance metadata

Do not retarget fingers for MVP. Store finger motion in HSIReference if available, but use wrist/palm contacts for SOMA tracking.

Validation criteria for SMPL-X -> SOMA retargeting:

mean marker error < 5 cm
hand-object contact error < 5-8 cm during grasp/contact
foot skate not worse than source by more than 20%
root path deviation < 10 cm average
no large joint velocity spikes at segment boundaries

If these criteria are not met, do not use retargeted SOMA references for policy training; use SMPL-X tracking or keep the source as kinematic-only baseline.

10. Kinematic Filtering

Every generated HSIReference should be filtered before physics refinement:

root path deviation
object path deviation
human-scene collision proxy
held-object-scene collision proxy
hand-object contact error
support contact error
foot skate
joint velocity/acceleration spikes
object final pose error

Task-specific pass criteria:

navigate:
  root reaches goal
  no major scene collision

sit:
  pelvis near support
  body not penetrating support too much

lie:
  torso/pelvis aligned with support region
  no large body-scene penetration

lift/carry/place:
  hands near object during contact
  object follows planned path approximately
  object final pose near target
  held object has clearance from scene

push/pull:
  hand contact persists during motion
  object displacement matches goal
  object does not tunnel through obstacles

Only top candidates should enter physics refinement.

11. Recommended MVP Plan

Week 1:

implement HSIReference dataclass / json schema
write Kimodo -> HSIReference adapter
write HOI-FHLI -> HSIReference adapter
export one navigate+sit and one lift/carry/place example

Week 2:

add A* route export for carry-through-clutter
feed A* path to HOI-FHLI planned_obj_path
add kinematic viewer and scorer

Week 3-4:

generate reference set:
  navigate+sit
  navigate+lie
  lift/carry/place
  push
  pull
  carry-through-clutter

Week 5+:

send top references to physics tracking/refinement
collect successful state-action trajectories
start BC / DAgger student training

12. Summary

Use Kimodo for human-only and passive-scene interaction references:

navigate, sit, lie, stand/get up, approach/retreat, pose/path constrained motion

Use HOI-FHLI for active-object references:

lift/pick/carry/place, push, pull, hand contact, object trajectory

Use both for the key task:

carry-through-clutter = SAGE/A* route + HOI-FHLI interaction + optional Kimodo approach/retreat

Unify them through HSIReference, not by forcing Kimodo and HOI-FHLI to share native formats.