PhysHSI Reference-to-Policy Technical Plan

日期：2026-05-28

0. 结论

本项目建议采用四阶段路线。这里的 Stage 1 不是从零新做，仓库里已经有 utils/sage_task_generator/ 和 utils/kimodo/plan_to_constraints.py。本文档把它放在 pipeline 里，只是为了说明后续步骤直接复用这些已有输出。不建议把 Stage 1 包装成新的复杂接口或论文贡献。

SAGE long-horizon task
  -> existing task_plan.json / task_validation.json / Kimodo constraints
  -> Kimodo / HOI-FHLI / other kinematic reference generation
  -> ResMimic-style physics teacher and residual refinement
  -> field-conditioned state-based policy by BC / DAgger / RL

其中：

第 1 阶段直接复用已有 SAGE task/constraint 代码，只补一个很薄的 batch manifest。
第 2 阶段是 reference/data factory，目标是从 SAGE 场景和任务自动生成大量 HSIReference。
第 3 阶段是 physics compiler / teacher，目标是把 kinematic references 变成 full-physics 可执行的 expert rollouts。
第 4 阶段才是最终 policy，test-time 不输入完整 reference motion，只输入 proprioception、task command、object state 和 body-part sampled fields。

推荐论文主张：

We compile long-horizon SAGE human-scene tasks into multi-source kinematic references,
physically refine them into expert state-action trajectories, and distill a generalized
field-conditioned humanoid policy that jointly handles cluttered obstacle avoidance and
active object interaction.

术语约定：

Kimodo/HOI-FHLI 阶段输出的是 constraints 和 references。
ResMimic-style 阶段输出的是 physically valid expert trajectories。
Field 主要出现在 physics reward、student observation 和 final policy training 中。

1. 研究目标

目标是在 SAGE-style cluttered 3D scenes 中完成 long-horizon physical human-scene interaction：

navigate through clutter
sit / stand
pick up object
carry object through clutter
place object
push / pull / open / close

现有 baseline 的典型短板：

TokenHSI / UniHSI / LessMimic-style interaction 方法能做交互，但对 crowded obstacle avoidance 不是核心能力。
HumanoidPF / CAT / APEX-style 方法强化 traversal 和 obstacle avoidance，但不覆盖 sit、carry、place 等 active object interaction。
Kimodo / HOI-FHLI 等 kinematic generators 可以产生有用 reference，但 reference 不保证 full-physics 成功，也不能直接作为 test-time policy。

本项目的核心价值是把三件事连成闭环：

大规模自动造 paired interaction references。
用 full physics 把 reference 编译成成功轨迹和 expert action-state data。
训练一个 generalized state-based policy，使其不依赖 test-time reference motion。

2. 当前代码基础

本仓库已经有可复用骨架：

模块	现状	本方案中的角色
`utils/sage_task_generator/`	可读 SAGE scene，生成 `scene_context.json`、`task_plan.json`、`task_validation.json`	已有 Stage 1 task graph/task plan 能力，直接复用
`utils/kimodo/plan_to_constraints.py`	将 SAGE plan 转成 Kimodo root path / heading / timeline	已有 Stage 1 到 Stage 2 的 Kimodo constraint adapter，直接复用
`utils/kimodo/`	已有 SOMA keyframe pipeline、SMPLX-RP scripts、render utilities	Kimodo reference generation 和 visualization
`submodules/kimodo`	支持 Kimodo SOMA、SMPLX22、G1 等 motion representation	Kinematic reference source
`submodules/hoifhli_release`	HOI-FHLI official code，SMPL-X based，含 object trajectories 和 physics tracking json	Pick/carry/place reference source
`submodules/ProtoMotions`	Full-physics tracking/control stack	Stage 3 physics teacher/refinement
`submodules/PHC`	SMPL/SMPL-X/PHC 相关 tracking 方向	SMPLX/SMPLH physics adapter 备选
`utils/tracking_adapters/`	已有 ProtoMotions/PHC/SAGE trace adapters	Stage 3 data bridge
`utils/metrics/`	success/collision/fall/big-force metrics	统一评测
`baseline/TokenHSI`, `baseline/UniHSI`, `baseline/CLoSD`, `baseline/UniPhys`	已接入或可接入 SAGE adapter	Baseline comparison

缺口不是“从零实现 task graph / constraints”，也不是发明复杂 schema。真正要做的是 reference adapters、physics refinement 和 final student policy training。

3. 数据产物设计

3.1 Thin Task Manifest

不要新造复杂的 TaskGraph / MotionConstraintBundle 系统。已有 task_plan.json、task_validation.json 和 Kimodo constraints 应该继续作为 source of truth。

这里最多需要一个很薄的 task_bundle.json，作用是把一次实验需要的文件路径和少量索引信息放在一起，方便批量跑 Kimodo、HOI-FHLI、physics 和 evaluator：

{
  "scene_id": "sage_xxx",
  "task_id": "carry_box_to_table_001",
  "task_plan_path": ".../task_plan.json",
  "task_validation_path": ".../task_validation.json",
  "kimodo_constraints_path": ".../constraints.json",
  "active_objects": ["box_17"],
  "target_objects": ["table_03"],
  "task_type": "pick_carry_place"
}

这个 manifest 的意义只有三点：

避免每个后续脚本重新猜 scene/task/object 路径。
保证 Kimodo、HOI-FHLI、physics refinement 和 metrics 用的是同一个任务实例。
记录可复现实验所需的输入文件。

它不是论文贡献，也不应该变成新的 ontology。

3.2 HSIReference

Stage 2 所有 generator 都统一导出 HSIReference：

reference_id
source: kimodo | hoifhli | other
scene_id
task_id
fps
human_model_type: smplx22 | smplx52 | soma30 | soma77 | g1
root_trans[T, 3]
body_local_rot[T, J, ...] optional
global_joints[T, K, 3]
body_markers[T, M, 3]
object_tracks:
  object_id:
    pos[T, 3]
    quat[T, 4]
    mesh_or_bbox
contacts:
  phase, object_id, body_part, start, end, confidence
generator_scores:
  kinematic_collision
  object_goal_error
  contact_error
  foot_skate
  path_deviation

关键原则：

不强行让 HOI-FHLI 直接兼容 Kimodo。
不强行把 SMPL-X 直接转 SOMA。
所有 source 先落到统一 reference schema，再分别接 physics tracker。

3.3 PhysicsExpertTrajectory

Stage 3 输出：

episode_id
reference_id
scene_id
task_id
policy_teacher_id
states[T]
actions[T]
root_state[T]
joint_state[T]
object_state[T]
contact_state[T]
field_samples[T]
rewards[T]
success_flags
failure_reason
quality_scores

这是 Stage 4 BC / DAgger / RL 的核心训练数据。

4. Stage 1: Existing SAGE Task and Constraint Layer

这一阶段不是新的核心研发任务。当前代码已经有：

SAGE scene -> scene_context.json
SAGE scene/instruction -> task_plan.json
task_plan.json -> task_validation.json
task_plan.json -> Kimodo root/heading/timeline constraints

因此 Stage 1 的实际工作是：

reuse existing generator
write a thin task_bundle.json manifest
batch validate generated tasks
export only the minimum extra inputs needed by HOI-FHLI / physics / evaluator

新增工作量应控制在几十行到几百行 adapter 级别，不应占用主要研究时间。

4.1 输入

SAGE scene geometry
object metadata
object category and semantic labels
object bbox / mesh proxy / support surfaces
top-down navigability map
language instruction or generated task template

4.2 输出

existing task_plan.json
existing task_validation.json
existing Kimodo constraints.json
thin task_bundle.json with paths, ids and task type
small HOI-FHLI / physics / evaluator input files when needed

4.3 Existing Long-Horizon Task Extraction

已有 generator 中 VLM/LLM 的作用是提出 task plan / task graph：

instruction + scene summary -> ordered action nodes

但所有几何可行性必须由 deterministic validator 检查：

object exists
target support surface exists
root approach region is navigable
object is reachable
object category supports action
held object path has enough clearance
success checks are measurable

建议 action set：

navigate
reach
sit
stand
pick_up
carry
place
push
pull
open
close
touch
lie_down
get_up

MVP 只做：

navigate + sit
navigate + pick_up + carry + place
navigate + push

4.4 Constraint Export

已有 Kimodo adapter 已经能把 task plan 转成 root/heading/timeline constraints。这里需要补的是统一 bundle，以及 HOI-FHLI 和 physics/evaluator 需要的字段：

root path:
  sparse waypoints from free-space route

heading:
  face active object during reach/grasp/place

end-effector targets:
  hand to object grasp/contact anchor
  pelvis to seat support region
  hand to push handle/contact patch

object goals:
  object pose before/after carry/place/push

contact windows:
  when hands should attach/release
  when pelvis should contact seat
  when feet should remain on support

这里的输出是 constraints，不命名为 field。

4.5 验收标准

MVP 验收：

50 个 SAGE scenes 中，至少 45 个能通过现有 generator 生成合法 task plan。
每个 task plan 都有一个 task_bundle.json 指向已有 plan、validation 和 constraints。
Kimodo constraints 继续保持现有可用性。
HOI-FHLI / physics / evaluator 只补必须字段，不引入大而全的新接口。
task_validation.json 能明确列出 success checks 和 failure checks。
人工抽查 20 个 long-horizon plans，动作顺序、目标物体、接触窗口合理。

5. Stage 2: Multi-Source Kinematic Reference Generation

5.1 Kimodo Reference Source

Kimodo 用于生成：

root-path following
walk-to-sit
reach / body keyframe
fullbody pose constraints
style-varied motion proposals

Kimodo 不负责最终物理成功。它的角色是 scalable reference generator。

Kimodo adapter 应支持：

existing Kimodo constraints.json -> Kimodo generation
Kimodo motion.npz -> HSIReference
Kimodo SMPLX AMASS-style output -> HSIReference
Kimodo SOMA output -> HSIReference via global joints / markers

5.2 HOI-FHLI Reference Source

HOI-FHLI 用于生成：

pick / carry / place
long sequence object interaction
human-object paired trajectory
SMPL-X body and hand pose
object pose over time
contact phase hints

本地代码确认 HOI-FHLI 是 SMPL-X based：

root_trans[T, 3]
body local axis-angle: 22 joints
finger local axis-angle: 30 joints
object COM and rotation
physics_tracking seq_*.json with humanoid/object frame states

HOI-FHLI adapter 应支持：

HOI-FHLI pkl/json -> HSIReference
object trajectory -> object_tracks
finger pose/contact -> grasp/contact labels
SMPL-X joints -> global_joints/body_markers

5.3 Multi-Proposal Strategy

每个 task 不只生成一个 reference。建议生成：

N_kimodo = 4 to 8
N_hoifhli = 2 to 4
optional procedural variants = 2 to 4

然后用 cheap kinematic scorer 过滤：

score = w_task * task_goal_error
      + w_contact * contact_anchor_error
      + w_collision * kinematic_collision
      + w_object * object_goal_error
      + w_smooth * foot_skate_or_acceleration
      + w_path * root_path_deviation

只把 top-k 送进 Stage 3 full-physics refinement。

5.4 验收标准

MVP 验收：

每类任务至少能生成 3 个 kinematic candidates。
HSIReference schema 可同时承载 Kimodo 和 HOI-FHLI 输出。
对 pick/carry/place，reference 必须包含 object track 或 object goal，不接受只有 human motion 的数据。
kinematic scorer 能淘汰明显穿墙、脚滑、抓空、放错物体的 candidates。

6.1 目标

Stage 3 把 kinematic reference 编译成 full-physics expert data：

kinematic reference
  -> tracking policy
  -> residual refinement
  -> successful physical rollout
  -> state-action expert dataset

这一步解决的问题是：

Kimodo/HOI-FHLI motion 可能抓到视觉上正确的位置，但物理里没抓稳。
Object trajectory 可能和 humanoid contact dynamics 不一致。
拥挤场景中 kinematic motion 没有考虑 whole-body collision impulse。
Final student 需要 action-state pair data，而不是只需要 motion clips。

6.2 Physics Skeleton 决策

推荐使用统一中间 schema，避免过早绑定单一 skeleton：

HSIReference stores:
  global joints
  body markers
  optional local rotations
  object tracks
  contacts

Physics adapter 再选择目标 skeleton：

如果 SMPLX/PHC-X tracking 稳定，优先用 SMPLX/SMPLH 路线，HOI-FHLI 接入最自然。
如果 ProtoMotions/SOMA checkpoint 更稳，MVP 用 SOMA，HOI-FHLI 只通过 markers/contact/object tracks retarget 到 SOMA。
不把 detailed fingers 作为 MVP 必需项。MVP 把 hand contact 简化为 wrist/palm/end-effector contact。

这样可以降低格式风险。

6.3 Teacher/Residual Policy

ResMimic-style refinement 可以写成：

a_t = pi_track(s_t, r_t) + delta_pi(s_t, r_t, o_t)

其中：

s_t: humanoid physical state
r_t: reference local target / phase / object target
o_t: object state and scene state
delta_pi: residual action correction

训练 reward：

R = R_human_track
  + R_object_track
  + R_contact
  + R_task_success
  + R_scene_clearance
  + R_balance
  + R_style
  - R_energy
  - R_collision
  - R_force_spike

建议细分：

R_human_track:
  root pose
  key body markers
  selected joints

R_object_track:
  object position and orientation
  object final support/contact condition

R_contact:
  hand-object contact during grasp/carry
  pelvis-seat contact during sit
  no unwanted foot/object collision

R_scene_clearance:
  passive obstacle SDF margin
  held object SDF margin
  body-part collision penalty

R_task_success:
  per-node success
  final task success

6.4 Curriculum

建议课程：

Phase A: tracking in empty scene
Phase B: tracking with active object only
Phase C: tracking with passive obstacles as collision geometry
Phase D: residual refinement with object dynamics
Phase E: randomized object mass/friction/scale and clutter perturbation

可以借鉴 ResMimic 的 virtual object controller 思路：早期用较强 object stabilization 帮助 contact 学起来，随后逐步降低辅助。

6.5 Failure Filtering

每条 rollout 标记：

success
partial_success
fail_grasp
fail_object_goal
fall
human_scene_collision
held_object_scene_collision
excessive_force
timeout

训练数据分三类：

expert_success_buffer：BC/DAgger 主数据。
recovery_buffer：用于 recovery policy 或 DAgger hard negatives。
discard_buffer：明显错误 reference，不进入 policy。

6.6 验收标准

MVP 验收：

sit physical success rate 超过 60%。
pick/carry/place physical success rate 初期超过 30%，优化后超过 50%。
成功 rollout 能导出 state/action/object/contact/field samples。
可以复现每条 expert trajectory 的 video/trace 和 metrics。
对失败 reference 有可解释 failure reason。

7. Stage 4: Field-Conditioned Generalized State Policy

7.1 目标

Stage 4 训练最终 policy：

pi_student(a_t | s_t, c_t, F_t)

test-time 输入：

proprioception
task command
active object state
compact body-part field samples

test-time 不输入：

full reference motion
future human pose sequence
Kimodo generated trajectory
HOI-FHLI generated trajectory

7.2 Observation

建议 observation：

proprio:
  root velocity
  root orientation
  joint positions / velocities
  contact sensors

task command:
  action token
  phase token
  target object id/category
  goal position in egocentric frame
  desired object final pose

object state:
  active object pose
  active object velocity
  object relative pose to hands/root
  held flag

F_geo:
  passive obstacle signed distance
  obstacle normal / gradient
  goal-conditioned guidance direction
  clearance margin

F_inter:
  active object distance to body parts
  active object SDF gradient
  grasp/contact anchor vectors
  support plane distance
  desired relative flow for carry/place/push
  contact on/off phase

Body parts sampled：

pelvis
head
torso
left/right shoulder
left/right elbow
left/right wrist or palm
left/right knee
left/right foot
held object bbox corners or keypoints

7.3 Field Definitions

F_geo: Passive Obstacle Field

HumanoidPF-style computed field：

occupancy -> SDF
SDF gradient -> obstacle normal
goal and obstacle occupancy -> geodesic guidance
body-part samples -> compact policy observation

用途：

observation:
  policy knows where nearby obstacles are and how to move around them

reward:
  body-part collision penalty
  guidance alignment near clutter
  held-object clearance

MVP 可以先做 2.5D/3D voxel field，不需要 neural field。

F_inter: Active Object Interaction Field

MVP 先用 computed object geometry and task phase：

object SDF
object part/contact anchors
body-part-to-anchor vector
body-part-to-object distance
desired contact window
desired object flow
support surface distance
release/place target

进阶版本学习 interaction latent：

successful expert rollouts -> field trajectory encoder / VAE
current policy field trajectory -> discriminator or AIP reward
latent z_inter captures grasp/sit/carry/push phase geometry

这样可以参考 LessMimic 的 DF latent/AIP 思路，但不依赖其未发布代码。

F_task: Command and Phase

F_task 不是几何 field，而是 task conditioning：

current node id
action token
phase token
target object id/category
goal id/category
remaining node count

7.4 Training

训练分三步：

Step 1: BC Warm Start

expert_success_buffer -> supervised behavior cloning
loss = action MSE + optional value/phase prediction + contact prediction

目的：

learn basic actuation
learn phase-conditioned contact behavior
avoid pure RL cold start

Step 2: DAgger

student rollout in randomized scenes
teacher/refiner labels correction action when student drifts
append corrected states to dataset
repeat

重点采样：

near obstacle
near grasp
object slipping
pre-place
post-contact recovery

Step 3: RL Fine-Tuning

reward：

R = R_task
  + R_contact
  + R_geo
  + R_inter
  + R_style
  - R_collision
  - R_fall
  - R_energy
  - R_force_spike

如果实现 AIP/AMP：

R_style: AMP over humanoid motion
R_inter: AIP/discriminator over field/contact trajectories

RL 阶段不再使用 dense reference tracking reward。可以保留 sparse command and phase guidance。

7.5 验收标准

MVP 验收：

Student 在 held-out SAGE scenes 中不输入 reference motion。
navigate+sit 成功率高于 raw Kimodo/TokenHSI/UniHSI。
carry-through-clutter 的 human-scene collision 和 held-object-scene collision 明显下降。
w/o F_geo collision 上升。
w/o F_inter grasp/place/sit 成功率下降。
w/o physics refinement 成功率和 force metrics 下降。

8. Dataset and Experimental Splits

8.1 MVP Split

建议先做：

SAGE-HSI-MVP-100
  train: 70 scenes
  val: 10 scenes
  test: 20 scenes

tasks:
  walk-to-sit
  pick-carry-place
  push-to-target

每个 scene 生成 3 到 5 个 tasks。

8.2 Paper Split

后续扩展：

SAGE-Crowd-1k
  train: 800 scenes
  val: 100 scenes
  test: 100 scenes

并按维度分层：

clutter density: low / medium / high
task length: 2 / 3 / 5+ nodes
object category: seat / box / table / door / drawer
object size: small / medium / large
layout novelty: seen categories / unseen layouts / unseen categories

8.3 Scale-Up

SAGE-10k 不作为三个月内必须完成目标。它是 final scaling target：

use reference factory to generate large candidate pool
physics teacher filters and refines a smaller successful subset
student policy trains on success + recovery buffers

9. Baselines

9.1 Required Baselines

Baseline	比较目的
TokenHSI	HSI task token baseline，验证 crowded collision 问题
UniHSI	Chain-of-Contacts / prompted HSI baseline
Kimodo raw reference	证明 kinematic reference 本身不等于 physical success
HOI-FHLI raw/reference tracking	证明 human-object generation 需要 physics refinement
Tracking-only teacher without residual	证明 residual refinement 的价值
HumanoidPF/CAT-style traversal	只在 navigation/avoidance 子任务上比较，说明 traversal field 很强但没有 active object interaction
LessMimic-style DF interaction	如果官方代码不可用，则用相同 observation family 的 ablation/reimplementation 对齐

9.2 Fairness Policy

三个月内不建议把所有 baseline 都在 SAGE-1k 上重新训练。可分阶段：

MVP:
  use official/pretrained baselines with SAGE adapters when possible
  report as zero-shot or adapter-based comparison
  focus on failure taxonomy and collision metrics

Paper:
  retrain or fine-tune feasible baselines on the same train split
  if retraining infeasible, state compute/code limitation and compare in a clearly labeled setting

必须公平的部分：

same test scenes
same task graph
same object goals
same success/collision metrics
same simulator validation
same number of rollout attempts where applicable

9.3 Ablations

必做：

Ours w/o F_geo
Ours w/o F_inter
Ours w/o physics refinement
Ours w/o DAgger
Ours BC only
Ours Kimodo-only references
Ours HOI-FHLI-only references
Ours tracking-only teacher

建议：

body-part field sampling vs root-only field
computed F_inter vs learned interaction latent
single proposal vs multi-proposal reference factory
with held-object clearance vs without held-object clearance

10. Metrics

10.1 Task Success

overall success
per-node success
sit success
stand recovery success
grasp success
carry success
place success
push success

10.2 Physical Validity

fall rate
mean penetration depth
max penetration depth
human-scene collision count
held-object-scene collision count
unwanted active-object collision
big-force count
contact impulse spikes
foot sliding
energy / torque

10.3 Interaction Quality

hand-object contact duration
object slip distance
object final pose error
support stability
seat contact quality
release timing error

10.4 Avoidance Quality

minimum body-part clearance
minimum held-object clearance
path efficiency
near-obstacle velocity alignment
collision-free traversal rate

10.5 Data Engine Quality

task graph validity rate
reference generation success rate
kinematic filtering pass rate
physics refinement success rate
expert data yield per scene
rollout wall-clock cost

11. Implementation Plan

Phase 0: Schema and Adapters

Time: week 1

Deliverables:

utils/hsi_reference/schemas.py
utils/hsi_reference/io.py
utils/hsi_reference/kimodo_adapter.py
utils/hsi_reference/hoifhli_adapter.py
docs/hsi_reference_data_generation.md

Acceptance:

Kimodo output -> HSIReference
HOI-FHLI output -> HSIReference
schema validator passes
small viewer can display human markers + object tracks

Phase 1: Thin Task Bundle and Batch Validation

Time: week 1 to 2

Deliverables:

task_bundle.json writer from existing task_plan.json / task_validation.json / constraints.json
Kimodo constraints export smoke test
minimal HOI-FHLI prompt/planner export
minimal physics/evaluator config export

Acceptance:

50 MVP scenes produce valid task plans with existing generator
all generated task bundles point to existing files and required object ids
all generated tasks have measurable success checks
manual inspection passes for selected scenes

Phase 2: Reference Factory

Time: week 2 to 4

Deliverables:

multi-proposal generation runner
kinematic scorer
reference dataset manifest
reference preview renderer

Acceptance:

at least 300 references over MVP tasks
top-k scorer removes obvious bad samples
references include object tracks for carry/place

Phase 3: Physics Teacher

Time: week 4 to 7

Deliverables:

ProtoMotions/PHC adapter for HSIReference
tracking reward config
residual refinement config
success/failure logger
expert trajectory exporter

Acceptance:

physical sit success > 60%
physical carry/place success > 30% initially
state-action expert data exported
failure reasons logged

Phase 4: Field Builder and Student Policy

Time: week 7 to 10

Deliverables:

utils/field_policy/geo_field.py
utils/field_policy/inter_field.py
utils/field_policy/observation.py
BC training script
DAgger data collection script
RL fine-tuning config

Acceptance:

student can run without full reference motion
w/o F_geo ablation increases collision
w/o F_inter ablation reduces interaction success

Phase 5: Evaluation and Paper Figures

Time: week 10 to 12

Deliverables:

baseline runner
metrics table
failure taxonomy
ablation table
qualitative videos
dataset/data-engine statistics

Acceptance:

MVP results complete
clear comparison against TokenHSI/UniHSI/Kimodo/HOI-FHLI/tracking-only
all claims backed by metrics

12. Paper Positioning

Suggested title direction:

PhysHSI: Compiling Generated Human-Scene References into Field-Conditioned Physical Interaction Policies

or:

Reference-to-Policy Physical Human-Scene Interaction in Cluttered 3D Scenes

Core contributions:

A multi-source kinematic reference factory using existing SAGE task plans, Kimodo, HOI-FHLI and compatible generators.
A ResMimic-style physics compiler that converts generated references into successful expert state-action data.
A field-conditioned state policy that unifies passive obstacle avoidance and active object interaction.
A cluttered-scene benchmark protocol with interaction success, collision, held-object clearance and physical validity metrics.

Main experimental story:

Generated kinematic references provide scale.
Physics refinement provides validity.
Body-part fields provide generalization to clutter and interaction.

13. Key Design Decisions

13.1 Field Belongs to the Policy Interface

Use fields for:

student observation
physics reward
collision/contact diagnostics

Do not require fields for:

Kimodo generation internals
HOI-FHLI generation internals
VLM task extraction

13.2 Active Object vs Passive Obstacle

Every task should label:

active object:
  object to sit on, grasp, carry, place, push, open, close

passive obstacles:
  objects that must be avoided by human body and held object

An object can change role across phases:

chair:
  passive obstacle during navigation
  active support during sit

box:
  active grasp/carry object
  collision body against scene while held

table:
  passive obstacle during approach
  active support during place

13.3 Reference Motion Is Training Data, Not Test-Time Input

The final claim should always state:

At test time, the policy does not receive a full future reference motion.
It receives compact task commands, object state and fields.

This separates the method from pure tracking systems.

13.4 Detailed Fingers Are Optional for MVP

For MVP:

hand interaction = wrist/palm/end-effector contact

For paper-scale:

add SMPL-X/MANO fingers when physics stack is stable

This avoids blocking the whole project on hand dynamics.

14. Open Technical Questions

Which physics skeleton is most stable today: ProtoMotions/SOMA or PHC-X/SMPLX?
Can Kimodo-SMPLX access and generation be made reliable locally?
How much of HOI-FHLI object trajectory can be reused directly before physics refinement?
Is object grasp best represented as attach constraint, soft contact reward, or learned residual?
How expensive is physics refinement per candidate, and how many candidates per task can we afford?
Should learned F_inter be included in first paper version, or kept as ablation/future extension?

These questions do not change the high-level pipeline. They determine the first implementation branch.

15. Immediate Next Steps

Implement HSIReference schema and validators.
Export one Kimodo sit motion and one HOI-FHLI carry motion into the schema.
Build a simple viewer for human markers, object tracks and contact windows.
Select the first physics skeleton by running one tracking smoke test.
Implement F_geo from SAGE proxy occupancy and sample it at body parts.
Implement MVP F_inter from active object SDF/contact anchors.
Run one full loop:

SAGE scene
  -> existing task_plan / constraints
  -> thin task_bundle.json
  -> reference
  -> physics rollout
  -> expert data
  -> BC student
  -> held-out evaluation

The first complete loop is more important than scaling. Once one loop works, scaling to more scenes and tasks becomes a data-engine problem.

PhysHSI Reference-to-Policy Technical Plan

0. 结论

1. 研究目标

2. 当前代码基础

3. 数据产物设计

3.1 Thin Task Manifest

3.2 HSIReference

3.3 PhysicsExpertTrajectory

4. Stage 1: Existing SAGE Task and Constraint Layer

4.1 输入

4.2 输出

4.3 Existing Long-Horizon Task Extraction

4.4 Constraint Export

4.5 验收标准

5. Stage 2: Multi-Source Kinematic Reference Generation

5.1 Kimodo Reference Source

5.2 HOI-FHLI Reference Source

5.3 Multi-Proposal Strategy

5.4 验收标准

6. Stage 3: ResMimic-Style Physics Teacher and Residual Refinement

6.1 目标

6.2 Physics Skeleton 决策

6.3 Teacher/Residual Policy

6.4 Curriculum

6.5 Failure Filtering

6.6 验收标准

7. Stage 4: Field-Conditioned Generalized State Policy

7.1 目标

7.2 Observation

7.3 Field Definitions

F_geo: Passive Obstacle Field

F_inter: Active Object Interaction Field

F_task: Command and Phase

7.4 Training

Step 1: BC Warm Start

Step 2: DAgger

Step 3: RL Fine-Tuning

7.5 验收标准

8. Dataset and Experimental Splits

8.1 MVP Split

8.2 Paper Split

8.3 Scale-Up

9. Baselines

9.1 Required Baselines

9.2 Fairness Policy

9.3 Ablations

10. Metrics

10.1 Task Success

10.2 Physical Validity

10.3 Interaction Quality

10.4 Avoidance Quality

10.5 Data Engine Quality

11. Implementation Plan

Phase 0: Schema and Adapters

Phase 1: Thin Task Bundle and Batch Validation

Phase 2: Reference Factory

Phase 3: Physics Teacher

Phase 4: Field Builder and Student Policy

Phase 5: Evaluation and Paper Figures

12. Paper Positioning

13. Key Design Decisions

13.1 Field Belongs to the Policy Interface

13.2 Active Object vs Passive Obstacle

13.3 Reference Motion Is Training Data, Not Test-Time Input

13.4 Detailed Fingers Are Optional for MVP

14. Open Technical Questions

15. Immediate Next Steps