PhysHSI Refiner Development README

本文档定义 PhysHSI 后续 physics refiner 的数据组织、data loading 方式和代码落地方案。

它是一个开发期 README，目标是让后续 coding agent 和人类开发者按同一套接口写代码。高层路线见 docs/sugar_inspired_physhsi_framework_plan_20260611.md；本文件以工程实现为准。

1. 核心原则

Refiner 的科学起点是把一条 KIMODO-generated kinematic HSI / HOI trajectory 变成物理可执行 teacher trajectory。允许 per-reference policy、pretrained tracker adaptation、RePHO-style alternating refinement、shared refiner或 trajectory optimization作为独立 baseline。
单条 case overfit 是 architecture selection 和数据可行性实验，不代表最终 scaling方案。选定family后，多条trajectory组成motion pool并训练shared refiner / expert，语义对齐SUGAR和InterPrior的full-reference expert阶段。
case_000001/ 是数据存储单位。它可以对应一个per-reference refiner run，也可以被shared refiner采样；训练单位必须由实验config显式声明。
Refiner/teacher允许读取dense reference和privileged physics state。最终 state-based student是独立阶段，部署时只给current state和sparse task goals，不读取dense HOI reference。Dynamic pick–place首先使用object goal state；通用HSI/HOI再比较TokenHSI-style task tokens、UniHSI-style contact chain和 VLM-RMD-style relation goals。
PhysHSI 使用 digital human motion 表示，不把数据格式锁死在 SMPL-X、SOMA 或 G1；具体 source skeleton 和 tracker backend 由 manifest.json 与运行 config 控制。
不 hardcode 50 Hz。每条 case 的真实 fps / dt 写在 manifest.json。
ProtoMotions 是运行时依赖，可以从 submodules/ProtoMotions import。
SUGAR、InterPrior、RePHO 和 ResMimic 只作为算法与数据流参考；除独立 adapter/reproduction外，不作为正式主pipeline的隐式运行时依赖。
第一版接口优先简单、文件驱动、容易 debug。不要提前设计复杂 type system。

2. Runtime Environment

MVP refiner 代码默认运行在 Isaac Lab / ProtoMotions stack，不是 Isaac Sim standalone，也不是单纯离线 NumPy pipeline。

3. 和 SUGAR 的对应关系

SUGAR 的 refiner dataset 是 folder-based dataset：

data/PushBox/
  data_000/
    robot_50hz.npz
    obj_motion_global_50hz.pkl
    contact_labels_50hz.npy

  data_001/
    robot_50hz.npz
    obj_motion_global_50hz.pkl
    contact_labels_50hz.npy

训练启动时，SUGAR 的 loader 会扫 motion_folder/data_*，把所有 trajectory 读入内存并 pad 成 [num_motion, max_T, ...] tensor。训练 reset 时，每个 parallel env 采样一个 motion_id 和一个 timestep。

PhysHSI 保留这个思想，但换成 skeleton-agnostic 的文件名：

SUGAR:
  data_000/robot_50hz.npz
  data_000/obj_motion_global_50hz.pkl
  data_000/contact_labels_50hz.npy

PhysHSI:
  case_000001/motion.npz
  case_000001/tracker_motion.motion
  case_000001/object.pkl
  case_000001/contact.npy
  case_000001/manifest.json

For the first MVP, motion.npz is the canonical Kimodo-SOMA kinematic motion, and tracker_motion.motion is the ProtoMotions soma23 tracker input converted from it. The older 22-joint Kimodo cached motions are only used as case seeds and metadata sources.

4. Dataset Layout

推荐的 refiner 数据根目录：

data/refiner/
  prior/
    train.txt
    val.txt
    test.txt

    cases/
      case_000001/
        manifest.json
        motion.npz
        tracker_motion.motion
        object.pkl
        contact.npy
        scene_context.json
        task_plan.json
        task_validation.json

      case_000002/
        manifest.json
        motion.npz
        tracker_motion.motion
        object.pkl
        contact.npy
        scene_context.json
        task_plan.json
        task_validation.json

  refined/
    run_YYYYMMDD_HHMMSS/
      config.json
      metrics_summary.json

      cases/
        case_000001/
          manifest.json
          motion.npz
          object.pkl
          contact.npy
          rollout.npz
          metrics.json

        case_000002/
          manifest.json
          motion.npz
          object.pkl
          contact.npy
          rollout.npz
          metrics.json

4.1 `prior/`

prior/ 存 Kimodo / K-model / motion generation 阶段产生的 kinematic HSI reference。它不要求已经物理可执行。

train.txt、val.txt、test.txt 每行写一个 case id 或相对路径，推荐第一版只写 case id：

case_000001
case_000002
case_000003

loader 读取时实际路径为：

data/refiner/prior/cases/<case_id>/

4.2 `refined/`

refined/ 存训练好的 refiner policy rollout 后的物理可执行 trajectory。每次实验写入一个新的 run_* 目录，不覆盖旧结果。

refined/run_*/cases/case_000001/ 应保持和 prior/cases/case_000001/ 类似的核心文件名，这样后续 tracker / state-policy 可以用同一个 loader 读 prior 或 refined。

5. Case File Format

每个 case 最少包含四个核心文件：

manifest.json
motion.npz
object.pkl
contact.npy

scene_context.json、task_plan.json、task_validation.json 是推荐文件。早期 smoke test 可以缺省，但正式数据生成必须补齐。

SOMA MVP 还必须包含：

tracker_motion.motion

这是 ProtoMotions soma23 / soma-bones tracker 直接消费的 motion file。不要把 22-joint motion.npz 直接伪装成 SOMA 或 ProtoMotions motion。

5.1 `manifest.json`

manifest.json 记录不可从数组本身可靠推断的约定。

示例：

{
  "case_id": "case_000001",
  "source": "kimodo",
  "fps": 30.0,
  "dt": 0.0333333333,
  "num_frames": 180,
  "motion_file": "motion.npz",
  "source_skeleton": "soma77",
  "source_num_joints": 77,
  "source_convention": "kimodo_soma77_npz_v1",
  "default_tracker_backend": "soma23",
  "coord_system": "z_up_world",
  "root_frame": "world",
  "active_object_id": "chair_001",
  "contact_names": ["left_hand", "right_hand", "pelvis", "left_foot", "right_foot"],
  "notes": ""
}

必须字段：

case_id
fps
dt
num_frames
motion_file
source_skeleton
source_num_joints
source_convention
coord_system
active_object_id
contact_names

fps 只能来自真实数据或显式 resampling，不能靠文件名推断。

5.2 `motion.npz`

motion.npz 存人体 reference。它不承诺一定是 SMPL-X、SOMA 或 G1；真实 skeleton 由 manifest.json["source_skeleton"] 和 manifest.json["source_convention"] 说明。

第一版建议同时保存 source motion 的原始 cached fields 和 refiner 常用的 world-space cache，避免 PPO 训练时反复跑 body model 或 skeleton FK。

推荐字段：

root_positions         float32 [T, 3]
local_rot_mats         float32 [T, J, 3, 3]
global_rot_mats        float32 [T, J, 3, 3] optional
joints_w               float32 [T, J, 3]

source_trans           float32 [T, 3] optional
source_root_orient     float32 [T, 3] or [T, 4] optional
source_body_pose       float32 [T, ...] optional
left_hand_pose         float32 [T, ...] optional
right_hand_pose        float32 [T, ...] optional
hand_keypoints_w       float32 [T, H, 3] optional

body_pos_w             float32 [T, B, 3]
body_quat_w            float32 [T, B, 4] optional
body_lin_vel_w         float32 [T, B, 3] optional
body_ang_vel_w         float32 [T, B, 3] optional

第一版 loader 最少需要：

root_positions
local_rot_mats
joints_w

如果 global_rot_mats、body_lin_vel_w 或 body_ang_vel_w 缺失，data.py 可以用 FK / finite difference 计算。长期推荐在 convert.py 阶段预计算并写入文件。

5.3 `object.pkl`

object.pkl 存 active object 的 trajectory 和物理 proxy metadata。第一版只要求 single active object；多物体场景先写进 scene_context.json，refiner 只对 active object 建 reward。

推荐内容是一个普通 Python dict：

{
    "object_id": "chair_001",
    "category": "chair",
    "mesh_path": "assets/...",
    "proxy_type": "box",
    "scale": [1.0, 1.0, 1.0],
    "mass": 8.0,

    "pos_w": np.ndarray,       # float32 [T, 3]
    "quat_w": np.ndarray,      # float32 [T, 4]
    "lin_vel_w": np.ndarray,   # float32 [T, 3], optional
    "ang_vel_w": np.ndarray,   # float32 [T, 3], optional
}

第一版 loader 最少需要：

object_id
pos_w
quat_w

如果速度缺失，data.py 或 convert.py 计算。

5.4 `contact.npy`

contact.npy 存 contact hint。

5.5 Scene and Task JSON

scene_context.json 保存当前 case 需要的 local scene proxy，不保存完整 SAGE 数据库。

推荐字段：

{
  "scene_id": "layout_84b703fb",
  "active_object_id": "chair_001",
  "static_objects": [],
  "height_map": null,
  "local_origin_w": [0.0, 0.0, 0.0]
}

task_plan.json 保存 VM / SAGE task planner 输出。

task_validation.json 保存 motion generation 阶段的 kinematic validation 结果，例如是否到达目标、是否穿模、是否满足 object affordance。

6. Data Loading Semantics

src/refiner/data.py 应实现 SUGAR-like loader，但使用 PhysHSI 文件名。

训练开始时：

1. 读取 data/refiner/prior/train.txt
2. 对每个 case id 找到 prior/cases/<case_id>/
3. 读取 manifest.json, motion.npz, tracker_motion.motion, object.pkl, contact.npy
4. 检查长度、fps、坐标系、source convention、tracker backend
5. 必要时 resample 到当前训练配置的 target_fps
6. pad / pack 成 tensor buffer

训练循环里不要每 step 从磁盘读文件。第一版可以在启动时全部读入内存：

human buffer:   [N, max_T, ...]
object buffer:  [N, max_T, ...]
contact buffer: [N, max_T, C]
lengths:        [N]
metadata:       list[dict]

其中 N 是 case 数量。

每个 env reset 时：

case_index = sample from [0, N)
frame_index = sample from valid frames of this case
reference = buffers[case_index, frame_index]

这点很重要：所有 cases 一起训练一个 shared refiner policy。case_000001/ 只是数据目录，不对应单独模型。

6.1 FPS Policy

PhysHSI 不沿用 SUGAR 的 *_50hz 文件名。

训练时有两种合法方式：

convert.py 统一把所有 case resample 到一个 target_fps，并写入新的 manifest.json。
data.py 启动时检测混合 fps，并调用 resample.py 对齐到本次训练的 target_fps。

第一版建议方式 1，因为更容易 debug。

target_fps 应由当前 simulator / ProtoMotions tracker 配置决定，不默认等于 Kimodo 30 Hz，也不默认等于 SUGAR 50 Hz。

7. Refiner Code Layout

推荐代码目录：

src/
  refiner/
    data.py
    convert.py
    resample.py
    tracker.py
    policy.py
    reward.py
    env.py
    train.py
    rollout.py

不需要在 src/refiner/ 里放 README。若以后需要约束 coding agent，可以放很短的 src/refiner/AGENTS.md，但正式设计文档放在 docs/。

7.1 `convert.py`

职责：

把 Kimodo / K-model / SAGE 输出整理成 data/refiner/prior/cases/<case_id>/。
写 manifest.json。
写 canonical motion.npz、ProtoMotions tracker_motion.motion、object.pkl、contact.npy。
生成或更新 train.txt、val.txt、test.txt。

它不训练模型，不 import SUGAR / ResMimic。

7.2 `data.py`

职责：

读取 split 文件。
读取 case 文件。
做 shape / length / fps / convention 检查。
把多条 case pack 成 SUGAR-like motion pool。
提供按 case_index、frame_index 取 reference 的函数。

第一版不需要复杂 dataset class hierarchy。一个清楚的 loader 函数和一个轻量对象即可。

7.3 `resample.py`

职责：

对 human / object / contact 做 fps 对齐。
对 position 用 interpolation。
对 rotation 用 slerp 或项目已有 rotation utility。
对 contact 用 nearest 或 thresholded interpolation。

所有 fps 变化都必须更新 manifest.json 的 fps、dt、num_frames。

7.4 `tracker.py`

职责：

这里是 src/refiner/ 中唯一允许直接接 ProtoMotions 的地方。
加载 pretrained ProtoMotions human tracker checkpoint。SOMA MVP 使用 soma23 backend 和 soma-bones tracker。
给 refiner env 提供 base action 或 base rollout。

ResMimic-style residual refiner 的动作组合在这里或 env.py 中完成：

action = base_action + alpha * residual_action

alpha 应从小值开始，或者 residual actor 最后一层 near-zero 初始化，避免训练初期破坏 base tracker。

7.5 `policy.py`

职责：

定义 residual policy network。
输入是当前 simulated state、reference state、object state、contact hint、scene proxy 等。
输出是 residual action。

不要把 SUGAR / ResMimic 的大段 trainer 直接搬进来。我们只复用 residual learning 思想。

7.6 `reward.py`

职责：

实现 SUGAR-inspired reward，但使用 PhysHSI / motion-neutral 命名。

第一版 reward 组成：

R = w_track * R_track
  + w_object * R_object
  + w_contact * R_contact
  + w_scene * R_scene
  + w_task * R_task
  + w_reg * R_regularization

其中：

R_track：human body / joint / root tracking。
R_object：active object pose / velocity tracking。
R_contact：contact hint 和实际 human-object contact 的一致性。
R_scene：地面支撑、scene penetration、object penetration。
R_task：任务终态是否达成，例如 sit / push / pick / place。
R_regularization：action magnitude、residual magnitude、jerk、fall penalty。

不同 interaction 不应该复制一套完全不同的 reward 文件。推荐 shared reward + 少量 task config：

sit:
  contact_names: ["pelvis"]
  task_success: pelvis near seat and object stable

push:
  contact_names: ["left_hand", "right_hand"]
  task_success: object moved to target

pick:
  contact_names: ["left_hand", "right_hand"]
  task_success: object lifted and controlled

7.7 `env.py`

职责：

连接 simulator、motion pool、base tracker、residual policy action 和 reward。
reset 时从 data.py 的 motion pool 采样 case 和 frame。
step 时推进 physics，计算 reward，处理 termination。

训练语义：

reset:
  sample case_index
  sample frame_index
  initialize human and object state from prior or state pool

step:
  base_action = tracker(...)
  residual_action = policy(...)
  action = base_action + alpha * residual_action
  simulator.step(action)
  reward = reward(...)

可以借鉴 SUGAR progressive state pool，但第一版先把 motion reference initialization 跑通。state pool 是第二阶段优化项。

7.8 `train.py`

职责：

读取配置。
调用 data.py 加载 prior/train.txt。
创建 parallel env。
训练一个 shared residual refiner policy。
保存 checkpoint、训练 config、日志。

命令形态建议：

python -m src.refiner.legacy.train \
  --data-root data/refiner/prior \
  --split train \
  --target-fps 30 \
  --tracker-ckpt path/to/protomotions_tracker.pt \
  --out output/refiner/run_YYYYMMDD_HHMMSS

7.9 `rollout.py`

职责：

加载训练好的 refiner checkpoint。
对 prior/cases/ 中的 case 批量 rollout。
写入 data/refiner/refined/run_*/cases/<case_id>/。

命令形态建议：

python -m src.refiner.legacy.rollout \
  --data-root data/refiner/prior \
  --split train \
  --ckpt output/refiner/run_YYYYMMDD_HHMMSS/checkpoints/best.pt \
  --out data/refiner/refined/run_YYYYMMDD_HHMMSS

rollout 输出必须包含：

refined motion.npz
refined object.pkl
refined or copied contact.npy
rollout.npz
metrics.json

8. Training Pipeline

完整 refiner 阶段流程：

SAGE / VM task
  -> Kimodo / K-model generates HSI prior
  -> MVP: regenerate / export Kimodo-SOMA prior for the selected cases
  -> src/refiner/convert.py writes prior cases and soma23 tracker motions
  -> src/refiner/train.py trains one shared residual refiner over all train cases
  -> src/refiner/rollout.py exports refined cases
  -> tracker / teacher training consumes refined cases
  -> state-based policy distillation consumes tracker rollouts

训练时的关键点：

不是每个 case 一个 refiner。
不是把 SUGAR 当 runtime package。
不是把 ResMimic 当 runtime package。
是用 ProtoMotions base tracker 提供 pretrained motion ability，再训练 residual policy 修 object / contact / scene。

9. First MVP Milestone

当前第一优先级不是做完整 benchmark，而是先打通：

12 historical sit case seeds
  -> regenerate Kimodo-SOMA motions
  -> convert SOMA77 motion.npz to ProtoMotions soma23 tracker_motion.motion
  -> refiner prior cases
  -> data.py motion pool
  -> ProtoMotions / Isaac Lab soma23 base tracker
  -> residual refiner train / rollout
  -> refined cases

MVP 的 backend 固定为 SOMA -> ProtoMotions soma23。不要把现有 22-joint cached motion.npz 直接当成 SOMA 输入。

9.1 MVP Case Seeds

下面 12 条是第一批 MVP case seeds。它们的 motion.npz 已经确认是 22-joint cached body motion，shape 类似 [T, 22, 3, 3]。这些文件只用于定位 task、layout、quality metrics 和 segment metadata；canonical MVP motion 必须重新用 Kimodo-SOMA 生成。

01 output/auto_experiments/kimodo_vlm_pipeline_20260607/kimodo/E52_generic_heading_policy_q4/layout_2a8fbb56/motion.npz
02 output/auto_experiments/kimodo_vlm_pipeline_20260607/kimodo/E55_generic_path_heading_blend_q4/layout_2a8fbb56/motion.npz
03 output/auto_experiments/kimodo_vlm_pipeline_20260607/kimodo/E48_generic_seataware_prompt_duration_q4/layout_2a8fbb56/motion.npz
04 output/auto_experiments/kimodo_vlm_pipeline_20260609/kimodo/E67_generic_sit_tail_heading_align/layout_2a8fbb56/motion.npz
05 output/auto_experiments/kimodo_vlm_pipeline_20260607/kimodo/E52_generic_heading_policy_q4/layout_4f321f72/motion.npz
06 output/auto_experiments/kimodo_vlm_pipeline_20260607/kimodo/E55_generic_path_heading_blend_q4/layout_4f321f72/motion.npz
07 output/auto_experiments/kimodo_vlm_pipeline_20260607/kimodo/E48_generic_seataware_prompt_duration_q4/layout_4f321f72/motion.npz
08 output/auto_experiments/kimodo_vlm_pipeline_20260609/kimodo/E74_true_vlm10_full_after_dynamic_object_fix/layout_4f321f72/motion.npz
09 output/auto_experiments/kimodo_vlm_pipeline_20260609/kimodo/E74_true_vlm10_full_after_dynamic_object_fix/layout_2a8fbb56/motion.npz
10 output/auto_experiments/kimodo_vlm_pipeline_20260608/kimodo/E57_current_taskgen_E56_q4/layout_2a8fbb56/motion.npz
11 output/auto_experiments/kimodo_vlm_pipeline_20260609/kimodo/E78_true_vlm10_current_code_after_getup_prompt/layout_2a8fbb56/motion.npz
12 output/auto_experiments/kimodo_vlm_pipeline_20260609/kimodo/E78_true_vlm10_current_code_after_getup_prompt/layout_4f321f72/motion.npz

For each seed, preserve pointers to:

seed motion.npz, only for debugging and metadata.
pipeline_manifest.json.
motion_quality_metrics.json.
original task_plan.json.
original layout json.

Do not silently fall back to the seed motion.npz if SOMA regeneration fails. The case should fail fast.

9.2 Known Working References

Kimodo-SOMA generation has already been exercised locally:

output/protomotions_sage_drop_pen_keep_sit/kimodo_soma_motion.npz

The concrete pipeline entry is:

python -m tools.kimodo.legacy.soma_keyframe_pipeline \
  --config <case_soma_pipeline_config.yaml> \
  --stages control generate

The pipeline chooses the SOMA path when pipeline_family / skeleton is soma, or when the config uses model: Kimodo-SOMA-RP-v1.1. It calls:

tools.kimodo.legacy.soma_keyframe_control
tools.kimodo.generate_with_first_heading --model Kimodo-SOMA-RP-v1.1

Useful config templates:

tools/kimodo/legacy/configs/soma_keyframe_pipeline_drop_pen_keep_sit.yaml
tools/kimodo/legacy/configs/soma_keyframe_pipeline_local_constraints.yaml

ProtoMotions SOMA conversion has also been exercised locally:

output/protomotions_sage_drop_pen_keep_sit/mingxuan_soma_proto/mingxuan_sit_fullbody.motion
output/protomotions_sage_drop_pen_keep_sit/mingxuan_soma_proto/motions.yaml

The concrete ProtoMotions converter is in the submodule:

submodules/ProtoMotions/data/scripts/convert_soma23_npz_to_proto.py
submodules/ProtoMotions/data/scripts/convert_soma23_to_proto.py

Run it from the ProtoMotions root with the ProtoMotions environment:

cd /mnt/data/PhysHSI/submodules/ProtoMotions
/mnt/data/PhysHSI/env_protomotions_isaaclab/bin/python \
  data/scripts/convert_soma23_npz_to_proto.py \
  --input-dir <dir_with_soma77_npz> \
  --output-dir <proto_output_dir> \
  --input-fps 30 \
  --output-fps 30 \
  --yaml-output-name motions.yaml \
  --coordinate-frame sage_local

This script imports ProtoMotions internals and subselects SOMA77 to soma23 using SOMASKEL77_TO_MJCF_INDICES. A coding agent can either call this CLI as a subprocess or import create_motion_from_soma23_data / SOMASKEL77_TO_MJCF_INDICES from the submodule after adding submodules/ProtoMotions/data/scripts to sys.path.

That conversion log shows the expected shape transition:

local_rot_mats: [T, 77, 3, 3]
root_pos:       [T, 3]
dof_pos:        [T, 66]
rigid_body_pos: [T, 23, 3]

The tracker used by previous experiments is:

tracker_type: protomotions_isaaclab
robot: soma23
checkpoint: /mnt/data/PhysHSI/submodules/ProtoMotions/data/pretrained_models/motion_tracker/soma-bones/last.ckpt
robot_asset_usd: /mnt/data/PhysHSI/submodules/ProtoMotions/protomotions/data/assets/usd/soma23_humanoid_flat/soma23_humanoid_flat.usda

The Liu workspace also contains a useful reference:

/mnt/data/PhysHSI_liu/motion_data/protomotions_soma/soma_0519/offline_trace_sage_real_mesh/

9.3 MVP Case Output

convert.py should create:

data/refiner/prior/
  train.txt
  tracker_motions.yaml
  cases/
    case_sit_mvp_0001/
      manifest.json
      motion.npz
      tracker_motion.motion
      object.pkl
      contact.npy
      scene_context.json
      task_plan.json
      task_validation.json
    ...
    case_sit_mvp_0012/
      manifest.json
      motion.npz
      tracker_motion.motion
      object.pkl
      contact.npy
      scene_context.json
      task_plan.json
      task_validation.json

motion.npz is canonical Kimodo-SOMA source motion. It should have local_rot_mats with 77 SOMA joints.

tracker_motion.motion is the ProtoMotions soma23 file converted from motion.npz.

tracker_motions.yaml should list all 12 tracker motions in ProtoMotions-compatible format:

motions:
- file: cases/case_sit_mvp_0001/tracker_motion.motion
  fps: 30
- file: cases/case_sit_mvp_0002/tracker_motion.motion
  fps: 30

train.txt must contain exactly 12 lines:

case_sit_mvp_0001
case_sit_mvp_0002
...
case_sit_mvp_0012

9.4 MVP Conversion Contract

The conversion path is:

seed case metadata
  -> Kimodo-SOMA-RP-v1.1 generation
  -> case motion.npz with source_skeleton=soma77
  -> tracker_motion.motion with tracker_backend=soma23

motion.npz MVP fields:

root_positions         float32 [T, 3]
local_rot_mats         float32 [T, 77, 3, 3]
global_rot_mats        float32 [T, 77, 3, 3] optional
joints_w               float32 [T, 77, 3] optional
smooth_root_pos        float32 [T, 3] optional
global_root_heading    float32 [T] or [T, 2] optional
foot_contacts          float32/bool [T, ...] optional

If joints_w is not produced directly by Kimodo-SOMA, compute and cache it in convert.py if the local Kimodo utilities make that straightforward. If not, record it in manifest.json["missing_source_keys"] and rely on tracker_motion.motion for simulator-side body states.

manifest.json MVP must contain:

{
  "case_id": "case_sit_mvp_0001",
  "source": "kimodo_soma_mvp_sit",
  "seed_motion": "output/.../motion.npz",
  "seed_motion_convention": "kimodo_cached_body22_debug_only",
  "source_motion": "motion.npz",
  "tracker_motion": "tracker_motion.motion",
  "source_pipeline_manifest": "output/.../pipeline_manifest.json",
  "source_motion_quality_metrics": "output/.../motion_quality_metrics.json",
  "source_task_plan": "output/.../task_plan.json",
  "source_layout_json": "data/.../layout_*.json",
  "kimodo_model": "Kimodo-SOMA-RP-v1.1",
  "fps": 30.0,
  "dt": 0.0333333333,
  "num_frames": 264,
  "source_skeleton": "soma77",
  "source_num_joints": 77,
  "source_convention": "kimodo_soma77_npz_v1",
  "default_tracker_backend": "soma23",
  "tracker_checkpoint": "/mnt/data/PhysHSI/submodules/ProtoMotions/data/pretrained_models/motion_tracker/soma-bones/last.ckpt",
  "coord_system": "kimodo_soma_world",
  "active_object_id": "seat",
  "contact_names": ["pelvis_seat"]
}

object.pkl MVP protocol:

active object is a static seat.
Use a plain dict, not a custom class.
Required keys: object_id, category, is_static, pos_w, quat_w, proxy_type.
pos_w shape is [T, 3]; for a static seat, repeat one pose.
quat_w shape is [T, 4]; for a static seat, repeat layout orientation or identity.
If exact seat pose cannot be recovered from layout, estimate from the sit segment endpoint and set pose_source = "estimated_from_sit_segment".

contact.npy MVP protocol:

Shape is [T, 1].
dtype is float32 or bool.
manifest["contact_names"] = ["pelvis_seat"].
First rule: find the sit segment in pipeline_manifest.json; set the last 60% of the sit segment to 1 and all other frames to 0.
If no sit segment exists, fail conversion for that case.

task_plan.json MVP:

Copy source task_plan.json when available.
If source manifest can only expose segments, write a minimal task plan stub and record task_plan_source = "segments_stub" in manifest.json.

scene_context.json MVP:

Store active seat proxy and source layout path.
Do not block MVP on full SAGE scene reconstruction.

9.5 MVP Implementation Checklist

This checklist is for a handoff coding agent. The first goal is a working SOMA MVP, not the full research system.

9.5.1 Create Package Skeleton

Create:

src/
  __init__.py
  refiner/
    __init__.py
    convert.py
    data.py
    resample.py
    tracker.py
    policy.py
    reward.py
    env.py
    train.py
    rollout.py

Do not create extra nested project directories such as PhysicsHSI/PhysicsHSI, source/, or refinement/. Keep helpers inside these files until duplication is real.

9.5.2 Implement `convert.py`

Command:

python -m src.refiner.legacy.convert_prior \
  --preset sit_mvp_12 \
  --backend soma23 \
  --generate-soma \
  --out data/refiner/prior \
  --overwrite

Required behavior:

Read or embed the 12 seed paths in 9.1.
Create case_sit_mvp_0001 to case_sit_mvp_0012.
Regenerate each case using tools.kimodo.legacy.soma_keyframe_pipeline with Kimodo-SOMA-RP-v1.1; output canonical motion.npz.
Verify canonical motion.npz has local_rot_mats.shape[1] == 77.
Convert each canonical motion.npz to tracker_motion.motion with submodules/ProtoMotions/data/scripts/convert_soma23_npz_to_proto.py.
Write tracker_motions.yaml.
Write object.pkl, contact.npy, manifest.json, scene_context.json, task_plan.json, task_validation.json.
Write train.txt with exactly 12 case ids.

Fail fast if Kimodo-SOMA generation fails, if the output is still 22-joint, if the sit segment is missing, or if array lengths disagree.

9.5.3 Implement `data.py`

Expose:

load_cases(data_root, split)

Return a lightweight object or dict with:

case_ids
lengths
fps
motion
object
contact
mask
tracker_motion_files
manifests

Required behavior:

Read data/refiner/prior/train.txt.
Load all 12 cases into memory at startup.
Check manifest["num_frames"], motion.npz, object.pkl, and contact.npy have the same T.
Check manifest["source_skeleton"] == "soma77" and manifest["default_tracker_backend"] == "soma23" for the MVP.
Pad into [N, max_T, ...] buffers with N == 12.
Provide a helper to fetch reference by case_index and frame_index.
Never read from disk inside the training step.

9.5.4 Implement Offline Smoke Validation

Before simulator work:

python -m src.refiner.legacy.convert_prior --preset sit_mvp_12 --backend soma23 --generate-soma --out data/refiner/prior --overwrite
python -m src.refiner.legacy.data --data-root data/refiner/prior --split train --check

The check must print or write:

num_cases = 12
case_ids = case_sit_mvp_0001 ... case_sit_mvp_0012
max_T
fps list
source_skeleton list
tracker_backend list
contact positive frames per case
tracker_motion.motion exists per case

If this fails, do not start PPO.

9.5.5 Implement `tracker.py`

tracker.py is the only src/refiner/ file that directly touches ProtoMotions / Isaac Lab.

Reuse or wrap:

utils/tracking_adapters/protomotions/run_isaaclab_tracking.py

Default MVP tracker config:

simulator: isaaclab
robot: soma23
checkpoint: /mnt/data/PhysHSI/submodules/ProtoMotions/data/pretrained_models/motion_tracker/soma-bones/last.ckpt
motion_file: data/refiner/prior/cases/<case_id>/tracker_motion.motion

Command:

python -m src.refiner.legacy.tracker_case \
  --data-root data/refiner/prior \
  --case-id case_sit_mvp_0001 \
  --simulator isaaclab \
  --out output/refiner/debug_tracker_case_sit_mvp_0001

Required behavior:

Load tracker_motion.motion for the selected case.
Run a ProtoMotions / Isaac Lab soma23 base rollout.
Write rollout trace and metrics.
Record checkpoint, robot asset, motion file, and tracker type in metrics.

9.5.6 Implement `policy.py`

First version is only a residual head:

Input current simulated state plus reference state.
Output residual action.
Near-zero initialize the final layer, or merge with small alpha.
Keep a residual_scale=0 path for comparing base tracker versus residual refiner.

Do not copy ResMimic trainer code wholesale.

9.5.7 Implement `reward.py`

First version supports sit MVP:

R = w_track * R_track
  + w_contact * R_contact
  + w_scene * R_scene
  + w_reg * R_regularization

Required terms:

root / body tracking.
pelvis-seat contact consistency.
fall penalty.
seat penetration penalty using a simple box or height proxy.
residual magnitude penalty.

Do not create one reward file per interaction. Later push / pick / place should use config.

9.5.8 Implement `env.py`

Reset:

sample case_index from 12 cases
sample frame_index from valid frames
initialize human and object state

Step:

base_action = tracker(...)
residual_action = policy(...)
action = base_action + alpha * residual_action
simulator.step(action)
reward = reward(...)

State pool can wait. It is not required for the first MVP.

9.5.9 Implement `train.py`

Command:

python -m src.refiner.legacy.train \
  --data-root data/refiner/prior \
  --split train \
  --target-fps 30 \
  --simulator isaaclab \
  --tracker-backend soma23 \
  --out output/refiner/sit_mvp_12_debug

Required behavior:

Load all 12 cases through data.py.
Create shared env / vector env.
Train one shared residual policy, not one policy per case.
Save config.json, checkpoint, and training log.
Log the 12 case ids and the soma23 checkpoint path.

Short debug training is acceptable, but it must execute reset, step, reward, and optimizer.

9.5.10 Implement `rollout.py`

Command:

python -m src.refiner.legacy.rollout \
  --data-root data/refiner/prior \
  --split train \
  --ckpt output/refiner/sit_mvp_12_debug/checkpoints/latest.pt \
  --out data/refiner/refined/run_sit_mvp_12_debug

Required behavior:

Roll out all 12 case_sit_mvp_* cases.
For each refined case, write manifest.json, motion.npz, object.pkl, contact.npy, rollout.npz, and metrics.json.
Write metrics_summary.json.
Preserve original case_id, seed metadata, SOMA source metadata, and tracker metadata.

9.5.11 Minimum Done Definition

The handoff agent should be able to run:

python -m src.refiner.legacy.convert_prior --preset sit_mvp_12 --backend soma23 --generate-soma --out data/refiner/prior --overwrite
python -m src.refiner.legacy.data --data-root data/refiner/prior --split train --check
python -m src.refiner.legacy.tracker_case --data-root data/refiner/prior --case-id case_sit_mvp_0001 --simulator isaaclab --out output/refiner/debug_tracker_case_sit_mvp_0001
python -m src.refiner.legacy.train --data-root data/refiner/prior --split train --target-fps 30 --simulator isaaclab --tracker-backend soma23 --out output/refiner/sit_mvp_12_debug
python -m src.refiner.legacy.rollout --data-root data/refiner/prior --split train --ckpt output/refiner/sit_mvp_12_debug/checkpoints/latest.pt --out data/refiner/refined/run_sit_mvp_12_debug

After these commands run, tune reward weights, scene proxies, state pool, and benchmark settings.

9.6 MVP Acceptance Criteria

MVP is complete only if:

data/refiner/prior/train.txt has exactly 12 lines.
data/refiner/prior/cases/ has case_sit_mvp_0001 to case_sit_mvp_0012.
Each case has manifest.json, motion.npz, tracker_motion.motion, object.pkl, contact.npy, scene_context.json, and task_plan.json.
Each manifest.json has source_skeleton == "soma77" and default_tracker_backend == "soma23".
Each motion.npz has local_rot_mats.shape[1] == 77.
Each contact.npy has shape [T, 1] and at least one positive frame.
tracker_motions.yaml lists all 12 .motion files.
data.py loads 12 cases and packs them into [12, max_T, ...] buffers.
tracker.py runs at least one case with ProtoMotions / Isaac Lab soma23 tracker.
train.py starts shared refiner training over 12 cases.
rollout.py writes refined output for all 12 cases.

Not complete:

only generating 2 cases.
using the 22-joint seed motion.npz as canonical motion.npz.
missing tracker_motion.motion.
contact.npy all zeros.
pure offline NumPy loader with no ProtoMotions / Isaac Lab path.
one policy per case.
runtime import of SUGAR or ResMimic.

9.7 Deferred Formal Minimal Experiment

正式的最小实验是：从 10 个 refined SAGE benchmark scenes 里强制生成 10 个独立 seat-only / walk+seat tasks，每个 scene 一个。

这个实验优先级暂时降低。它不应该阻塞当前 MVP，因为当前目标是先把 generation -> refiner pipeline 打通。等 12 条 MVP case 能完成 conversion、training、rollout 后，再做 10-scene independent seat-only benchmark。

10. Validation Checklist

每次生成新的 prior 或 refined dataset，至少检查：

manifest.json 的 fps * dt 一致。
manifest.json["num_frames"] 和所有数组长度一致。
motion.npz、object.pkl、contact.npy 的 T 一致。
SOMA MVP 的 tracker_motion.motion 存在，并且 tracker_motions.yaml 能列到它。
坐标系、source convention 和 tracker backend 没有混用。
contact channel 数量等于 contact_names。
loader 不在 training step 中反复读磁盘。
training log 明确记录使用了哪些 case。
rollout 输出没有覆盖旧 run。
refined trajectory 仍能追溯到原始 case_id、SAGE scene 和 task plan。

11. Naming Rules

使用：

src/refiner/
motion.npz
tracker_motion.motion
object.pkl
contact.npy
prior/
refined/
case_000001

避免：

robot_50hz.npz
obj_motion_global_50hz.pkl
PhysicsHSI/PhysicsHSI
source/
refinement/
RefinerInput
BatchTrajectory

这些复杂类型和重复目录会让代码难读，第一版没有必要。

PhysHSI Refiner Development README

1. 核心原则

2. Runtime Environment

3. 和 SUGAR 的对应关系

4. Dataset Layout

4.1 prior/

4.2 refined/

5. Case File Format

5.1 manifest.json

5.2 motion.npz

5.3 object.pkl

5.4 contact.npy

5.5 Scene and Task JSON

6. Data Loading Semantics

6.1 FPS Policy

7. Refiner Code Layout

7.1 convert.py

7.2 data.py

7.3 resample.py

7.4 tracker.py

7.5 policy.py

7.6 reward.py

7.7 env.py

7.8 train.py

7.9 rollout.py

8. Training Pipeline

9. First MVP Milestone

9.1 MVP Case Seeds

9.2 Known Working References

9.3 MVP Case Output

9.4 MVP Conversion Contract

9.5 MVP Implementation Checklist

9.5.1 Create Package Skeleton

9.5.2 Implement convert.py

9.5.3 Implement data.py

9.5.4 Implement Offline Smoke Validation

9.5.5 Implement tracker.py

9.5.6 Implement policy.py

9.5.7 Implement reward.py

9.5.8 Implement env.py

9.5.9 Implement train.py

9.5.10 Implement rollout.py

9.5.11 Minimum Done Definition

9.6 MVP Acceptance Criteria

9.7 Deferred Formal Minimal Experiment

10. Validation Checklist

11. Naming Rules

4.1 `prior/`

4.2 `refined/`

5.1 `manifest.json`

5.2 `motion.npz`

5.3 `object.pkl`

5.4 `contact.npy`

7.1 `convert.py`

7.2 `data.py`

7.3 `resample.py`

7.4 `tracker.py`

7.5 `policy.py`

7.6 `reward.py`

7.7 `env.py`

7.8 `train.py`

7.9 `rollout.py`

9.5.2 Implement `convert.py`

9.5.3 Implement `data.py`

9.5.5 Implement `tracker.py`

9.5.6 Implement `policy.py`

9.5.7 Implement `reward.py`

9.5.8 Implement `env.py`

9.5.9 Implement `train.py`

9.5.10 Implement `rollout.py`