跳转至

Kimodo HOI Pipeline Spec

Superseded / legacy

本页描述的 specialized runner 已迁入 tools/kimodo/legacy/，大量中间文件也不再是 current pipeline contract。当前运行方式以 Project Pipeline 为准；本页只用于历史追溯。

Historical status: this was the working contract on 2026-07-01 and is now superseded.

This file was the canonical description of the specialized Kimodo HOI pipeline for pick/carry/place tasks. It is frozen as decision history; do not update its stage order, artifact contracts, or default quality rules.

Scope

The specialized HOI entrypoint is:

tools/kimodo/run_kimodo_hoi_pipeline.py

It is used for task plans whose main interaction is pick -> carry -> place. The broader batch dispatcher may call it from:

utils/kimodo_pipeline/run_kimodo_motion_batch.py

The pipeline is adapter-only: it should not modify Kimodo, HOI-FHLI, or their submodule source trees. It builds scene/task constraints, runs Kimodo body generation, attaches the task object, bakes/refines hands, audits quality, and renders the final full-scene motion.

Runtime Contract

Use one Python for the main Kimodo/ProtoMotions pipeline and a separate Python for official HOI-FHLI grasp candidate stages when needed:

# Main pipeline in this workspace.
env_protomotions_isaaclab/bin/python

# Official HOI-FHLI grasp probe/apply route.
/home/user/anaconda3/envs/isaac-sim/bin/python

--official-grasp-python can override the official-grasp interpreter. If it is not passed, the pipeline uses /home/user/anaconda3/envs/isaac-sim/bin/python when that executable exists, otherwise the current Python.

Stage Order

The expected mainline stage order follows main() in run_kimodo_hoi_pipeline.py:

  1. Resolve task geometry.
  2. Read task_plan.json and layout_json.
  3. Resolve pick/carry/place step IDs.

  4. Resolve manipulated object, support object, scene origin, object mesh, object extents, initial object center, target object center, pick root, and carry endpoint in Kimodo coordinates.

  5. Build Kimodo inputs.

  6. Write meta.json, info.json, and baseline constraints.json.

  7. Snapshot baseline constraints as constraints_base.json.

  8. Materialize expected contact labels.

  9. Materialize task-level expected labels into contact.npy and contact_labels.json.

  10. Add body constraints.

  11. Optional carry pose keyframes for two-hand carry.
  12. Single-hand carry defaults to Root2D/text only. Do not insert fullbody carry pose keyframes by default because they constrain legs and can turn walking into repeated pose interpolation.
  13. Place final reach keyframe. Single-hand place defaults to the final fullbody keyframe only; approach keyframes are an explicit ablation because they can perturb gait and release timing.
  14. Pick reach keyframe.

  15. Single-hand tasks use active-wrist anchors; two-hand tasks use wrist midpoint behavior unless overridden.

  16. Optional automatic per-object official grasp.

  17. Enabled by --auto-official-pick-grasp.
  18. Build a one-frame source motion for the current task object.
  19. Run solve_grasp_keyframe.py as a bootstrap candidate generator.
  20. By default, single-hand candidates continue through:
    • probe_official_hoifhli_single_frame_grasp.py
    • apply_official_hoifhli_grasp_to_smplx_keyframe.py
  21. --disable-auto-official-grasp-probe-apply keeps only the bootstrap candidate result.
  22. Score candidates by hand/object distance, contact ratio, energy, and penetration.
  23. Select the best passing keyframe.
  24. Feed the selected keyframe into the official pick-grasp replacement stage.

  25. Switch object binding to active_hand_local unless --keep-auto-official-grasp-binding-mode is set.

  26. Optional official grasp replacement.

  27. Enabled by --official-pick-grasp-keyframe-npz.
  28. Also runs after automatic grasp selection, because the automatic stage writes the selected keyframe into this same argument.
  29. Replaces the generated pick keyframe before Kimodo generation.
  30. Single-hand official pick replacement defaults to --official-pick-grasp-constraint-mode active_hand, so the initial grasp frame is left-hand/right-hand rather than fullbody. This preserves the active grasp target without locking the inactive arm from the pickup reference pose.
  31. Extracts pose_hand as a hand-bake target when --hand-mode is still prior and no explicit hand pose or hand schedule was provided.
  32. With --object-binding-mode active_hand_local, reuses the same keyframe as the object-in-hand binding anchor.
  33. --official-pick-grasp-hold-offset-frames 8,16,24,34,45 repeats the official grasp keyframe shortly after attach by default, preventing the hand/object pair from sinking into the pickup support while preserving the generated grasp relation.

  34. Hold frames use --official-pick-grasp-hold-constraint-mode active_hand by default for single-hand tasks. This emits left-hand/right-hand end-effector constraints instead of fullbody constraints, so the pickup torso bend is not locked into the post-grasp recovery.

  35. Optional place grasp keyframe optimization.

  36. Enabled by --enable-place-grasp-keyframe-optimize.
  37. Requires an official pick grasp keyframe from --official-pick-grasp-keyframe-npz or --auto-official-pick-grasp.
  38. Runs solve_grasp_keyframe.py on bound_place_pose.npz, reusing the official HOI-FHLI object mesh and active-hand contact setup, but with the final layout/support object rotation used by the object binding stage.
  39. Replaces only the final place constraint with place_hoifhli_grasp_keyframe.
  40. Single-hand place grasp replacement defaults to --place-grasp-constraint-mode active_hand, so final placement constrains the active hand while leaving the inactive arm under the generated motion.
  41. Writes place_grasp_hand_pose_schedule.json, which keeps the pick grasp through carry and fades to the optimized place grasp near release.
  42. By default, aligns the object release end to the optimized place frame so the object is already at the final position and orientation when the place grasp keyframe is enforced.
  43. The optimized place keyframe is forwarded to object attachment as the active_hand_local release binding anchor. The release window therefore follows the active hand while transitioning from pickup hand-local object transform to placement hand-local object transform, instead of making the object rotate to the support target by itself in world space.

  44. Disables wrist-anchor correction for this schedule because the schedule entries are pure hand-pose targets, not returned-wrist anchor targets.

  45. Generate Kimodo base motion.

  46. Run tools.kimodo.generate_with_first_heading.
  47. Output motion.npz.

  48. Run root despike unless disabled, preserving motion_raw_kimodo.npz.

  49. Attach the dynamic object.

  50. Output motion_augmented.npz.
  51. Default anti-floating behavior keeps the object static before validated pickup contact.
  52. anchor_path binds the object to a hand-center path.
  53. active_hand_local computes an object-in-active-hand transform and expands it over attach/carry/release. This is the preferred route when an official grasp keyframe is available.

  54. active_hand_local must also enforce the pickup support plane from attach through the end of the pick segment, so a hand closing downward cannot press the object into the table or floor.

  55. Hand stage.

  56. Supported --hand-mode: prior, npz, schedule, optimize.
  57. Optional wrist-anchor correction applies scheduled HOI-FHLI hand anchors to the Kimodo arms.

  58. Bake SMPL-X vertices and hands into motion_augmented_hands.npz.

  59. Optional local grasp refinement.

  60. Output motion_hoi_refined.npz when enabled.
  61. Official pick grasp routes preserve the generated hand/object relation by default when --preserve-official-grasp-relation is active, so this stage is skipped and motion_augmented_hands.npz remains the final motion.
  62. Bounded post-attach object-local correction is enabled by default and can be disabled with --disable-local-grasp-object-correction.
  63. Must not move the object before pickup to pass contact metrics.
  64. Its pickup surface lock is off for frozen official grasp runs because clamping only the object breaks the hand/object local transform.

  65. The selected final motion is recorded as artifacts.final_motion.

  66. Materialize measured contact labels.

    • Run after the final motion exists.
    • Output measured_contact.npy and measured_contact_labels.json.

  67. Run metrics and strict pose audit.

    • Object-motion metrics: motion_object_metrics.json.
    • Hand-object metrics: hand_object_contact_metrics.json.
    • Strict hand/object pose audit: strict_hand_object_audit.json.

  68. Render and aggregate case quality.

    • Render the full scene with opaque scene objects by default.
    • Output the configured video and final-frame image.
    • Run audit_kimodo_hoi_case_quality.py after render, using the final motion, metrics, strict audit, render path, and final frame.

--stop-after boundaries are:

build       after build_inputs + expected_contact_labels
constraints after carry/place/pick constraints and any official grasp replacement
generate    after Kimodo generation + root_despike
object      after dynamic object attachment
hands       after hand stage, local grasp refine, and measured contact labels
metrics     after metrics + strict hand/object audit
render      after render + case_quality_audit

Key Artifacts

Each run should write pipeline_manifest.json incrementally. The final motion path used by metrics and render must be recorded in the manifest under artifacts.final_motion.

Important artifacts:

pipeline_manifest.json
meta.json
info.json
constraints_base.json
constraints.json
contact.npy
contact_labels.json
pick_wrist_target_metadata.json
bound_carry_pose_*.npz
bound_carry_pose_metadata_*.json
bound_place_metadata.json
bound_place_pose.npz
bound_pick_pose.npz
bound_pick_metadata.json
auto_official_pick_grasp_source_motion.npz
auto_official_pick_grasp_source_motion.json
auto_official_pick_grasp/<candidate>/bootstrap/grasp_keyframe_before.npz
auto_official_pick_grasp/<candidate>/bootstrap/grasp_keyframe.npz
auto_official_pick_grasp/<candidate>/official/official_grasp_probe.json
auto_official_pick_grasp/<candidate>/apply/final_official_grasp_smplx_keyframe.npz
auto_official_pick_grasp_selection.json
official_pick_grasp_hand_pose_target.npz
official_pick_grasp_replacement_metadata.json
place_hoifhli_grasp/grasp_keyframe.npz
place_hoifhli_grasp/grasp_keyframe.json
place_hoifhli_grasp_hand_pose_target.npz
place_grasp_hand_pose_schedule.json
place_grasp_keyframe_replacement_metadata.json
motion_raw_kimodo.npz
motion.npz
object_motion_metadata.json
motion_augmented.npz
wrist_anchor_correction_metadata.json
motion_augmented_wrist_corrected.npz
hand_bake_metadata.json
motion_augmented_hands.npz
local_grasp_refine_metadata.json
motion_hoi_refined.npz
measured_contact.npy
measured_contact_labels.json
motion_object_metrics.json
hand_object_contact_metrics.json
strict_hand_object_audit.json
motion_scene_dynamic_object_hands.mp4
motion_scene_dynamic_object_hands_final_frame.png
case_quality_audit.json

Not every artifact appears in every run. For example, auto_official_pick_grasp_selection.json only appears when --auto-official-pick-grasp is enabled. place_hoifhli_grasp/*, place_hoifhli_grasp_hand_pose_target.npz, and place_grasp_hand_pose_schedule.json only appear when --enable-place-grasp-keyframe-optimize is enabled. motion_hoi_refined.npz only appears when local grasp refinement is enabled. motion_augmented_wrist_corrected.npz only appears when a hand pose schedule exists and wrist-anchor correction is enabled. Render names can be changed with --render-output-name and --final-frame-name.

Grasp Quality Contract

Carry is object-agnostic, but good grasp quality is object-specific. Do not reuse a cup grasp anchor for a different object category unless the run is a diagnostic ablation.

Preferred route for realistic grasp:

--auto-official-pick-grasp

This creates per-object grasp candidates and selects a keyframe for the current object. The selected keyframe is then used for both:

  • pick fullbody constraint replacement;
  • active_hand_local object binding.

For single-hand pick/carry/place tasks where final placement quality matters, also enable:

--enable-place-grasp-keyframe-optimize

This uses the final bound place pose as the set position and the final layout/support rotation as the set orientation, then runs the same single-frame HOI-FHLI grasp optimizer against that pose and object mesh. The pickup grasp remains the carry hand target, and the hand fades toward the optimized place grasp during the release window. The same optimized place keyframe becomes the release binding anchor for active_hand_local, so object rotation is explained by the hand-local grasp transition instead of an independent world-space slerp.

After hand baking and optional local grasp refinement, the final output motion frame at the optimized place frame is overwritten with the exact HOI-FHLI place keyframe. This is the stage that makes the final upright object pose and the human/hand pose an exact keyframe, instead of only using that keyframe as a soft constraint. --disable-place-grasp-align-object-release keeps the old configured release end for ablations; --disable-place-grasp-release-binding-anchor keeps the old world-space release interpolation; --disable-place-grasp-apply-final-keyframe keeps the generated/refined final frame instead of forcing the exact HOI-FHLI place keyframe. Production runs should leave all three defaults enabled.

Default candidate settings:

seeds: 17,41,73
bootstrap batch size: 16
bootstrap iterations: 120
official probe batch size: 32
official probe iterations: 600
surface samples: 1200

Default quality gates:

closest50_mean <= 0.012 m
p05 <= 0.030 m
ratio_lt_2cm >= 0.05
penetration energy <= 0.15

If no candidate passes, the default policy is fail. Use --auto-official-grasp-no-valid-policy use_best only for smoke tests or visual diagnostics.

Hand Routing Contract

Hand policy must be consistent across all stages.

Resolution order:

  1. Explicit CLI --object-hand-policy left/right/both.
  2. Task-plan hand metadata on carry, pick, then place steps.
  3. Legacy default both.

When --object-hand-policy left/right is passed, the pipeline writes an effective task plan in the run directory:

task_plan_effective_left_hand.json
task_plan_effective_right_hand.json

All downstream stages must use the effective task plan rather than the original source task plan:

  • Kimodo input builder;
  • expected contact labels;
  • object attachment;
  • hand baking/refinement;
  • measured contact labels;
  • metrics and strict audit.

This avoids the failure mode where a right-hand run still materializes left_hand_object contact labels from a left-hand source task.

Keyframe Policy

The pipeline has separate keyframe responsibilities:

  • Pick: one active-wrist/fullbody reach keyframe, optionally replaced by an official HOI-FHLI grasp keyframe.
  • Carry: Root2D path and text should own walking. Fullbody carry pose keyframes are allowed for two-hand or diagnostic ablations, but are off by default for single-hand carry.
  • Place: the default single-hand route uses only the final place keyframe. Active-hand approach keyframes remain available as explicit ablations, but they are not the default because they can perturb the walking segment and introduce place-stage snaps.
  • Place grasp: when an official pick grasp exists, use --enable-place-grasp-keyframe-optimize to optimize the final place hand pose against the same object mesh and final support/layout object rotation. This changes the release keyframe, hand-pose schedule, and active-hand-local release anchor without adding dense walking constraints.

Do not use a static fullbody carry pose as a walking keyframe. Reusing the same reference frame at multiple carry times copies its leg configuration and can produce foot shuffling, toe-walking, or a bent-leg/straight-leg gait.

Do not place dense approach keyframes directly on top of Root2D samples, carry/place boundaries, or object-release windows. Single-hand approach constraints must be active-hand only unless the run is an explicit fullbody ablation, and they should begin after the place segment has enough room to blend from the carry posture. If an approach-keyframe run shows a large object or wrist step near place, revert to the final-only template before tuning hand pose or release windows.

Do not make --place-use-official-grasp-relation the default. That switch derives the place wrist target directly from the official pick object-in-wrist transform and is useful as an ablation, but on low tabletop targets it can place the desired wrist below the floor-locked fullbody reference and fail the place reference quality gate. The preferred production route is:

final place fullbody keyframe
-> solve_grasp_keyframe.py at bound_place_pose.npz with final object rotation
-> replace final place constraint with active-hand place_hoifhli_grasp_keyframe
-> align object release end to that optimized place keyframe
-> use the optimized place keyframe as the active-hand-local release anchor
-> fade hand pose from pick grasp to place grasp near release

This keeps walking and turning controlled by Root2D/text while letting HOI-FHLI own the grasp at both pickup and final placement.

Turn And Gait Contract

For pick/carry/place tasks, the default turn is a stable in-place heading schedule. It does not consume the carry path. Use nonzero --turn-walk-distance only for explicit ablations, because the walk-while-turn route can destabilize the lower-body gait.

Default behavior:

--turn-s 1.9
--turn-rate-deg-per-sec 54
--turn-min-s 0
--turn-min-heading-deg 20
--turn-walk-distance 0
--turn-walk-max-path-fraction 0
--carry-s 9.0
--carry-speed-mps 0.60
--root2d-stride 5
--single-hand-place-template final_keyframe
--place-approach-fractions 0.48,0.62
--object-release-start-fraction 0.65
--object-release-end-fraction 0.95
--place-grasp-align-object-release
--place-grasp-release-binding-anchor
--hand-release-start-fraction 0.65
--hand-release-end-fraction 0.95

--auto-carry-duration is available for ablations or unusually short paths, but do not enable it by default for object-carry clips unless the target speed and max duration have been checked against foot sliding. On this cup task, shortening the carry segment from 9.0 s to about 5.6 s doubled the root speed and caused visible shuffling.

The builder records the split as:

turn_timing
turn_walk
kimodo_effective_turn_path_xz
kimodo_effective_carry_path_xz
turn_constraint.root_mode

The evaluator records body-relative foot/knee order metrics by segment. The turn segment must pass:

turn_foot_body_lateral_order_margin_m_p05 >= -0.03
turn_knee_body_lateral_order_margin_m_p05 >= -0.02

Negative foot order margins during a turn usually indicate crossed feet or leg-leg penetration, even when the older foot-separation metric still passes.

Single-Hand Carry Placement Contract

For single-hand carry, distinguish two different coordinate contracts:

  • Grasp-local transform: the object-to-hand transform from the official grasp keyframe. This keeps the cup in the fingers.
  • Carry placement: the global object location relative to the walking body. This keeps the carried object natural and away from the face/body.

Carry placement is measured in the root/body frame:

forward axis: current body heading
lateral axis: positive body-left
height: object_center_y - root_y
active lateral: left uses +body-left, right uses -body-left

Default single-hand carry bounds:

object forward from root median: 0.10 m to 0.55 m
object active-side lateral median: -0.03 m to 0.35 m
object height-from-root p95: < 0.38 m

Interpretation:

  • Cups/bottles should be carried in front of the torso, slightly on the active hand side or near centerline.
  • They should not cross substantially to the opposite side during carry.
  • They should not rise toward the face unless the task explicitly says drink, inspect, or show the object.
  • Place approach can lift the object only near the target support and should use the support height plus a small clearance, not a mouth-level carry trajectory.

evaluate_kimodo_task_object_motion.py records these fields in motion_object_metrics.json and adds quality gates for single-hand carry.

Anti-Floating Contract

The object must not move toward the hand before pickup contact is validated, and it must not be pressed below its initial support during pickup.

Required behavior:

  • pre-attach object trajectory is static at the layout pose;
  • no legacy pre-pickup blend unless --allow-pre-object-attach-blend is passed;
  • default --no-valid-object-attach-policy fail refuses fake carry data when the hand never reaches the object;
  • active_hand_local with an official binding anchor may bypass the old wrist-center readiness gate, but only after verifying post-binding anti-floating metrics;
  • pick_object_bottom_delta_from_initial_m_min must stay above -0.02 m.

Relevant metadata fields in object_motion_metadata.json:

resolved_binding_start_frame
pre_attach_static
pre_attach_max_delta_m
pre_attach_vertical_lift_max_m
attach_snap_distance_m
anti_floating_gate_pass
post_binding_anti_floating_gate_pass
binding.mode
binding.local_object_translation_norm_m
binding.pickup_surface_lock

Relevant metadata fields in local_grasp_refine_metadata.json:

object_correction.pickup_surface_lock

Frozen official grasp runs should record the local_grasp_refine stage as disabled with preserve_official_grasp_relation=true.

Full automatic per-object grasp route:

cd /mnt/data/PhysHSI
PYTHONPATH=. env_protomotions_isaaclab/bin/python \
  tools/kimodo/run_kimodo_hoi_pipeline.py \
  --out-dir output/auto_experiments/<run_name> \
  --task-plan <task_plan.json> \
  --layout-json <layout.json> \
  --auto-official-pick-grasp \
  --enable-place-grasp-keyframe-optimize \
  --scene-object-source all \
  --scene-alpha 1.0 \
  --camera-view human-front \
  --camera-distance 1.3 \
  --camera-elevation 8 \
  --fov 78

Fast constraints-only smoke for wiring checks:

cd /mnt/data/PhysHSI
PYTHONPATH=. env_protomotions_isaaclab/bin/python \
  tools/kimodo/run_kimodo_hoi_pipeline.py \
  --out-dir output/auto_experiments/<smoke_run_name> \
  --task-plan <task_plan.json> \
  --layout-json <layout.json> \
  --auto-official-pick-grasp \
  --auto-official-grasp-seeds 17 \
  --auto-official-grasp-bootstrap-batch-size 2 \
  --auto-official-grasp-bootstrap-iterations 1 \
  --auto-official-grasp-batch-size 2 \
  --auto-official-grasp-iterations 1 \
  --auto-official-grasp-object-surface-samples 64 \
  --auto-official-grasp-no-valid-policy use_best \
  --stop-after constraints \
  --skip-render

Manual official grasp route when a vetted keyframe already exists:

cd /mnt/data/PhysHSI
PYTHONPATH=. env_protomotions_isaaclab/bin/python \
  tools/kimodo/run_kimodo_hoi_pipeline.py \
  --out-dir output/auto_experiments/<run_name> \
  --task-plan <task_plan.json> \
  --layout-json <layout.json> \
  --official-pick-grasp-keyframe-npz <final_official_grasp_smplx_keyframe.npz> \
  --object-binding-mode active_hand_local

Maintenance Rules

Update this file when any of the following changes:

  • stage order;
  • default artifact names;
  • default grasp route or quality gates;
  • object attachment or anti-floating policy;
  • hand mode semantics;
  • metric/audit/render final-motion path;
  • required runtime environments.

Avoid documenting one-off experiment paths as default behavior. Put those in change logs or case reports, and only promote stable behavior here after the main pipeline uses it.