跳转至

HSI Generation Refactor Plan

Date: 2026-07-08

This document is the current design and cleanup plan for utils/hsi_generation. It is not a historical migration log. Old run directories, old manifests, and old v13 alignment notes are provenance only; they should not drive runtime state or public interfaces.

Current Goal

HSI generation should be a task-first, general motion generation system:

tools/run_hsi_generation.py --task-plan task_plan.json --layout-json layout.json --out-dir case_dir

The public interface stays small:

  • --task-plan
  • --layout-json
  • --out-dir

Runtime/debug strategy should not become public config shape, and task json should not become a hidden execution config carrying dated paths, old case state, or experiment knobs.

Core Problem

The current simplified path has the right stage outline but has not preserved all v13 motion-generation behavior. The biggest behavioral gap is not the overall flow; it is that some action operations are incomplete or not triggered from task semantics.

For pick/carry/place, the v13-quality chain must be present:

build scaffold inputs
-> build contact labels
-> pick/place/carry keyframe constraints
-> official or optimized pick grasp replacement
-> place grasp keyframe optimize
-> Kimodo generate
-> root cleanup
-> object motion with active-hand binding anchor
-> hand pose schedule bake
-> preserve official/optimized grasp relation or skip destructive refine
-> final place keyframe apply
-> metrics/render

The new implementation should keep this behavior, but not as a hard-coded v13 recipe. It should be triggered by generic action operations derived from the task plan.

Universality Requirement

HSI generation must not be a hand-crafted recipe for the N1 v13 pick-carry-place cup case. That case is a regression target, not the architecture.

The task plan describes semantics:

  • ordered actions
  • manipulated objects
  • targets/supports
  • active hands
  • contact expectations
  • interaction annotations

Kimodo text prompts are scaffold constraints derived from these semantics. They should not be public runtime config, and should not fall back to object-specific sentences such as "box" or "table" unless those labels came from the task/layout.

The pipeline derives an execution plan from those semantics. It should support the SAGE action set as a finite collection of composable operations:

reach
pick
carry
place
push
pull
kick
sit
get_up
lie_down

open and close are not a current target because SAGE generation currently disables them. They should fail clearly or remain unsupported until the generator enables them.

Passing the v13 pick/carry/place comparison is necessary but not sufficient. It proves regression compatibility for one important case; it does not prove generality.

Unified Representation

HSI generation uses one simple chain across locomotion, HOI, and HSI:

Action Plan -> Anchors -> Constraints -> Scaffold Motion -> Refined Motion

The terms are strict:

  • Action Plan: ordered task actions parsed from task json.
  • Anchors: semantic intent records. They are not motion arrays, not .npz files, and not solver-specific inputs.
  • Constraints: concrete inputs derived from anchors for a backend or postprocess step. This is a broad HSI term, not only a Kimodo term.
  • Scaffold Motion: the full-body human motion generated by one Kimodo call.
  • Refined Motion: scaffold motion after object binding/release, hand bake/refine, final placement, and support/contact correction.

Examples of constraints:

  • Kimodo root/path/facing/full-body keyframe constraints.
  • HOI-FHLI grasp or release keyframe optimization inputs.
  • Contact schedules for labels, audit, render overlay, and hand/object refine.
  • Object relation schedules such as object-in-hand and hand-to-support transfer.

Do not call contact schedules "Kimodo contact constraints"; Kimodo does not currently consume contact constraints directly. Contact schedules are constraints only in the broader HSI sense.

Scene-agnostic priors belong under data/hsi_primitives/, ideally as atomic keyframe candidates. Scene-conditioned outputs belong under output/. Current full-motion backend priors are transitional Kimodo/keyframe-builder debt; they must not become the conceptual HSI primitive library.

Target Flow

The target flow is:

task_plan + layout_json
-> build_clips(task_plan)
-> build_execution_plan(clips, geometry)
-> build_action_facing(task_plan, layout_json)
-> build_anchors(execution_plan)
-> build_constraints(anchors, task, geometry)
-> build_scaffold_spec(constraints)
-> kimodo.generate(scaffold_spec) once
-> kimodo.root_cleanup(scaffold_motion)
-> refine_interaction(scaffold_motion, constraints)
-> evaluation.metrics_render()

refine_interaction is a conceptual phase, not necessarily a new wrapper file. It owns post-scaffold interaction correction:

apply object binding/release
-> bake scheduled hand pose
-> optional local hand/object refinement
-> final place/support correction

Object motion belongs inside this phase. It should not appear as a peer of Kimodo scaffold generation in the top-level pipeline.

Action-facing headings are task semantics. They are derived from task source points, interaction targets, and layout targets, then stored only as info["action_facing"]["by_step_id"]. Pick/place keyframe operations read the heading through planning.heading_for_event(info, event). Do not recreate old heading fields such as pick_facing, place_facing, or recommended_first_heading_angle.

refine_interaction produces an interaction motion. This is one joint motion state, not a human-only motion plus a loosely attached side product:

interaction_motion
  human_motion: full-body/root/hand motion
  object_motion: optional per-object motion
  contacts/supports: optional interaction windows

object_motion is optional and action-dependent:

  • Pure locomotion and turn actions have no object motion.
  • Static support interactions such as sit, get_up, lie_down, and stand use scene support geometry/contact constraints, but normally do not produce a rigid object motion.
  • Movable-object interactions such as pick, carry, place, push, pull, and kick should produce per-object rigid 6D object motion when the object is expected to move.
  • Articulated interactions such as open/close, if enabled later, should use an articulated object motion rather than pretending the whole object is a rigid body.

Current native code only routes the pick/carry/place release chain into object motion refinement. Push/pull/kick anchors describe the target contract, but they must not reuse the pick/carry/place attach/release implementation until a contact-motion object solver exists.

Downstream tracking should consume this interaction motion contract:

human_motion
+ optional object_motion[object_id] = per-frame object pose/motion
+ optional contact/support windows

The pipeline should not make every stage rediscover object motion by probing stage files. If a tracking input file is needed, write it once from the interaction motion near the end of refinement.

This matches the current HOI-FHLI behavior: grasp keyframe optimization solves the hand against a fixed object pose, while sequence local refine keeps the Kimodo body/root scaffold mostly fixed but can refine the hand pose and bounded object motion together in the interaction window.

Keyframe Role

A keyframe is a constraint artifact, not a top-level pipeline phase. It can be produced by interaction logic and then used in different places:

  • Pre-scaffold keyframes become Kimodo scaffold constraints, for example pick reach, pick grasp, place reach, or place grasp constraints.
  • Post-scaffold keyframes can correct or lock the interaction motion, for example final place/release correction.
  • The solver/helper code can be shared, but the logical role must stay clear: generating a scaffold constraint is not the same operation as refining an already generated interaction motion.

So grasp-related code belongs to the interaction domain, while the resulting keyframe may be consumed either before Kimodo generation or during post-scaffold refinement.

Refine Interaction Contract

Keep the refine interaction boundary simple:

input:
  scaffold_motion
  constraints
  task/layout geometry
  segment timing

output:
  interaction_motion

interaction_motion always contains human motion. It contains object motion only when the action semantics require a moving object.

interaction_motion:
  human_motion: required
  object_motion: optional
  contacts: optional
  support_windows: optional

Do not pass a long object-specific request dict through the pipeline. The inputs should be semantic and compact:

  • scaffold_motion: the Kimodo-generated human scaffold.
  • constraints: action-derived constraints such as hand/object keyframes, object relation schedules, contact windows, support windows, and local-refine settings.
  • task/layout geometry: object ids, support ids, meshes, scene coordinates, and active hands.
  • segment timing: pick/carry/place/sit/etc. frame windows.

Output rules:

  • No movable object: return human-only interaction_motion; do not invent empty dynamic-object fields.
  • Static support interaction: return human motion plus support/contact windows; no rigid object motion unless the support itself is meant to move.
  • Movable rigid object: return human motion plus per-object rigid 6D object_motion.
  • Articulated object: later return articulated object motion, not fake rigid object motion.

The .npz representation may keep the current dynamic_object_* fields for compatibility, but conceptually those fields are the serialized object_motion inside interaction_motion.

Code Design

Use a small data-flow design, not a class hierarchy:

info = kimodo.prepare_task_scaffold(runtime)
constraints = constraint.prepare(runtime, info)
scaffold_motion = kimodo.generate_scaffold(runtime, constraints)
interaction_motion = refine_interaction(runtime, scaffold_motion, constraints, info)
metrics = evaluation.measure(runtime, interaction_motion, info)

refine_interaction can remain visible in pipeline.py while the code is being simplified. It should be a grouped orchestration block, not a new wrapper layer:

interaction_motion = _refine_interaction(runtime, scaffold_motion, constraints, info)

The important boundary is the data contract:

  • constraint.prepare(...) consumes semantic anchors and returns a compact dict of action-derived artifacts: scaffold constraint paths, object-motion binding/release specs, hand-motion specs, post-refine specs, object relation schedules, contact windows, and support windows.
  • kimodo.generate_scaffold(...) consumes only the scaffold-relevant part of those constraints.
  • refine_interaction consumes the interaction-relevant part and returns one interaction_motion.
  • object_motion is optional inside interaction_motion; functions must not assume every action has dynamic_object_* fields.

Avoid copied request bags with many task ids, paths, and runtime knobs. Object motion refinement should receive the scaffold motion plus the compact interaction constraints it actually owns.

Prefer semantic handoff:

constraints["interaction"] = {
    "op_events": [
        {"op": "pick_grasp", "object_id": "...", "step_id": 1},
        {"op": "bind_object", "object_id": "...", "step_id": 1},
        {"op": "keep_object_bound", "object_id": "...", "step_id": 2},
        {"op": "prepare_place_release", "object_id": "...", "step_id": 3},
        {"op": "release_object", "object_id": "...", "step_id": 3},
    ],
    "object_motion": {
        "mode": "pick_carry_place_release",
        "required": True,
        "bound_intervals": [
            {
                "object_id": "...",
                "bind_step_id": 1,
                "bound_step_ids": [2],
                "release_step_id": 3,
                "binding": {...},
                "release": {...},
            },
        ],
    },
    "hand_motion": {
        "pick_pose": {"artifact": "..."},
        "place_pose": {"artifact": "..."},
        "schedule": {"artifact": "..."},
    },
    "post_refine": {
        "final_place_keyframe": {...},
        "preserve_pick_grasp_source": "...",
    },
    "object_relations": [...],
    "contact_windows": [...],
    "support_windows": [...],
}

Then each domain function reads only the entries it owns and fails clearly if a required interaction artifact is missing.

The difference from v13 is dispatch, not motion behavior:

  • v13: fixed script order plus many args.
  • new path: ordered task actions produce ordered execution intents.

Pipeline files should remain orchestration only: prepare state, call domain APIs, write final provenance.

Execution Plan

Use small plain-function layers:

utils/hsi_generation/planning.py
utils/hsi_generation/anchors.py
utils/hsi_generation/constraint.py    # anchors -> solver/postprocess inputs
utils/hsi_generation/object_motion.py # object relation apply inside refine
utils/hsi_generation/hands.py         # hand/object refinement inside refine

Keep it plain:

  • no class hierarchy
  • no registry framework
  • no validation state machine
  • no hidden config bundle
  • plain dict and list

Suggested API:

build_clips(task: dict) -> list[dict]
build_execution_plan(clips: list[dict], geometry: dict) -> list[dict]
build_anchors(plan: list[dict]) -> dict[str, list[dict]]
build_constraints(anchors: dict, task: dict, geometry: dict) -> dict
generate_scaffold(constraints: dict) -> MotionState
refine_interaction(scaffold_motion: MotionState, constraints: dict) -> InteractionMotion

InteractionMotion is a semantic contract, not a required class hierarchy. In the current code it may still be represented by one .npz/MotionState, but the meaning should be:

human motion: always present
object motion: optional, only for moving-object actions
contacts/supports: optional, for refinement and audit

Each intent is an ordered action record. The action is the ownership boundary, but the intent itself should stay semantic. It should not list executable operations; those are derived later by the constraint layer from anchors:

{
    "action": "pick",
    "step_id": 2,
    "object_id": "...",
    "target_id": "...",
    "active_hand": "left",
}

Do not put operations, needs, or requirements on intent records. The clean layering is:

planning.py   task -> intent
anchors.py    intent -> semantic anchors
constraint.py anchors -> executable op events

Cross-action dependency must be explicit and small. For example, place may consume the pick action's optimized grasp keyframe as its grasp/source relation, but pick_grasp_optimization remains owned by pick; it must not become a global fallback, public flag, or old-run lookup.

Action State Continuity

Every action must expose the state it changes, and the next action must start from any still-active state unless it explicitly releases, overrides, or ends that state. This is the core sequence contract:

action_i output state -> action_{i+1} input state

This should stay simple. Do not add a large lifecycle framework. Pass a plain small state record between ordered operations when the state is still active.

Examples:

sit
  end_state.posture = seated(support_id, root, heading, body_pose)

get_up
  start_state.posture must be the preceding seated state on the same support
  end_state.posture = standing(root)

pick
  end_state.object[object_id] = bound_to_hand(hand, local_pose)

carry
  start_state.object[object_id] must be bound_to_hand(...)
  end_state.object[object_id] remains bound_to_hand(...)

place
  start_state.object[object_id] must be bound_to_hand(...)
  end_state.object[object_id] = supported_on(support_id)

walk / reach
  end_state.root = path_end or root_target
  the next action starts from that root unless it owns a more specific state

The sit -> get_up case is the posture version of this rule: the seated root and seated full-body pose at the end of sit must also be the start state of the following get_up. Otherwise the solver can jump from the seat back to a standing point in one frame.

Anchors are plain semantic records. They are not config, not extra task schema, and not keyframes:

{
    "action": "pick",
    "step_id": 2,
    "target": "wrist_object",
    "purpose": "pick_reach",
    "object_id": "...",
    "target_id": "...",
    "active_hand": "left",
}

Anchors must stay semantic and sparse; they should not become a trace of v13 stage names. For example, place anchors describe approach, object-on-support, hand/object release, object/support contact, and hand-to-support transfer. The constraint layer owns the detailed expansion into place reach, release alignment, place grasp optimization, and final keyframe application.

Constraints are the first executable layer. They should be explicit by type:

{
    "kimodo_constraints": [...],
    "hoi_constraints": [...],
    "contact_schedule": [...],
    "object_relation_schedule": [...],
}

Keyframes are constraints, not a separate top-level stage. A grasp keyframe, posture keyframe, or final object placement keyframe belongs under the appropriate constraint bucket.

Action Operations

Initial operation mapping:

pick
  pick_reach_keyframe
  pick_grasp_optimization
  pick_grasp_constraint_replacement
  object_binding_anchor
  hand_grasp_pose

carry
  carry_scaffold
  preserve_object_in_hand
  carry_contact_labels

place
  place_reach_keyframe
  place_grasp_keyframe
  release_alignment
  release_binding_anchor
  hand_release_schedule
  final_place_keyframe

reach
  scaffold_constraint

internal turn
  scaffold_constraint
  root_facing waypoint anchor

sit / get_up / lie_down
  posture_scaffold_constraint
  posture_state_continuity
  support_contact_labels

push / pull / kick
  contact_scaffold_constraint
  object_contact_motion
  measured_contact_labels

object_contact_motion is a planned movable-object solver operation, not the same thing as pick/carry/place attach. Until that solver exists, native runtime support should fail clearly instead of routing push/pull/kick through pick/carry/place release logic.

For pick/carry/place, target behavior is v13-equivalent:

  • pick establishes a high-quality grasp keyframe and hand pose target.
  • the pick grasp becomes the active-hand object binding anchor.
  • carry preserves the hand-object local relation established by pick.
  • place builds a place keyframe and aligns release to that keyframe.
  • place supplies the release binding anchor.
  • hand pose baking uses a schedule, not only a generic prior.
  • if a trustworthy grasp relation exists, local refine preserves it.
  • final place keyframe application remains in the generation path.

This is a behavior requirement, not a path requirement. The new pipeline must run the required HOI-FHLI pick grasp optimization itself. Do not hard-code the N1 object, dated output directories, seed-specific files, or a v13 branch, and do not load old v13 artifacts as runtime input state.

If optimized grasp/keyframe assets are reused in future, selection must happen through a general domain mechanism keyed by action, object geometry/class, active hand, and target relation. Otherwise the operation should run the appropriate HOI-FHLI/Kimodo domain optimization or fail clearly instead of silently degrading to low-quality defaults.

Pick/Carry/Place Example

The current N1 case should map to the four-layer chain like this:

pick
  anchor:
    approach_object, pick_grasp, hand_object contact_start,
    object_in_hand binding_anchor
  constraint:
    Kimodo wrist/full-body keyframe constraint
    HOI-FHLI grasp keyframe optimization input/output
    hand-object contact schedule start
    object-in-hand relation schedule start
  scaffold motion:
    Kimodo-generated human motion with pick constraints
  refined motion:
    object motion follows the active hand
    grasp hand pose is baked/refined while preserving relation

carry
  anchor:
    carry_path, hand_object contact_hold, object_in_hand preserve_relation
  constraint:
    Kimodo root/path constraint
    hand-object contact hold schedule
    object-in-hand relation schedule
  scaffold motion:
    Kimodo-generated human motion along the carry path
  refined motion:
    object follows active-hand local binding through the carry window

place
  anchor:
    approach_support, object_on_support place_target,
    hand_object release, object_support support,
    object_transfer hand_to_support
  constraint:
    Kimodo place reach/final object target constraints
    optional HOI-FHLI release/grasp keyframe optimization input/output
    release contact schedule
    hand-to-support object relation schedule
  scaffold motion:
    Kimodo-generated human motion with place constraints
  refined motion:
    object releases from active hand to support/target
    hand pose opens/releases
    final placement correction is applied when needed

This is the general pattern. Other actions should add their own anchors and constraints, then reuse the same scaffold/refine phases instead of adding a new pipeline branch.

Target Structure

Keep the package small and layered:

utils/hsi_generation/
  pipeline.py        # orchestration only
  files.py           # paths/env/output/provenance, no hidden business policy
  planning.py        # task clips -> execution plan
  anchors.py         # execution plan -> semantic anchors
  constraint.py      # anchors -> solver/postprocess constraints
  kimodo.py          # scaffold spec/generation/root cleanup boundary
  keyframe.py        # temporary owner for grasp/object/hand keyframe logic
  object_motion.py   # object relation apply inside interaction refinement
  hands.py           # hand schedule, bake, and local hand/object refinement
  contact.py         # expected/measured contact schedules and timelines
  evaluation.py      # metrics/render/validation aggregation
  geometry.py        # reusable coordinate/rotation/scene geometry

constraint.py is a target consolidation point, not a wrapper layer. Keep the direct object_motion and hands calls visible in pipeline.py until a real shared interaction-refine domain API exists. Group them as one top-level refine_interaction phase, because object binding, hand bake, and local refine are all post-scaffold refinement.

Introduce subfolders only when a file becomes genuinely too large and the split removes complexity. Do not split one action per file as a default.

If actions/ is needed later, prefer coarse grouping:

utils/hsi_generation/actions/
  object_ops.py      # pick/carry/place/push/pull/kick
  locomotion.py      # reach/turn/walk-like scaffold operations
  posture.py         # sit/get_up/lie_down

Layer Boundaries

Owned by hsi_generation:

  • task clips and execution operations
  • stage orchestration
  • object binding semantics
  • contact schedule coordination
  • canonical artifacts and validation aggregation

Owned by utils/kimodo:

  • Kimodo prompt/constraint/meta/info schema
  • scaffold generation
  • Kimodo motion adapters
  • root cleanup

Owned by utils/hoifhli:

  • HOI-FHLI/DexGraspNet-style grasp optimization
  • MANO/SMPL-X hand model glue
  • grasp keyframe solving

Owned by utils/visualization:

  • scene-human-object rendering
  • camera policy
  • preview/video export

Owned by utils/metrics:

  • contact distance
  • penetration
  • temporal/motion quality metrics
  • strict metric computation

Forbidden directions:

kimodo -> hsi_generation
hoifhli -> hsi_generation
visualization -> hsi_generation
metrics -> hsi_generation
hsi_generation -> tools/scripts

Non-Negotiable Constraints

  • pipeline_manifest.json is output provenance only. It must not be runtime state or input config.
  • The new pipeline must not infer config from old case directories or old manifests.
  • Do not reintroduce legacy baseline-config or alignment execution modes.
  • Do not put command-line or manual scripts under utils.
  • Do not create utils/scripts, utils/*/scripts, or utils/*/cli.py.
  • Do not add wrapper layers just to keep old call paths alive.
  • Required task/config fields should be accessed explicitly; missing fields should fail immediately.
  • Stage/runner files should not contain math, geometry, rotation logic, KNN, or long field mapping.
  • Shared geometry/math should live in a utility module.
  • Shared losses should live in solver/loss modules.
  • Old tools may stay under tools/ or utils/*/legacy, but new pipeline code must not depend on them.

Refactor Order

  1. Keep the public interface fixed at --task-plan --layout-json --out-dir.
  2. Keep planning.py and anchors.py small; anchors remain semantic records.
  3. Keep improving the constraint-building layer. It now converts anchors into ordered op events, executes implemented operations in event order, validates bind/release ordering, and passes object_motion.bound_intervals into interaction refinement. Operation state is keyed by object and step, so repeated pick/place intervals are represented as data instead of singleton stage state.
  4. Make Kimodo generation consume a scaffold spec and run once per sequence.
  5. Make post-Kimodo interaction refinement consume explicit constraints: object binding/release, hand bake/refine, support transfer, and final placement are one refine phase rather than separate top-level pipeline stages.
  6. Restore v13-equivalent pick/carry/place behavior through the generic constraints/refine event flow: pick grasp, binding anchor, place grasp, release alignment, hand schedule, preserve relation, final place correction.
  7. Continue moving stage code from implicit runtime settings/artifact guessing to explicit action/anchor/constraint consumption.
  8. Add clear unsupported-action behavior for actions whose domain primitive is not implemented yet.
  9. Extend constraints/refine support to sit/get_up/lie_down; each posture enter operation must expose a support state, and matching exit operations must inherit that state as their start state.
  10. Extend constraints/refine support to push/pull/kick.
  11. Continue file simplification and merging after behavior is stable.

Current Code Checkpoint

Completed in the current code:

  • Public HSI generation stays task-first: --task-plan --layout-json --out-dir.
  • planning -> anchors -> op_events exists for the supported action families.
  • Kimodo scaffold input now uses the ordered generic plan_to_constraints builder for task-plan steps, including pick/carry/place sequences. The new HSI pipeline no longer selects the old pick/carry/place task scaffold, so a task such as reach -> pick -> carry -> place keeps the initial reach in the scaffold timeline.
  • Task step prompt/text is preserved as the Kimodo scaffold text constraint when present; action templates are fallback only.
  • The constraint layer now dispatches implemented operations by ordered op_events instead of using global singleton _has_op(...) checks.
  • Object bind/release ordering is represented as interaction.object_motion.bound_intervals, not cycle or cycles.
  • Binding and release contracts live on the interval that owns them. The refine layer should not read global interaction.object_motion.binding or interaction.object_motion.release singleton state.
  • Hand bake and final place refine are also list-based: pick/place hand poses, schedules, and final place keyframes are recorded per event before the hand stage merges schedules or applies the keyframes in order.
  • The implemented object-motion backend supports one or more bound intervals per dynamic object and can serialize multiple dynamic objects in one sequence. For compatibility it still writes the first object to the legacy dynamic_object_* fields, and also writes per-object fields such as dynamic_object_<token>_pos, dynamic_object_<token>_rot_mat, and dynamic_object_<token>_vertices.
  • pipeline.py has one post-scaffold interaction-refine boundary: object motion, hand bake/refine, local interaction refine, and final place keyframe application are downstream of _refine_interaction(...).
  • Pick/place reach keyframe operations pass solver artifacts through explicit Python return values. Grasp optimization consumes those return values and no longer probes stage-root fixed filenames.
  • HSI operation contracts no longer pass operation_dir as cross-stage state. When an external solver needs files, downstream code derives the operation directory from the explicit keyframe_artifact it already received.
  • Scaffold constraints now move through explicit constraints_json artifacts returned by each operation. stage_dir/constraints.json is only the initial Kimodo scaffold input written by the scaffold builder; later HSI operations no longer copy their result back over that stage-root file as a hidden state bus.
  • Pick/place binding, hand-pose, and release handoffs use explicit contracts such as keyframe, keyframe_artifact, hand_pose, and schedule instead of leaking *_npz filenames as the interaction state shape.
  • write_pick_carry_place_object_motion(...) consumes explicit object_intervals contracts rather than a spread of script-style binding/release arguments.
  • pipeline_manifest.json remains provenance, not runtime state.
  • Full left/right task-plan generation is intentionally deferred until this refactor checkpoint passes focused tests and review. Do not launch long generation runs while the 1-7 cleanup is still in progress.
  • Pick grasp uses the higher-quality native search policy used by the old official-grasp path: batch size 32 and 600 iterations.
  • Pick grasp now restores the old official MANO probe/apply behavior as utils.hoifhli.official_grasp domain APIs. The pick operation passes the pre-grasp motion and official-applied keyframe in memory, then writes only the final pick keyframe artifact plus concise provenance/object-model cache needed by the official ObjectModel.
  • HSI hand-stage provenance now uses hand_pose_artifact and hand_pose_schedule_artifact; old *_npz/*_json names remain only at lower-level solver call boundaries where those functions still accept file paths.
  • Human-only scaffold tasks can now use the generic Kimodo plan_to_constraints domain builder from the task json interface. This covers reach, sit, get_up, and lie_down as scaffold actions without requiring a pick/place object pair; interaction refinement then records object motion as not applicable.
  • Generic scaffold no longer builds a temporary old plan_dir or manifest.json as a runtime state bus; HSI calls the in-memory write_kimodo_inputs_from_steps(...) domain API and only persists Kimodo's required meta.json, constraints.json, and info.json.

Still not complete:

  • Pick/carry/place still needs final left/right full-run validation after the official-grasp domain API integration. The expected flow is now present, but motion quality and v13 metric parity must be checked by regenerating the two task-plan cases once the code review checkpoint is accepted.
  • Multi-object object-motion serialization is implemented. Current render and metrics consumers should still be treated as primary-object consumers until they are extended to read dynamic_object_field_tokens.
  • push/pull/kick are not fully connected to the same constraint/refine backend path because the object contact-motion solver is still missing. They should keep failing clearly instead of pretending that scaffold-only motion moved the object.
  • File count and file boundaries can be reduced further after behavior is stable; the current utils/hsi_generation code is already below the 8k-line target.

Immediate Code TODO

Next cleanup should stay bounded:

  • Keep extending action coverage through the same anchor -> constraint -> scaffold motion -> refined motion flow.
  • Extend render and metrics to consume the per-object dynamic-object fields when multi-object visual/metric validation becomes necessary.
  • Do not reintroduce stage-root fixed solver filenames, manifest probing, or task-specific branches while adding action operations.
  • Run the left/right task-plan cases after the current code checkpoint is accepted, then compare contact metrics and visual quality against v13.
  • Merge files only after behavior is stable and tests cover the affected boundary.

Verification Plan

Focused tests after each meaningful change:

PYTHONPATH=/mnt/data/PhysHSI pytest -q \
  tests/test_hsi_generation_pipeline.py \
  tests/test_hsi_generation_binding.py \
  tests/test_expected_contact_labels.py \
  tests/test_measured_contact_labels.py \
  tests/test_kimodo_contact_region_labels.py \
  tests/test_physhsi_metrics.py

If touching Kimodo-facing modules:

PYTHONPATH=/mnt/data/PhysHSI pytest -q \
  tests/test_kimodo_plan_to_constraints.py \
  tests/test_object_keyframe_binding.py \
  tests/test_lie_keyframe_binding.py \
  tests/test_kimodo_render_contact_overlay.py

For motion-quality regression:

  • run full left task plan;
  • run full right task plan;
  • render full opaque scene previews;
  • compare against v13 reference behavior, not just pass/fail artifacts;
  • inspect root motion, object motion, hand-object contact, place/release, and final rendered motion quality.

When CUDA-specific HOI-FHLI/Kimodo behavior is involved, use the project Isaac/HOI environment rather than silently falling back to CPU behavior.

Review Workflow

Code changes that implement this plan should use the project review-agent workflow:

  • patch a complete bounded batch of related modules;
  • run focused tests for that batch;
  • spawn one reviewer subagent after the batch, not after every tiny patch;
  • ask the reviewer to check behavior equivalence, simplicity, no hidden wrappers, no manifest-as-state, no v13 hard-code, and no unnecessary config knobs;
  • integrate or reject reviewer findings before continuing.

Final Pass Criteria

The refactor is successful when:

  • public entry remains --task-plan --layout-json --out-dir;
  • utils/hsi_generation has a small, readable flow;
  • pick/carry/place generated from task operations is v13-equivalent in behavior;
  • v13 is a regression case, not a code branch;
  • action operations can represent the current SAGE action set;
  • unsupported operations fail clearly;
  • no new wrappers/legacy dependencies are introduced;
  • no runtime state is read from old manifests or case directories;
  • full left/right task runs produce motion quality no worse than the v13 reference by visual and metric sanity checks.