Kimodo Object 6D Init + HOI-FHLI Local Grasp Plan

Date: 2026-06-18

Current Decision

This document supersedes the earlier object attach/contact repair plans, including the "Critical Revision: General Contact-Gated Fix" wording.

The revised direction is deliberately simpler:

Kimodo generates the human base motion.
attach_task_object_to_kimodo_motion.py adds the initial object 6D trajectory.
One HOI-FHLI-style local refinement stage improves grasp/contact.
Strict audit and temporal visual QA decide pass/fail.

The previous multi-stage repair chain is not the default pipeline:

object_attach_plan-driven trajectory
-> coarse object trajectory repair
-> hand bake
-> contact-aware object trajectory repair
-> hand local repair
-> re-audit/render

That direction made the system harder to reason about and allowed broad object translation optimizers to compensate for a bad initial object pose. It also encouraged judging success by nearest-distance metrics instead of natural grasp semantics.

The old R1-R9 and N1-N6 pass labels remain invalid. A case can pass only after the new strict gates and visual QA run on the exact final motion.

Root Cause

There were two distinct failures:

The old object attachment and repair used a wrist/socket proxy. A cup could be numerically near a wrist-derived socket while not being naturally grasped by the actual SMPL-X/MANO hand mesh.
The attempted repair expanded into a multi-stage object trajectory pipeline. This was the wrong abstraction. Kimodo already produces the base motion; the repair should not redesign body/root motion or use broad object drift to chase contact metrics.

The correct fix is to improve the initial object 6D pose so the object is already plausibly located in the active hand, then run a bounded local HOI-FHLI-style grasp refinement.

Reference Behavior From The Previous Pipeline

The earlier working object 6D pose was not generated by Kimodo and was not estimated from video. It was added after motion.npz by:

tools/kimodo/attach_task_object_to_kimodo_motion.py

Its policy was:

translation:
  before attach: layout object center
  after attach/carry: active wrist anchor plus object-center offset
  place/release: blend to task support target

rotation:
  initial object layout rotation
  slerp toward support/table rotation during place

For N1 cup-left, the important fields are:

dynamic_object_pos      # per-frame object translation
dynamic_object_rot_mat  # per-frame object rotation
scene_origin_xy         # renderer/layout origin, must be preserved

This behavior is the baseline to improve. The improvement should be made in the object 6D init logic, not by adding downstream object trajectory repair stages.

Target Pipeline

The target default pipeline is:

1. Build Kimodo inputs and expected contact labels.
2. Generate Kimodo base motion:
     motion_raw_kimodo.npz
     motion.npz
3. Attach object 6D init:
     motion_augmented.npz
4. Run one HOI-FHLI-style local grasp refinement:
     motion_hoi_refined.npz
5. Run measured contact labels and strict audits.
6. Render full-scene opaque video and interaction review artifact.
7. Run temporal/visual QA.
8. Update report with pass/fail reasons.

The intended artifact chain is:

motion.npz
-> motion_augmented.npz
-> motion_hoi_refined.npz
-> audit/render/report

The final file name may change during implementation, but there should be one clear final motion path in pipeline_manifest.json. Metrics and renders must use that exact path.

Removed From Default Mainline

The following stages are legacy/debug only and must not be part of the N/R default pipeline:

object_attach_plan driving dynamic_object_pos
motion_augmented_object_repaired.npz
motion_augmented_object_contact_repaired.npz
coarse object trajectory repair
contact-aware object trajectory repair as a broad object translation optimizer
re-bake / repair / re-repair loops
passing cases by moving the object to satisfy nearest-distance gates

The previous scripts for those stages should not be kept in the default implementation. Historical outputs can still be audited by pointing the new strict audit at their final motion files, but no new N/R case should depend on an object_attach_plan.json or a broad object-trajectory repair stage.

Implementation Plan

1. Restore The Main Orchestrator To A Stable Baseline

run_kimodo_hoi_pipeline.py should be treated as the orchestration entrypoint, not as the place to accumulate experimental repair logic.

Required behavior:

keep Kimodo generation behavior close to the previous git pipeline;
keep single-hand carry default simple and motion-quality-preserving;
do not inject body/root corrections for grasp repair;
record the base motion path and final motion path in the manifest;
call only the simplified object attach and local HOI refinement stages.

2. Fix Object 6D Init In attach_task_object_to_kimodo_motion.py

This is the primary object-side fix.

The object init stage should:

leave Kimodo body/root/local rotations unchanged;
resolve active hand from task semantics;
place the object in a plausible grasp pose near the active hand during attach/carry;
keep object motion smooth through attach, carry, release, and place;
preserve the task/layout/support target for placement;
keep scene_origin_xy correct for render and audit;
record all offsets, anchors, rotations, and thresholds in metadata.

For a generated cup case, a good initial object pose should be close enough that local refinement only needs small corrections.

Object init may use general object/category rules, for example:

cup, single hand:
  active side = left/right
  object center near palm/finger grasp volume
  object up axis remains plausible
  cup body between palm and opposing fingers

both-hand medium object:
  object center between hands
  object rotation follows layout/support unless task semantics provide better

These rules must be category/geometry based, not case-ID hacks.

3. Keep HOI-FHLI-Style Left/Right Contact Labels

The current generated contact channels are already the right primary semantics:

left_hand_object
right_hand_object
both_hands_object
object_carried
object_support

The core HOI-FHLI contact condition is effectively per-frame left/right hand object contact. Our left_hand_object, right_hand_object, both_hands_object, and object_carried channels already provide the same kind of signal once they are materialized into frame windows.

Therefore, do not replace the primary contact labels with left_thumb/left_index/left_middle/left_palm. Those fine-grained labels are not the main generated contact condition in HOI-FHLI. They are useful as diagnostics or optional weak grasp-profile priors, but not as the primary task contact labels.

The required contact-label work is narrower:

keep generating the existing left/right/both/carried/support channels;
ensure the active hand and contact windows are correct for generated cases;
ensure object_carried resolves to the active hand side during carry;
make local refinement consume these labels as left/right contact windows;
report finger/palm participation in audit without making it the primary contact-label schema.

The contact-region cache remains useful for diagnostics and optional priors:

data/contact_regions/human/smplx_fingertip_palmar_v1.*

Audit output must report both:

active hand mesh distance metrics;
HOI-FHLI contact-candidate metrics for the active side;
optional palmar/fingertip diagnostics, such as thumb opposition, index/middle participation, and palm proximity.

The local refinement stage should be conceptually similar to HOI-FHLI stage2:

input:
  motion_augmented.npz
  left/right expected contact windows
  object mesh/pose

optimize:
  active hand pose delta

freeze:
  dynamic_object_pos
  dynamic_object_rot_mat
  root_positions
  body local_rot_mats outside hand pose
  non-hand body joints

This stage should not be a broad object trajectory repair. It should refine a good initial HOI state into a natural grasp. The default implementation keeps the object 6D trajectory fixed after attach_task_object_to_kimodo_motion.py; any future small object-delta ablation must be explicit and non-default.

Required constraints:

CUDA must be used for optimization;
no silent CPU fallback;
deltas must be bounded and reported;
object placement/release must not acquire side effects;
force-closure, penetration, and semantic contact terms should be enabled or explicitly reported as disabled;
if the initial object pose is too far from a plausible grasp, fail or return a clear failure reason instead of dragging the object through the sequence.

5. Keep Gate/QA Separate From Repair

Audit and render QA are hard gates. They should not drive hidden trajectory repair.

Required artifacts:

case_quality_audit.json
strict_hand_object_audit.json
motion_scene_dynamic_object_hands.mp4
interaction_review_strip.png or equivalent review artifact
visual_qa_report.json when visual QA is performed

Implemented gate scripts/stages:

tools/kimodo/audit_kimodo_hand_object_pose.py
tools/kimodo/audit_kimodo_hoi_case_quality.py
tools/kimodo/run_kimodo_hoi_pipeline.py::local_grasp_refine

Render defaults:

--scene-object-source all
--scene-alpha 1.0
--camera-view human-front
--camera-distance 1.3
--camera-elevation 8
--fov 78

Important render rule:

Always preserve/pass the correct scene origin, e.g. --scene-origin-xy.
Do not handwrite alternate render commands that drop scene-origin-xy.

Hard Gates

A case passes only if all required gates are true on the final motion:

active_hand_p05_m <= 0.04
active_hand_p50_m <= 0.12
active_hand_ratio_lt_5cm >= 0.12
strict_hand_object_audit_pass == true
temporal_render_qa_pass == true
body_freeze_delta_pass == true
release_place_non_contact_pass == true
final manifest path is the exact motion used by metrics and render

Additional semantic gates should be reported for grasp quality:

thumb/opposing-finger contact present when required
palm/fingertip contact-region labels satisfied
top closest contacts are not dominated by unlabeled full-hand vertices
penetration/intersection is within threshold
attach/carry/release/place windows pass separately

Aggregate metrics alone are not enough. Attach and carry-start frames must not be hidden by good median contact later in the sequence.

Body-freeze delta compares the final motion against the pre-local-refine motion:

root_positions max delta <= 1e-4 m
body posed_joints max delta <= 1e-4 m
body local_rot_mats Frobenius delta <= 1e-4
dynamic_object_pos max delta == 0 in the default pipeline
dynamic_object_rot_mat max delta == 0 in the default pipeline

Hand pose and re-baked hand vertices may change, but body motion must remain Kimodo's base motion unless an explicit non-default experiment says otherwise.

Experiment Order

Audit old R1-R9 and N1-N6 without modifying old outputs.

The expected result is not "old pass". Each case needs strict failure reasons or a fresh pass under the new gates.

Reproduce the Kimodo base motion for N1.

Confirm body/root motion quality before any object/hand refinement.

Fix and test attach_task_object_to_kimodo_motion.py.

For N1, inspect motion_augmented.npz before local refinement. The cup should already be near a plausible left-hand grasp volume.

Add the HOI-FHLI-style local stage.

Run N1 and require:

body freeze passes
object/hand deltas are bounded
attach/carry/release/place review looks natural
no penetration/occlusion illusion explains the apparent grasp

Run a diverse pilot:

left single-hand small object
right single-hand small object
both-hand object
support/clutter placement case

Rerun R1-R9.
Rerun N1-N6/N1-N9 as applicable.
Update docs/kimodo_object_attach_repair_case_report.md with strict metrics, temporal QA, visual QA, previews, and failure reasons.

Stop Conditions

Stop and report a failure reason instead of claiming pass if any of these are true:

the final object is not visibly grasped during expected contact windows;
the actor is off-screen or the review artifact cannot judge the interaction;
the object was moved broadly to pass nearest-distance metrics;
body-freeze delta fails in the default pipeline;
body quality is visibly worse than Kimodo base motion;
any R/N case lacks strict audit or case-quality audit artifacts;
any R/N case lacks a full-scene opaque video and interaction review artifact;
the fix depends on case-specific offsets, weights, or frame hacks.

Deliverables

The final implementation/report should provide:

modified files
new local HOI refinement script/stage, if added
old output audit summary
N1 object 6D init diagnostics
pilot summary
R1-R9 strict summary
N1-N6/N1-N9 strict summary
case_quality_audit.json paths
strict_hand_object_audit.json paths
interaction review artifact paths
actual preview frames
failure reasons for every failing case
verification commands

No case should be marked complete until every pass/fail label is backed by the new strict metrics and the correct-origin visual review artifacts.