跳转至

Kimodo Object 6D Init + Local Grasp Case Report

Date: 2026-06-18

Current Status

The old R1-R9 and N1-N6 pass labels are invalid. They came from a multi-stage object attach/repair attempt that could satisfy wrist/socket proxy metrics while still failing visible hand-object grasp.

This report now tracks the simplified plan:

motion.npz
-> motion_augmented.npz
-> motion_hoi_refined.npz
-> strict audit / interaction review / report

No case is currently marked final pass under the revised gates.

Cleanup Performed

The previous broad object-trajectory repair route has been removed from the active worktree:

build_kimodo_object_attach_plan.py
repair_kimodo_object_trajectory_contact.py
repair_kimodo_object_trajectory_hand_contact.py
run_kimodo_object_attach_repair_replay_matrix.py
run_kimodo_hoi_new_pipeline_matrix.py
audit_kimodo_hoi_case_quality.py
audit_kimodo_old_output_matrix.py
render_kimodo_interaction_review.py
docs/related_work.md
kimodo_hoi_refine_port/

The HOI-FHLI contact-region cache was moved out of the temporary port copy and into:

data/contact_regions/human/smplx_fingertip_palmar_v1.npz
data/contact_regions/human/smplx_fingertip_palmar_v1.summary.json

N1 Base Motion Evidence

Fixed no-object base motion found:

output/auto_experiments/20260616_handed_contact_pipeline/case_cup_left_from_planner_response_leftfix_hoi_v1/motion.npz

It contains only Kimodo human motion fields and no dynamic_object_* arrays.

The corresponding raw Kimodo output is:

output/auto_experiments/20260616_handed_contact_pipeline/case_cup_left_from_planner_response_leftfix_hoi_v1/motion_raw_kimodo.npz

Both are valid no-object base inputs. The next experiments should use motion.npz as the fixed base unless a raw-output ablation is explicitly requested.

Current Implementation Direction

Active code changes now target only:

  • object 6D initialization in attach_task_object_to_kimodo_motion.py;
  • expected left/right contact label materialization;
  • one local HOI-FHLI-style hand/grasp refinement stage;
  • hand-object contact metrics.

The current object init change adds a single-hand grasp-volume-mode=auto that derives object-center offsets from object size and Kimodo root heading. It does not edit root/body motion and does not run a broad object trajectory optimizer.

Still Required

The old broad trajectory audit/render scripts were replaced by simplified gate artifacts that do not consume object_attach_plan.json.

Current required artifacts:

case_quality_audit.json
strict_hand_object_audit.json
motion_scene_dynamic_object_hands.mp4
interaction review video and representative frames
visual_qa_report.json, if visual QA is performed

A case can pass only if the final motion satisfies:

active_hand_p05_m <= 0.04
active_hand_p50_m <= 0.12
active_hand_ratio_lt_5cm >= 0.12
strict_hand_object_audit_pass == true
temporal_render_qa_pass == true
body_freeze_delta_pass == true
release_place_non_contact_pass == true

Implemented Simplified Pipeline State

The main pipeline has been changed to use one final motion path:

motion.npz
-> motion_augmented.npz
-> motion_augmented_hands.npz
-> motion_hoi_refined.npz
-> metrics / strict audit / render / case_quality_audit.json

The relevant implementation files are:

tools/kimodo/attach_task_object_to_kimodo_motion.py
tools/kimodo/refine_kimodo_hand_sequence_contact.py
tools/kimodo/audit_kimodo_hand_object_pose.py
tools/kimodo/audit_kimodo_hoi_case_quality.py
tools/kimodo/run_kimodo_hoi_pipeline.py

The local HOI-FHLI-style stage freezes body/root/object trajectory and only optimizes bounded hand pose deltas. Side-level generated labels such as left_hand_object are consumed as HOI-FHLI-style left/right contact intent; inside the loss they expand to balanced fingertip/palmar candidates so one nearest palm patch cannot satisfy the entire hand-object label.

N1 Pilot: Fixed Base + Object 6D Init + Local Grasp

N1 uses the fixed no-object Kimodo base:

output/auto_experiments/20260616_handed_contact_pipeline/case_cup_left_from_planner_response_leftfix_hoi_v1/motion.npz

Current N1 pilot directory:

output/auto_experiments/20260618_contact_gated_repair/n1_jixuan_base_object_attach_probe_v3/palm_capture_auto/hand_refine_pilot_balanced_stride6_i60

Important artifact paths:

final motion:
  motion_augmented_hands_refined.npz

strict audit:
  strict_hand_object_audit.json

case quality audit:
  case_quality_audit.json

full-scene review video:
  interaction_review_full_scene_balanced.mp4

representative final frame:
  interaction_review_full_scene_balanced_final_frame.png

Strict active-hand distance metrics:

strict_hand_object_audit_pass: false
active_hand_p05_m: 0.003474632976576686
active_hand_p50_m: 0.026618951931595802
active_hand_ratio_lt_5cm: 0.8902154564857483
active_grasp_opposition_ratio: 0.0

Per-window strict records show why nearest-distance metrics were misleading:

pick_active left:
  p05 0.002530069788917899
  p50 0.01722383312880993
  ratio_lt_5cm 0.9701687097549438
  grasp_opposition_ratio 0.01098901098901099
  grasp_semantic_gate_pass false

carry left:
  p05 0.003474632976576686
  p50 0.025983788073062897
  ratio_lt_5cm 0.9142483472824097
  grasp_opposition_ratio 0.0
  grasp_semantic_gate_pass false

place_active left:
  p05 0.0032954965718090534
  p50 0.026618951931595802
  ratio_lt_5cm 0.8902154564857483
  grasp_opposition_ratio 0.16494845360824742
  grasp_semantic_gate_pass false

Freeze gate:

root_positions max delta: 0.0
local_rot_mats max delta: 0.0
global_rot_mats max delta: 0.0
posed_joints max delta: 0.0
smooth_root_pos max delta: 0.0
global_root_heading max delta: 0.0
dynamic_object_pos max delta: 0.0
dynamic_object_rot_mat max delta: 0.0
foot_contacts max delta: 0.0

Case-quality status:

final_motion_exists: true
strict_hand_object_audit_pass: false
active_hand_threshold_pass: true
grasp_semantic_gate_pass: false
body_freeze_delta_pass: true
render_artifact_exists: true
temporal_render_qa_pass: false
case_quality_pass: false
failure_reasons:
  strict_hand_object_audit_pass
  grasp_semantic_gate_pass
  temporal_render_qa_pass

Interpretation: N1 is a visual and semantic grasp failure. The active hand is near the cup, but the cup is not naturally grasped. Close-up review shows the cup suspended under/against the hand with weak thumb opposition. The new grasp-semantic gate catches this: carry has zero frames satisfying thumb-plus-opposing-region contact around the cup.

Subagent visual QA independently reached the same conclusion: N1 cup-left is not a natural grasp; proximity-only gates missed side-gluing/rim contact.

Immediate method implication: the current single-hand object init places the cup center too low relative to the active hand. The local hand-only HOI-FHLI stage can improve nearest distances, but it cannot turn a cup hanging under the hand into a thumb-opposed grasp. The next N1 repair should revise the single-hand cup grasp volume so the cup body, not just the rim/top, lies between thumb and opposing fingers before local refinement.