跳转至

Kimodo Multi-Action Pipeline Reorganization

Goal

Build a clean automated pipeline that consumes SAGE scenes and task-generator tasks, then produces Kimodo motions for multiple action families without manual per-scene edits.

Required action families:

  • sit
  • carry
  • pick
  • place
  • push
  • pull
  • kick

The first quality target is globally correct and humanlike body motion across random scenes. Fine hand-object binding is a later quality layer, not the base pipeline contract.

Design Principles

  1. Start from a clean scene/task benchmark manifest.
  2. Keep task generation, Kimodo constraint generation, motion generation, object binding, rendering, and evaluation as separate stages.
  3. Reuse stable code only through narrow adapters.
  4. Keep old experiment scripts intact, but stop using them as the main orchestration surface.
  5. Record every added variable in manifests, not in ad hoc file names.

Current First Step

utils/kimodo/legacy/pipeline/scene_bank.py adds a scene bank layer. It scans extracted SAGE scenes, scores them by affordance categories, and exports:

  • data/selected_SAGE10k/scene_bank_index.json
  • data/selected_SAGE10k/kimodo_benchmark_10_scenes.json

The selection policy requires each benchmark scene to have:

  • at least one seat candidate
  • at least one support/place-surface candidate
  • at least one push/pull candidate

This is not yet the final task success metric. It is a data selection guard that prevents the weekly benchmark from silently sampling scenes that cannot exercise the required primitives.

2026-06-06 Refined Selection

The first bootstrap selection was replaced by a refined selection after expanding the candidate pool with 40 deterministic random SAGE-10k downloads through hf-mirror.

Current selection artifacts:

  • data/selected_SAGE10k/download_logs/candidate_expansion_20260606.json
  • data/selected_SAGE10k/scene_bank_index_refined.json
  • data/selected_SAGE10k/kimodo_benchmark_10_scenes_refined.json
  • data/selected_SAGE10k/kimodo_benchmark_10_scenes_refined_summary.md

The refined pool has 63 layout records. 17 records satisfy all primitive-level requirements after semantic filtering. The final 10-scene benchmark additionally limits room-type repetition with max_per_room_type=2.

The refined selector explicitly rejects small decorative objects as pick/carry/kick candidates. Pick/carry/kick objects now need portable or kickable semantic types such as box, bin, basket, tray, book, bottle, cushion, pillow, or pouf, instead of passing only because their dimensions are small.

Reusable vs. Task-Specific Logic

Reusable now:

  • SAGE layout scanning and affordance counts
  • SAGE to Kimodo root/path/heading conversion in plan_to_constraints.py
  • prompt/timeline generation per action family
  • renderer invocation and metric collection

Still task-specific or experimental:

  • instrument-tray-specific object binding
  • manually chosen grasp frames
  • HOIFHLI-derived hand-pose anchors
  • one-off H-to-K / K-to-H stitching scripts

The clean pipeline may import these later, but only after their inputs, outputs, and failure modes are made explicit.

Current Kimodo Generation Flow

The VLM side stops at a validated task plan: a SAGE layout, scene context, humanoid spawn, low-level action steps, object ids, interaction anchors, and navigation paths. The Kimodo side then treats this task plan as a structured conditioning problem. Kimodo itself remains a text-and-constraint motion generator; the pipeline adds the missing scene and object semantics around it.

The current Kimodo flow is organized into the following stages.

1. Task-to-Motion Normalization

Primary code:

  • utils/kimodo/legacy/pipeline/run_kimodo_motion_batch.py
  • utils/kimodo/legacy/pipeline/run_kimodo_task_motion_pipeline.py

The batch runner scans layout_*/task_plan.json, chooses the backend, records one manifest entry per scene, and performs lightweight preflight checks before generation. The generic task-motion pipeline then loads the task plan, layout, scene_context.json, and humanoid parameters. It resolves every step into a Kimodo segment with:

  • an action prompt, for example walk, sit, get up, carry, place, push, pull, or kick;
  • a duration and frame range;
  • a root target trajectory in Kimodo coordinates;
  • optional full-body, hand, foot, or object-contact constraints.

This stage is deliberately separate from VLM task generation. The VLM chooses what should happen; this adapter decides how Kimodo can condition motion on it.

2. Coordinate, Root, and Heading Conversion

Kimodo and SAGE do not use the same root/path convention. The pipeline converts SAGE world-space xy/z anchors into Kimodo root coordinates, computes local heading vectors, and writes Root2D constraints for navigation-like segments. For reach, carry, and movement before/after interactions, this is the main control signal: the model receives text plus a 2D root path rather than an object-aware physics objective.

This stage is what makes ordinary Kimodo walking usable in arbitrary SAGE rooms. Without it, a valid task path can appear as a wrong root target, a wrong initial facing direction, or a large first-frame pull in the viewer.

3. HSI Sit/Get-Up Binding

Primary code:

  • utils/sage_task_generator/seat_geometry.py
  • utils/task_validator/seat_capability.py
  • utils/kimodo/legacy/pipeline/sit_support.py
  • utils/kimodo/object_keyframe_binding.py

Kimodo does not natively know which part of a chair, sofa, ottoman, stool, or bench is the actual support surface. The pipeline therefore turns a SAGE seat into a bound sitting keyframe.

The current logic is:

  • estimate the physical support surface from the object mesh, using area-supported upward-facing faces instead of the highest visible geometry;
  • annotate the target with a kimodo_sit_capability, separating scene affordance from current motion-library support;
  • select a canonical sit group such as chair_like, sofa_like, bench_like, or low_seat_like;
  • bind candidate canonical sitting poses to the concrete object;
  • keep only candidates whose pelvis/thigh contact patch lies on the support surface, whose foot and hip heights are plausible, and whose body mesh does not penetrate the seat volume too much;
  • add the selected bound pose as a full-body keyframe near the end of sit;
  • hold the seated keyframe across the sit/get-up boundary to prevent collapse or hovering;
  • add upright anchors during get_up so the model does not remain folded or slide out of the seat.

This is the main reason the current pipeline can generalize across different seat types. It does not ask Kimodo to infer a "chair" from text alone; it binds a human pose to the actual support surface of the target object. If no valid mesh-bound candidate exists, static fallback is disabled for data synthesis: the sample is rejected and the failure reason is written to the manifest.

4. HOI Pick/Carry/Place Adapters

Primary code:

  • tools/kimodo/build_kimodo_pick_reach_keyframe_constraint.py
  • tools/kimodo/build_kimodo_place_reach_keyframe_constraint.py
  • utils/kimodo/legacy/pipeline/run_kimodo_task_motion_pipeline.py
  • tools/kimodo/attach_task_object_to_kimodo_motion.py

Kimodo was not trained as a full object-interaction simulator. It can generate plausible human motion from text and constraints, but it does not automatically keep an arbitrary SAGE object in the hands or place it on a target support. The pipeline adds an adapter layer around pick, carry, and place:

  • pick adds reach/contact keyframes near the object, using object geometry and a human stance target rather than relying only on the prompt;
  • carry uses the task path as Root2D conditioning and prompts the model to carry the object, while the object itself is attached by the object adapter;
  • place builds a final reach/place keyframe from the target support surface, so the hand target is near the table/desk/counter rather than at the support object center;
  • after generation, the dynamic-object adapter attaches the object to the carried phase and releases it at the computed support-surface target.

This is not true physical grasp simulation. It is a deterministic scene-aware binding layer that makes Kimodo's human motion and the rendered object trajectory consistent enough for synthetic-data rollout. Fine finger contact and physical force closure remain later quality layers.

5. Push, Pull, and Kick Conditioning

For push, pull, and kick, the current goal is body-level correctness rather than exact object dynamics. The pipeline turns the target object into a reachable stance, action prompt, and local contact cue. The generated motion is then evaluated for root stability, path deviation, foot sliding, and large joint spikes. Object displacement can be added by adapter logic, but the current base contract is that the human approaches the correct target and performs a plausible push/pull/kick action without falling, teleporting, or standing on the object.

6. Generation, Postprocess, and Selection

After all segment prompts and constraints are written, the wrapper calls Kimodo with a fixed seed or a seed list. When multiple seeds are provided, candidate runs are generated without rendering, scored by motion-quality metrics, and the best candidate is rerun with rendering enabled.

The postprocess/evaluation layer records:

  • root step and root height statistics;
  • joint-step spikes;
  • foot sliding;
  • path error against task waypoints;
  • object-motion consistency for dynamic-object tasks;
  • sit-binding candidate validity and failure reasons;
  • video paths, motion paths, and per-scene manifests.

The important design point is that failure is explicit. If the current Kimodo adapter cannot bind a seat, cannot satisfy quality gates, or cannot preserve object consistency, the sample is marked failed or quality_failed rather than silently accepted.

Current Gaps Found During Setup

  • utils.sage_task_generator.vlm_task_generator currently enables pick/place/carry/sit/stand/lie; open/close/pull/push are explicitly disabled in the prompt/action filter. The weekly goal requires push, pull, and kick, so the generator action set must be extended or supplemented by deterministic task templates.
  • IsaacSim topdown capture can finish writing images and still hang during app shutdown on some newly downloaded scenes. Batch runners therefore need process-group timeouts and/or capture caching before large automatic runs.
  • The legacy python -m utils.sage_task_generator entrypoint also runs non-Kimodo baseline translators. It is not the right entrypoint for the Kimodo-only pipeline because unrelated translator failures can invalidate a task run that already produced useful planner artifacts.