跳转至

Pure HOI-FHLI Path-Only Audit

Date: 2026-05-31

Scope

This audit focuses on the pure HOI-FHLI path-only route for the SAGE exact06 cushion pick/carry/place case.

No generator code is changed here. The goal is to compare our adapter/runtime inputs with the official HOI-FHLI long-sequence implementation and identify why previous SAGE-connected outputs looked worse than the official demo.

Confirmed Official Flow

Official sampling structure is:

path_data_list[p][0] = navigation path, interpreted as human root waypoints
path_data_list[p][1] = interaction path, interpreted as object COM waypoints
object_names_list[p] = ["", object_name]
action_names_list[p] = ["walking", "lift" | "push" | "pull"]
table_height_list[p][1] = final support height above the floor

Relevant code:

submodules/hoifhli_release/sample.py
submodules/hoifhli_release/manip/utils/trainer_utils.py
submodules/hoifhli_release/trainer_interaction_motion_diffusion.py

load_planned_path_as_waypoints() densifies interaction XY waypoints at roughly 0.6-0.8m spacing, duplicates the start/end waypoints, and then sets all interaction z values to the first z except the final waypoint. This is official behavior.

The interaction model then ignores most intermediate planned z values. For lift, intermediate object z conditions are sampled from an object/action heuristic, e.g. largebox lift uses about 0.8-0.9m, while the final waypoint z is preserved for putdown.

So low-to-high placement is supported by:

start object COM z = object center height on floor
intermediate object COM z = lift heuristic
final object COM z = table/support z + object center height on floor
table_height = support z

This means the earlier z reset itself is not a bug.

Current Adapter Behavior

The SAGE adapter has three path policies:

hoifhli_endpoint_control: object path = [pickup COM, place COM]
stage1_full_path: object path = pickup + SAGE carry path + place
root_offset_object_com: convert SAGE root path to object COM path by interpolating pickup/place root-to-object offsets

hoifhli_endpoint_control intentionally drops the SAGE A* route and gives HOI-FHLI a straight object path. This explains outputs where the actor/object walks through obstacles. It should only be treated as an endpoint-control sanity check, not the scene-aware route.

For exact06:

endpoint object path length: 3.393m
stage1 full path length: 9.329m
root_offset object COM path length: 7.252m
SAGE root path length: 7.880m raw, 7.526m resampled

The current default root_offset_object_com is the right starting point for pure HOI-FHLI, but it is still only a geometric conversion rule, not yet a proven final route.

Why Middle Short Clips Look Bad

The diagnostic clip_mode=middle is not a true "already carrying" clip. It rewrites the middle slice as a new standalone Lift the largebox, move the largebox, and put down the largebox interaction with floor start/end support.

That forces HOI-FHLI to perform a full pick and putdown inside the middle slice, so repeated crouching in this diagnostic is expected and should not be used to judge full-route carry quality.

For place-only clips, the newer handoff-style input sets:

pickup support z = original place support z
place support z = original place support z
interaction_z_policy = linear_pickup_to_place

That is closer to an already-carried handoff into a place motion.

Evidence From Existing Outputs

Official clean run:

submodules/hoifhli_release/results/interaction/moving_box/.../human_object_results.pkl
frames: 245
object XY path: 2.644m
object z range: 0.175m to 1.013m
wrist-object min/median/max: 0.228 / 0.228 / 1.375m
near wrist-object < 0.4m: 208 / 245 frames

SAGE endpoint run:

output/hoifhli_clean_sage_cushion_endpoint/human_object_results.pkl
frames: 274
object XY path: 3.511m
object z range: 0.175m to 1.032m
wrist-object min/median/max: 0.258 / 0.268 / 0.804m
near wrist-object < 0.4m: 239 / 274 frames

SAGE pickup short fast run:

output/hoifhli_clean_sage_cushion_root_offset_pickup_short_fast/human_object_results.pkl
frames: 213
object XY path: 1.411m
object z range: 0.175m to 0.967m
wrist-object min/median/max: 0.151 / 0.151 / 0.834m
near wrist-object < 0.4m: 152 / 213 frames
grasp_iter_override: 300

These numbers do not show a complete lack of contact. The visual failure is more likely from path distribution, initial/final orientation, proxy mesh mismatch, short-clip semantics, or fast grasp settings.

Main Risk List

  1. hoifhli_endpoint_control is not scene-aware. It bypasses A* and should not be used for the real SAGE case.
  2. stage1_full_path treats SAGE human-root waypoints as object COM waypoints. This is not semantically correct unless the SAGE path is already an object path.
  3. root_offset_object_com is plausible but unproven. It linearly interpolates root-to-object offsets along the route, so the carried object may deviate from the collision-free root path.
  4. Exact06 navigation end to object start distance is about 0.625m. Official random object approach uses about 0.25-0.5m for the first object offset, so this may be slightly out of distribution.
  5. Some "fast" outputs used grasp_iter_override=300. Official default is 5000, so these are not valid final-quality hand/finger tests.
  6. SAGE cushion is generated as HOI-FHLI largebox. Official visualizer shows the largebox proxy; full-scene rendering may replace or overlay a different SAGE mesh, which can make hand-object contact look wrong even if the proxy contact is reasonable.
  7. The local sample.py change only injects scenario_data and returns output paths. It should not change official behavior when scenario_data=None, but a final audit should also run once from an unmodified checkout for parity.

Fix Direction

The next pure HOI-FHLI experiment should use:

path_policy = root_offset_object_com
clip_mode = full
grasp_iter = official default, no fast override
object_name = largebox first, then object-size-specific proxy variants
official visualizer first, full-scene renderer second

Before running the expensive full route, add dry diagnostics that report:

navigation end -> object start distance
object COM path vs SAGE root path topdown
object path clearance against scene obstacles
root/object offset range
planned waypoint count after HOI-FHLI densification
expected frame count = (planned_waypoint_count - 1) * 30 before trimming

The acceptance order should be:

  1. Official clean sample from unchanged code still looks good.
  2. Injected official-style low-to-high two-point scenario looks good.
  3. SAGE root-offset full path looks acceptable in official visualizer.
  4. The same result remains acceptable after conversion into the full-scene renderer.

Until step 3 passes with default grasp optimization, I do not have factual 100% confidence in the current SAGE pure-HOI-FHLI path adapter.

2026-05-31 GPU Run

The full root-offset/default-grasp validation was rerun after GPU access became available.

Command:

CUDA_VISIBLE_DEVICES=0 /home/user/anaconda3/envs/hoifhli/bin/python -m tools.hoifhli.legacy.generate \
  --adapter-input output/hsi_1hour_smplx_toy_exact06/hoifhli_adapter_root_offset_full/hoifhli_input.json \
  --out-dir output/hoifhli_pure_sage_cushion_root_offset_full_default \
  --device 0

Result:

pre_generation_alignment_pass: True
gpu_generation_run: True
path_policy: root_offset_object_com
clip_mode: full
grasp_iter_override: null
frames in exported scene motion: 575
contact window: 41..509
object displacement: 3.393m
object path length: 8.183m
max frame-to-frame object jump: 0.079m
human root pickup/place XY error: 0.991m / 0.915m

Outputs:

output/hoifhli_pure_sage_cushion_root_offset_full_default/metrics.json
output/hoifhli_pure_sage_cushion_root_offset_full_default/human_object_results.pkl
output/hoifhli_pure_sage_cushion_root_offset_full_default/hoifhli_scene_motion.npz
submodules/hoifhli_release/visualizer_results/output_hoifhli_pure_sage_cushion_root_offset_full_default/0_1/output_0.mp4
output/hoifhli_pure_sage_cushion_root_offset_full_default/scene_preview.mp4
output/hoifhli_pure_sage_cushion_root_offset_full_default/topdown_overlay.png

Visual read:

  1. Official HOI-FHLI visualizer is now materially better than the earlier broken hybrid/short-clip outputs. The motion is continuous, the character bends for pickup, carries the object, and does not fragment at transitions.
  2. The full-scene renderer confirms the exported sequence uses one SMPL-X face topology and renders the dynamic SAGE cushion mesh, not only the largebox proxy.
  3. The top-down overlay still shows the core unresolved issue: the generated object COM path follows the converted root-offset object route, but that route runs very close to and partly through furniture footprint regions. This is not a diffusion failure; it is a path semantics/conversion failure. The source A* path is a human-root path, while HOI-FHLI controls object COM.
  4. Because human_root_pickup_xy_error_m and human_root_place_xy_error_m are around 0.9-1.0m, the generated human is offset from the object COM as expected for carrying. That offset is also why directly reusing an A* root path as object COM, or linearly deriving object COM from it without clearance checks, is not guaranteed collision-free.

Current confidence:

HOI-FHLI clean environment and official full generation: high confidence
SAGE pure-HOI-FHLI root-offset adapter as final scene-aware solution: not 100%
Main remaining blocker: object-COM path generation must be obstacle-aware after applying human-object carrying offset

The next fix should not change HOI-FHLI diffusion internals. It should replace root_offset_object_com with a verified object-COM planner: derive object clearance from the dynamic object's footprint and the human carry offset, then plan or optimize the object COM path directly in the SAGE traversability map.

2026-05-31 Object-COM A* Full Run

I added and validated a pure HOI-FHLI object_com_astar adapter mode. This mode rebuilds the SAGE navigation occupancy map, ignores only the carried cushion as a static obstacle, and plans the HOI-FHLI interaction_path_xy directly in object-COM space instead of deriving it from the SAGE human-root A* path.

Commands:

/home/user/anaconda3/envs/isaac-sim/bin/python -m utils.sage_task_generator.adapters.hoifhli \
  --run-dir utils/sage_task_generator/real_vlm_runs/20260424_real_vlm_exact06_qwen3vlplus \
  --out-dir output/hsi_1hour_smplx_toy_exact06/hoifhli_adapter_object_com_astar_full \
  --repo-root /mnt/data/PhysHSI \
  --path-policy object_com_astar \
  --target-spacing-m 0.7

CUDA_VISIBLE_DEVICES=0 /home/user/anaconda3/envs/hoifhli/bin/python -m tools.hoifhli.legacy.generate \
  --adapter-input output/hsi_1hour_smplx_toy_exact06/hoifhli_adapter_object_com_astar_full/hoifhli_input.json \
  --out-dir output/hoifhli_pure_sage_cushion_object_com_astar_full_default \
  --device 0

Result:

pre_generation_alignment_pass: True
gpu_generation_run: True
path_policy: object_com_astar
frames in exported scene motion: 1054
object displacement: 3.393m
object path length: 12.126m in metrics, 9.603m in SAGE XY occupancy check
max frame-to-frame object jump: 0.046m
human root pickup/place XY error: 0.485m / 0.926m
official video frames: 1306 at 30 fps

Path occupancy against the rebuilt SAGE map, with the carried cushion ignored:

stage1_human_root_astar:        0 / 272 occupied samples = 0.000
stage1_carry_object_com_source: 14 / 334 occupied samples = 0.0419
planned_object_com_astar:       14 / 334 occupied samples = 0.0419
generated_object_com:           127 / 1055 occupied samples = 0.1204
generated_human_root:           291 / 1055 occupied samples = 0.2758

Important detail: the raw A* route itself is collision-free. plan_path moves the occupied place goal to a nearest free endpoint:

pickup occ: False, clearance 0.10m
place occ: True, clearance 0.00m
raw A* last point: [4.0, 2.3]
exact place point: [3.58266498, 2.28888330]
raw A* occupancy: 0 / 319
exact-place-preserved occupancy: 14 / 334

So the remaining planned-path occupancy is not the old "route through furniture" bug. It is caused by preserving the exact final placement point, which lies inside the support surface footprint in the 2D occupancy map. The object has to end on that support surface, but the free-space carry path should be judged separately from the final place-on-support action.

Generated-object occupied frames cluster as:

frames 796..799: 4 frames, around [4.30, 3.72], far from place
frames 888..923: 36 frames, approach to support, 0.53-0.76m from place
frames 967..1053: 87 frames, final placement/resting on support

The first cluster is real residual drift from HOI-FHLI, not the adapter planner. The latter two clusters are dominated by the approach/final support footprint and need a task-aware occupancy metric that allows the target support during placement.

Hand/grasp status:

right hand grasp: succ, energy 1.0957
left hand grasp: fail, energy 80.6829

This matters for visual quality: if the arms or wrist look bad in this run, the first concrete culprit is the failed HOI-FHLI/DexGrasp left-hand optimization for the largebox proxy, not the SAGE path adapter.

Outputs:

output/hoifhli_pure_sage_cushion_object_com_astar_full_default/metrics.json
output/hoifhli_pure_sage_cushion_object_com_astar_full_default/path_occupancy_metrics.json
output/hoifhli_pure_sage_cushion_object_com_astar_full_default/hoifhli_scene_motion.npz
output/hoifhli_pure_sage_cushion_object_com_astar_full_default/topdown_overlay.png
output/hoifhli_pure_sage_cushion_object_com_astar_full_default/occupied_samples_overlay.png
output/hoifhli_pure_sage_cushion_object_com_astar_full_default/scene_preview.mp4
output/hoifhli_pure_sage_cushion_object_com_astar_full_default/scene_final_frame.png
submodules/hoifhli_release/visualizer_results/output_hoifhli_pure_sage_cushion_object_com_astar_full_default/0_1/output_0.mp4

Current confidence:

Root-offset path conversion was a real bug: high confidence.
Direct object-COM A* adapter fixes that bug: high confidence.
The full generated motion is usable as a pure HOI-FHLI baseline: medium confidence.
It is not yet final dataset quality: left-hand grasp failed, and final support occupancy needs task-aware evaluation.

Resampled Object-COM A* Follow-Up

Run:

output/hoifhli_pure_sage_cushion_object_com_astar_resampled058_full_default

Adapter:

output/hsi_1hour_smplx_toy_exact06/hoifhli_adapter_object_com_astar_resampled058_full/hoifhli_input.json

Changes:

object-COM A* path resampled to target spacing 0.58m
input interaction waypoints: 17
raw A* points: 34
object-COM path length: 9.074m

Generation result:

pre_generation_alignment_pass: True
gpu_generation_run: True
official video frames: 856 at 30 fps
exported interaction scene motion frames: 604 at 30 fps
contact window in exported interaction: 33..596
object path length in generated SAGE XY: 9.320m
object mean XY speed: 0.463m/s
human root mean XY speed: 0.388m/s
max frame-to-frame object jump: 0.062m

This fixed only part of the speed issue. The previous full object-COM A* run was roughly 0.27m/s; this run is roughly 0.46m/s for the object. It is still slow for normal walking because HOI-FHLI interaction loading densifies waypoints to about 0.6-0.8m spacing and then assigns one second per waypoint interval.

Visual result:

submodules/hoifhli_release/visualizer_results/output_hoifhli_pure_sage_cushion_object_com_astar_resampled058_full_default/0_1/output_0.mp4
output/hoifhli_pure_sage_cushion_object_com_astar_resampled058_full_default/scene_preview.mp4
output/hoifhli_pure_sage_cushion_object_com_astar_resampled058_full_default/frames/official_contact_sheet.jpg
output/hoifhli_pure_sage_cushion_object_com_astar_resampled058_full_default/frames/scene_contact_sheet.jpg

The carry posture is still not acceptable: during the middle of the carry segment the person remains visibly crouched/forward-leaning while holding the object high in front of the torso. This is visible in the raw HOI-FHLI official renderer, so it is not caused by the SAGE scene viewer.

Most likely causes:

1. The SAGE object is room_86161fe1_cushion_3f35ffae, but the HOI-FHLI object proxy is still largebox.
2. SAGE carry is mapped to HOI-FHLI native action lift because the released model supports lift/push/pull, not a separate carry action.
3. The whole 9m route is one long lift interaction. The model keeps contact from frame 33 to 596 and behaves like it is holding a lifted box, not like a natural long-distance carry/walk controller.
4. HOI-FHLI's interaction waypoint loader sets all intermediate vertical waypoints to the start floor height, and the interaction trainer then uses the hard-coded lift height heuristic. The user-provided path does not directly control intermediate carry height.

Current confidence after this run:

The original waypoint density bug was real: high confidence.
Resampling A* waypoints is necessary but not sufficient: high confidence.
The remaining posture problem is not a viewer/camera issue: high confidence.
Pure HOI-FHLI with one long lift segment is unlikely to produce natural long-distance upright carry without additional control or decomposition: medium-high confidence.
The next minimal diagnostic should test a better object proxy such as plasticbox/smallbox and/or a lower carry-height heuristic on a short middle clip before rerunning the full scene.