跳转至

InteractMove Handoff Result Note

Date: 2026-06-01

Summary

I downloaded the official Hugging Face data zip and Google Drive checkpoint zip, wired them into the upstream submodules/InteractMove layout with symlinks, and ran the official affordance and motion test scripts. The official eval did not reach inference because the public downloads available from the README/Hugging Face page do not include the processed custom/eval files that the released scripts require.

No metrics.txt, generated affordance maps, or generated motion sequences were produced. The SAGE cushion case is therefore not a successful InteractMove generation. A dry-run/proxy input was saved only to document capability and blockers.

Environment

Hard constraints followed:

  • No new conda env was created.
  • No torch, torchvision, torchaudio, CUDA, pytorch3d, or isaac package was installed, upgraded, or downgraded.
  • Tried hoifhli first.

Env attempts:

  • hoifhli: Python 3.8.18, torch 1.11.0, CUDA 11.3. Added missing pure Python deps hydra-core==1.3.2, natsort==8.3.1, smplkit==0.0.3. Could not build pointops: nvcc/host compiler and CUB/header incompatibility. Reusing dexgrasp's compiled pointops_cuda in hoifhli failed with shared-library loading issues, so I did not mix ABI.
  • dexgrasp: Python 3.8.20, torch 1.13.0+cu117, CUDA 11.7. Used as the closest runnable existing env because it already had compiled pointops_cuda. Added non-torch deps only: scipy==1.10.1, ftfy==6.1.1, natsort==8.3.1, smplkit==0.0.3, clip from OpenAI CLIP, easydict==1.10, einops==0.6.1.

Downloads

Data:

  • Hugging Face dataset: cxhcmhhh/InteractMove
  • Zip path: /mnt/data/PhysHSI/data/external/interactmove/downloads/InteractMove.zip
  • Extracted path: /mnt/data/PhysHSI/data/external/interactmove/InteractMove
  • SHA256: 07b42e4b90d8992a706399cde00f7da6bb2bc8ff2867116927894a7fe2c3fb60
  • Extracted contents observed: raw aligned data under done_align/.../{motion.pkl,anno.pkl} plus done_align/generate_log.txt; 4689 anno.pkl and 4689 motion.pkl files.

Checkpoints:

  • Google Drive folder from upstream README: 1w8sLEl7XUJEI1XDO-D44Ls-7rf33wVCs
  • Zip path: /mnt/data/PhysHSI/data/external/interactmove/checkpoints/InteractMove/ckpt.zip
  • Extracted checkpoints:
  • /mnt/data/PhysHSI/data/external/interactmove/checkpoints/CDM-Perceiver-IM/Perceiver.pt
  • /mnt/data/PhysHSI/data/external/interactmove/checkpoints/CMDM-Enc-IM/Enc.pt

Symlinks:

  • /mnt/data/PhysHSI/submodules/InteractMove/data -> /mnt/data/PhysHSI/data/external/interactmove/runtime/data
  • /mnt/data/PhysHSI/submodules/InteractMove/outputs -> /mnt/data/PhysHSI/data/external/interactmove/runtime/outputs
  • /mnt/data/PhysHSI/submodules/InteractMove/body_models/smplx -> /mnt/data/PhysHSI/baseline/interscene/InterScene/body_models/smplx
  • Runtime checkpoint links:
  • /mnt/data/PhysHSI/data/external/interactmove/runtime/outputs/CDM-Perceiver-IM/ckpt/model000000.pt -> /mnt/data/PhysHSI/data/external/interactmove/checkpoints/CDM-Perceiver-IM/Perceiver.pt
  • /mnt/data/PhysHSI/data/external/interactmove/runtime/outputs/CMDM-Enc-IM/ckpt/model000000.pt -> /mnt/data/PhysHSI/data/external/interactmove/checkpoints/CMDM-Enc-IM/Enc.pt

Commands

Download and setup:

wget -c -o output/scene_aware_generation/route2_interactmove/interactmove_eval_20260601/logs/download_interactmove_zip.log -O data/external/interactmove/downloads/InteractMove.zip https://huggingface.co/datasets/cxhcmhhh/InteractMove/resolve/main/InteractMove.zip
gdown --folder --remaining-ok --continue -O data/external/interactmove/checkpoints/ 'https://drive.google.com/drive/folders/1w8sLEl7XUJEI1XDO-D44Ls-7rf33wVCs?usp=sharing'
unzip -q -o data/external/interactmove/downloads/InteractMove.zip -d data/external/interactmove/InteractMove
unzip -o data/external/interactmove/checkpoints/InteractMove/ckpt.zip -d data/external/interactmove/checkpoints

Official scripts:

cd /mnt/data/PhysHSI/submodules/InteractMove
CUDA_VISIBLE_DEVICES=1 conda run -n dexgrasp bash scripts/affordance/test.sh outputs/CDM-Perceiver-IM 2023
CUDA_VISIBLE_DEVICES=1 conda run -n dexgrasp bash scripts/motion/test.sh outputs/CMDM-Enc-IM outputs/CDM-Perceiver-IM/eval/test-0601-144303 2023

Official Eval Result

Affordance script status: failed before inference.

Evidence:

  • Initial run failed on missing ./body_models/smplx; fixed by symlink.
  • Retry failed because dexgrasp had old einops without einops.einsum; fixed by installing einops==0.6.1.
  • As-is official script then failed because ./data/custom/anno.csv was absent. The downloaded official HF zip does not contain data/custom.
  • A native custom dependency probe using real InteractMove text raise the desk and local ScanNet scene0008_00 then failed before model construction because processed statistics/contact files are absent:
  • ./data/Mean_Std_Cont_HumanML3D_HUMANISE_PROX_InteractMove_contact_cont_joints_0.8_fur.npz
  • ./data/custom/contact_motion/contacts/00000.npz

Motion script status: failed before inference.

Evidence:

  • Because affordance did not produce pred_contact, motion had no valid affordance output.
  • The motion script failed while preparing dataset statistics:
  • ./data/Mean_Std_CM_HumanML3D_HUMANISE_PROX_InteractMove_pos.npz
  • Then ContactMotionCustomDataset hit ValueError: too many values to unpack (expected 5) while falling back to recomputing stats from custom all-data tuples.

Additional dependency gap not yet reached by a clean official run:

  • ./data/POINTTRANS_C_N8192_E300/model.pth is referenced by configs/model/cdm.yaml and is not in the HF data zip, checkpoint zip, or local search results. The CDM checkpoint does not contain scene model weights, so this pretrained point transformer appears to be another required official artifact.

Metrics:

  • No metrics file was generated.
  • Checked output directories under /mnt/data/PhysHSI/data/external/interactmove/runtime/outputs; only failed-run test.log files exist.

SAGE Cushion Support

SAGE cushion mesh:

/mnt/data/PhysHSI/data/sage/generated_tasks/layout_84b703fb/tokenhsi/task_plans/sage_auto_layout_84b703fb/objects/cushion/room_86161fe1_cushion_3f35ffae/geom/mesh.obj

Capability from code inspection:

  • Arbitrary custom scene: partially supported as data/custom/points/{i:04d}.npz plus data/custom/scenes/{scene_id}.ply.
  • Arbitrary custom object mesh: not supported by the eval/generation path. The custom datasets never read object mesh paths.
  • Explicit target object/path/goal: not supported. ContactMapCustomDataset reads only scene id, scene translation, utterance, and point cloud.
  • Text instruction: supported through utterance.
  • Human root path control: not supported.
  • Object trajectory output: not supported by the released eval code.
  • SMPL-X output: motion evaluator can save joints/{index}.pkl with joints and SMPL-X params after successful motion generation.
  • Finger output: not available; saved params are body-level, not explicit hand/finger results.

SAGE/proxy dry-run artifacts:

  • /mnt/data/PhysHSI/output/scene_aware_generation/route2_interactmove/sage_cushion_proxy_probe_20260601/interactmove_input.json
  • /mnt/data/PhysHSI/output/scene_aware_generation/route2_interactmove/sage_cushion_proxy_probe_20260601/proxy_mapping.json
  • /mnt/data/PhysHSI/output/scene_aware_generation/route2_interactmove/sage_cushion_proxy_probe_20260601/path_xy.npy

SAGE cushion result: failed/not run as a true InteractMove-conditioned generation. Given the released code, InteractMove can only be treated as a weak text+point-cloud native/custom baseline, not a SAGE-conditioned generator with cushion mesh, root path, pickup/place targets, or object trajectory.

Logs And Preview

Logs:

  • /mnt/data/PhysHSI/output/scene_aware_generation/route2_interactmove/interactmove_eval_20260601/logs/download_interactmove_zip.log
  • /mnt/data/PhysHSI/output/scene_aware_generation/route2_interactmove/interactmove_eval_20260601/logs/download_checkpoints_gdown.log
  • /mnt/data/PhysHSI/output/scene_aware_generation/route2_interactmove/interactmove_eval_20260601/logs/official_affordance_test.log
  • /mnt/data/PhysHSI/output/scene_aware_generation/route2_interactmove/interactmove_eval_20260601/logs/official_affordance_test_retry_body_models.log
  • /mnt/data/PhysHSI/output/scene_aware_generation/route2_interactmove/interactmove_eval_20260601/logs/official_affordance_test_retry_einops.log
  • /mnt/data/PhysHSI/output/scene_aware_generation/route2_interactmove/interactmove_eval_20260601/logs/official_affordance_test_native_custom_probe.log
  • /mnt/data/PhysHSI/output/scene_aware_generation/route2_interactmove/interactmove_eval_20260601/logs/official_motion_test_after_affordance_failure.log

Preview:

  • /mnt/data/PhysHSI/output/scene_aware_generation/route2_interactmove/sage_cushion_proxy_probe_20260601/native_probe_input_topdown.png
  • This is an input/proxy diagnostic only, not a generated InteractMove result.

Concrete Blocker

The public artifacts downloaded from the official links are insufficient for the released scripts/affordance/test.sh and scripts/motion/test.sh to run to inference. The missing required artifacts are processed eval/custom files and statistics, not just raw aligned motions:

  • data/custom/anno.csv
  • data/custom/points/*.npz
  • data/custom/contact_motion/contacts/*.npz
  • data/Mean_Std_Cont_HumanML3D_HUMANISE_PROX_InteractMove_contact_cont_joints_0.8_fur.npz
  • data/Mean_Std_CM_HumanML3D_HUMANISE_PROX_InteractMove_pos.npz
  • likely data/POINTTRANS_C_N8192_E300/model.pth

The raw HF data also references ScanNet scene ids and the README says ScanNet V2, GRAB, and BEHAVE meshes are required. The current machine has a small ScanNet subset and some GRAB-related data, but not the full official processed InteractMove eval/custom package needed by the released scripts.