generate_dataset Fails After Successful Teleoperation and Annotation With Custom Robot Model

[Yuchili2022]

I encountered some issues while working on Teleoperation and Imitation Learning with Isaac Lab Mimic, and I would like to discuss possible improvements.

1. Custom robot model configuration
   Since I am using my own robot model, I modified the following environment configurations to match my setup:
   • stack_joint_pos_env_cfg
   • stack_ik_real_env_cfg
   • stack_ik_real_mimic_env
   • stack_ik_real_mimic_env_cfg
   Additionally, because my gripper has been replaced, I also updated the "observation" settings in the stack task accordingly.

2. Teleoperation and demonstration recording are working correctly
   The following steps can all be completed successfully:
   • I can run teleop_se3_agent.py and the robot performs correct Cartesian motions at the TCP, including both translation and rotation.
   • record_demos works and I can collect 10 successful demonstrations that satisfy the success conditions.
   • annotate_demos also runs without issues.

3. The issue occurs when running generate_dataset
   When I run generate_dataset, it always results in failures.
   Based on the documentation, high quality demonstrations should follow these principles:
   • Keep demonstrations short
   • Use direct motion paths
   • Avoid pauses and maintain smooth, continuous movement

Based on this, I suspect the following issues in my demonstrations:
a. Keyboard input makes smooth motions difficult
Using the keyboard often results in discrete, non smooth movements or unintended pauses.
b. I need to open two fixed views (front + right) to perform accurate demonstrations
Currently, I open both a front view and a right view simultaneously to better align with objects.
Although this helps with accuracy, it complicates the teleoperation process and may reduce overall smoothness.

Questions I would like to ask:

1. How well does the SpaceMouse work for Mimic demonstrations?
   Does it significantly improve the smoothness and quality of the teleoperation process?
2. The official documentation also mentions CloudXR Teleoperation.
   Would you recommend directly using CloudXR for this type of task?
3. Are there recommended camera/view configurations in Isaac Lab Mimic that can reduce visual difficulties during demonstrations?

Thank you!

[GioiaZheng]

I would separate demonstration quality from the custom Mimic integration first. Since recording and annotation both work but every generated rollout fails, check the Mimic-specific contract before changing teleoperation devices: the object_ref names in subtask_configs, the 0 -> 1 edges of each subtask termination signal, and whether get_robot_eef_pose, action_to_target_eef_pose, and target_eef_pose_to_action all use the same frame and action scale. I would also verify the replacement gripper mapping independently. A successful recorded demo proves that the original actions execute, but not that those actions can be converted and recombined correctly by Mimic.

A useful diagnostic is one generation trial with action noise disabled, subtask offsets set to (0, 0), and --pause_subtask enabled. Watch where the first divergence appears. If it consistently starts at the same subtask boundary, the likely cause is the signal/object/frame configuration. If this minimal case succeeds and failures return only after offsets, interpolation, or noise are restored, then demonstration coverage and smoothness become the next suspects.

A SpaceMouse can make collection easier, but it will not fix a frame or subtask-boundary mismatch. The two fixed camera views are also fine if they help the operator; they do not directly affect state-based generation. I would move to CloudXR only if the interaction itself requires it, after the minimal generation case works.