[pull] develop from isaac-sim:develop

.github/workflows/build.yaml

@@ -800,8 +800,15 @@ jobs:
         isaacsim-version: ${{ needs.config.outputs.isaacsim_image_tag }}
         dockerfile-path: docker/Dockerfile.curobo
         cache-tag: cache-curobo
-        include-files: "test_contrib_environments.py"
+        include-files: >-
+          test_contrib_environments_kitless.py,
+          test_contrib_environments_kit.py,
+          test_contrib_environments_kit_cameras.py
         warp-cache: restore
+        ovrtx-shader-cache: restore
+        ovrtx-shader-cache-trees: kit
+        # The three files run side by side; the kit and kitless files split across two workers each.
+        test-jobs: "5"
         container-name: isaac-lab-contrib-environments-test
 
   test-record-video:

CONTRIBUTORS.md

@@ -218,6 +218,7 @@ Guidelines for modifications:
 * Xiaodi Yuan
 * Xinjie Yao
 * Xinpeng Liu
+* Xin Xu
 * Xu Li
 * Yang Jin
 * Yanzi Zhu

docs/source/features/isaac_teleop.rst

@@ -51,6 +51,11 @@ input modes, which determine which retargeters and control schemes are available
      - Isaac Teleop plugin (bundled)
      - Migrated from the now-deprecated ``isaac-teleop-device-plugins`` repo.
        Combine with an external wrist-tracking source for wrist positioning. See :ref:`manus-vive-handtracking`.
+   * - Haptikos Exoskeletons
+     - Exoskeleton hand tracking with controller wrist poses
+     - Isaac Teleop plugin (separate executable)
+     - Requires the Haptikos App, exoskeletons, and an OpenXR headset with controllers.
+       See :ref:`haptikos-quest-handtracking`.
 
 
 .. _isaac-teleop-control-schemes:
@@ -86,7 +91,7 @@ starting point, then see the detailed pipeline examples below.
      - 28
      - ``fixed_base_upper_body_ik_g1_env_cfg.py``
    * - Complex dex hand (e.g. GR1T2, G1 Inspire)
-     - Hand tracking / Manus gloves
+     - Hand tracking / Manus gloves / Haptikos exoskeletons
      - Bimanual ``Se3AbsRetargeter`` + ``DexBiManualRetargeter``
      - 36+
      - ``pickplace_gr1t2_env_cfg.py``
@@ -1769,9 +1774,9 @@ There are two levels of device integration:
 
 **Isaac Teleop plugin (C++ level)**
    For new hardware that requires a custom driver or SDK. Plugins push data via OpenXR tensor
-   collections. Existing plugins include Manus gloves, OAK-D camera, controller synthetic hands,
-   and foot pedals. After creating the plugin, update the retargeting pipeline config to consume
-   data from the new plugin's source node.
+   collections. Existing plugins include Manus gloves, Haptikos exoskeletons, OAK-D camera,
+   controller synthetic hands, and foot pedals. After creating the plugin, update the retargeting
+   pipeline config to consume data from the new plugin's source node.
 
    See the `Plugins directory <https://github.com/NVIDIA/IsaacTeleop/tree/main/src/plugins/>`_ for examples.
 

docs/source/how-to/cloudxr_teleoperation.rst

@@ -803,6 +803,72 @@ Start teleoperation
 Move your hands and the simulated follower will mirror the glove-tracked finger joints in real
 time.
 
+.. _haptikos-quest-handtracking:
+
+Haptikos Exoskeletons with Quest
+--------------------------------
+
+The `Haptikos plugin <https://github.com/NVIDIA/IsaacTeleop/tree/release/1.4.x/src/plugins/haptikos>`_
+combines controller wrist poses with exoskeleton finger tracking from the Haptikos Core App and
+pushes hand joints into the OpenXR runtime. Isaac Lab receives them through Isaac Teleop's
+standard hand-tracking input, so no Haptikos-specific Isaac Lab device is needed. The plugin
+supports Linux and has been tested with Meta Quest headsets; other headsets with controllers may
+also work.
+
+Build the plugin
+^^^^^^^^^^^^^^^^
+
+The Haptikos plugin is built from Isaac Teleop source; it is not included in Isaac Lab's
+``teleop`` extra. Check out the release branch matching Isaac Lab's ``isaacteleop`` pin (currently
+``1.4.x``), obtain the `Haptikos Robotics API <https://github.com/Haptikostech/HaptikosAPI>`_,
+and copy its ``HaptikosCpp_API_Shared`` directory into ``src/plugins/haptikos``. The C++ API is
+required to build the tracking plugin, even if you do not use haptic feedback. Follow the
+`plugin's setup instructions <https://github.com/NVIDIA/IsaacTeleop/tree/release/1.4.x/src/plugins/haptikos>`_
+for Haptikos account and licensing requirements.
+
+.. code-block:: bash
+
+   git clone https://github.com/NVIDIA/IsaacTeleop.git
+   cd IsaacTeleop
+   git checkout release/1.4.x
+   # Copy HaptikosCpp_API_Shared to src/plugins/haptikos before building.
+   cmake -S . -B build -DENABLE_CLANG_FORMAT_CHECK=OFF
+   cmake --build build --target haptikos_hands_plugin --parallel 4
+
+Run Isaac Lab and the plugin
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Attach a controller to each exoskeleton using the included mount. Calibrate the exoskeleton
+forward direction against the headset, then keep the Haptikos Core App, exoskeletons, and
+controllers active. Use the hand-tracking task below.
+
+Haptikos uses an external OpenXR push device. The shipped CloudXR profiles disable push devices,
+so enable them in a custom profile before launching Isaac Lab:
+
+.. code-block:: bash
+
+   cp $(uv run --extra teleop,isaacsim python -c \
+       "from isaaclab_teleop import CLOUDXR_JS_ENV; print(CLOUDXR_JS_ENV)") ~/haptikos.env
+   sed -i 's/NV_CXR_ENABLE_PUSH_DEVICES=0/NV_CXR_ENABLE_PUSH_DEVICES=1/' ~/haptikos.env
+
+   uv run --extra teleop,isaacsim isaaclab teleop run \
+       --task IsaacContrib-PickPlace-GR1T2-WaistEnabled-Abs \
+       --visualizer kit --xr --cloudxr_env ~/haptikos.env
+
+Once CloudXR is waiting for a connection, open a separate terminal and start the plugin with
+the runtime environment created by Isaac Lab:
+
+.. code-block:: bash
+
+   cd /path/to/IsaacTeleop
+   source ~/.cloudxr/run/cloudxr.env
+   ./build/src/plugins/haptikos/haptikos_hands_plugin
+
+Connect the Quest using :ref:`the Quest/Pico connection steps <connect-quest-pico>`, then start
+teleoperation from the headset.
+
+See :ref:`isaac-teleop-cloudxr-profiles` for more on custom profiles.
+
 
 Run with Docker
 ---------------

source/isaaclab/changelog.d/camera-perf-01-proxyarray-dtype.rst

@@ -0,0 +1,6 @@
+Fixed
+^^^^^
+
+* Fixed :func:`~isaaclab.envs.mdp.observations.image` passing the sensor's ``ProxyArray`` to the
+  image normalization, which left colorized semantic segmentation unscaled in ``[0, 255]`` and
+  skipped the fused normalization kernel.

source/isaaclab/changelog.d/core-cleanup-string-matchers.skip

@@ -0,0 +1 @@
+Unified the regex name matchers in isaaclab.utils.string; behavior is unchanged.

source/isaaclab/changelog.d/height-field-generator-origins.rst

@@ -0,0 +1,8 @@
+Fixed
+^^^^^
+
+* **Breaking:** Fixed height-field terrain origins to use the location chosen by each generator. Inverted pyramid
+  slopes now place the origin on the center platform for any platform width. Height-field generator
+  functions wrapped with :func:`~isaaclab.terrains.height_field.utils.height_field_to_mesh` now return
+  ``(height_field, origin)``; custom generators must return their local origin [m] as a three-element
+  array alongside the discretized height field.

source/isaaclab/isaaclab/envs/mdp/observations.py

@@ -395,7 +395,7 @@ def image(
         The images produced at the last time-step
     """
     sensor: Camera | RayCasterCamera = env.scene.sensors[sensor_cfg.name]
-    images = sensor.data.output[data_type]
+    images = sensor.data.output[data_type].torch
     # depth image conversion
     if (data_type == "distance_to_camera") and convert_perspective_to_orthogonal:
         images = math_utils.orthogonalize_perspective_depth(images, sensor.data.intrinsic_matrices)

source/isaaclab/isaaclab/terrains/height_field/hf_terrains.py

@@ -19,7 +19,9 @@
 
 
 @height_field_to_mesh
-def random_uniform_terrain(difficulty: float, cfg: hf_terrains_cfg.HfRandomUniformTerrainCfg) -> np.ndarray:
+def random_uniform_terrain(
+    difficulty: float, cfg: hf_terrains_cfg.HfRandomUniformTerrainCfg
+) -> tuple[np.ndarray, np.ndarray]:
     """Generate a terrain with height sampled uniformly from a specified range.
 
     .. image:: ../../_static/terrains/height_field/random_uniform_terrain.jpg
@@ -34,9 +36,7 @@ def random_uniform_terrain(difficulty: float, cfg: hf_terrains_cfg.HfRandomUnifo
         cfg: The configuration for the terrain.
 
     Returns:
-        The height field of the terrain as a 2D numpy array with discretized heights.
-        The shape of the array is (width, length), where width and length are the number of points
-        along the x and y axis, respectively.
+        The discretized height field with shape (width, length) and its origin [m] with shape (3,).
 
     Raises:
         ValueError: When the downsampled scale is smaller than the horizontal scale.
@@ -77,11 +77,14 @@ def random_uniform_terrain(difficulty: float, cfg: hf_terrains_cfg.HfRandomUnifo
     y_upsampled = np.linspace(0, cfg.size[1] * cfg.horizontal_scale, length_pixels)
     z_upsampled = func(x_upsampled, y_upsampled)
     # round off the interpolated heights to the nearest vertical step
-    return np.rint(z_upsampled).astype(np.int16)
+    height_field = np.rint(z_upsampled).astype(np.int16)
+    return height_field, _terrain_origin(height_field, cfg)
 
 
 @height_field_to_mesh
-def pyramid_sloped_terrain(difficulty: float, cfg: hf_terrains_cfg.HfPyramidSlopedTerrainCfg) -> np.ndarray:
+def pyramid_sloped_terrain(
+    difficulty: float, cfg: hf_terrains_cfg.HfPyramidSlopedTerrainCfg
+) -> tuple[np.ndarray, np.ndarray]:
     """Generate a terrain with a truncated pyramid structure.
 
     The terrain is a pyramid-shaped sloped surface with a slope of :obj:`slope` that trims into a flat platform
@@ -102,9 +105,7 @@ def pyramid_sloped_terrain(difficulty: float, cfg: hf_terrains_cfg.HfPyramidSlop
         cfg: The configuration for the terrain.
 
     Returns:
-        The height field of the terrain as a 2D numpy array with discretized heights.
-        The shape of the array is (width, length), where width and length are the number of points
-        along the x and y axis, respectively.
+        The discretized height field with shape (width, length) and its origin [m] with shape (3,).
     """
     # resolve terrain configuration
     if cfg.inverted:
@@ -146,11 +147,14 @@ def pyramid_sloped_terrain(difficulty: float, cfg: hf_terrains_cfg.HfPyramidSlop
     hf_raw = np.clip(hf_raw, min(0, z_pf), max(0, z_pf))
 
     # round off the heights to the nearest vertical step
-    return np.rint(hf_raw).astype(np.int16)
+    height_field = np.rint(hf_raw).astype(np.int16)
+    return height_field, _terrain_origin(height_field, cfg, height_field[center_x, center_y])
 
 
 @height_field_to_mesh
-def pyramid_stairs_terrain(difficulty: float, cfg: hf_terrains_cfg.HfPyramidStairsTerrainCfg) -> np.ndarray:
+def pyramid_stairs_terrain(
+    difficulty: float, cfg: hf_terrains_cfg.HfPyramidStairsTerrainCfg
+) -> tuple[np.ndarray, np.ndarray]:
     """Generate a terrain with a pyramid stair pattern.
 
     The terrain is a pyramid stair pattern which trims to a flat platform at the center of the terrain.
@@ -169,9 +173,7 @@ def pyramid_stairs_terrain(difficulty: float, cfg: hf_terrains_cfg.HfPyramidStai
         cfg: The configuration for the terrain.
 
     Returns:
-        The height field of the terrain as a 2D numpy array with discretized heights.
-        The shape of the array is (width, length), where width and length are the number of points
-        along the x and y axis, respectively.
+        The discretized height field with shape (width, length) and its origin [m] with shape (3,).
     """
     # resolve terrain configuration
     step_height = cfg.step_height_range[0] + difficulty * (cfg.step_height_range[1] - cfg.step_height_range[0])
@@ -207,11 +209,14 @@ def pyramid_stairs_terrain(difficulty: float, cfg: hf_terrains_cfg.HfPyramidStai
         hf_raw[start_x:stop_x, start_y:stop_y] = current_step_height
 
     # round off the heights to the nearest vertical step
-    return np.rint(hf_raw).astype(np.int16)
+    height_field = np.rint(hf_raw).astype(np.int16)
+    return height_field, _terrain_origin(height_field, cfg, height_field[width_pixels // 2, length_pixels // 2])
 
 
 @height_field_to_mesh
-def discrete_obstacles_terrain(difficulty: float, cfg: hf_terrains_cfg.HfDiscreteObstaclesTerrainCfg) -> np.ndarray:
+def discrete_obstacles_terrain(
+    difficulty: float, cfg: hf_terrains_cfg.HfDiscreteObstaclesTerrainCfg
+) -> tuple[np.ndarray, np.ndarray]:
     """Generate a terrain with randomly generated obstacles as pillars with positive and negative heights.
 
     The terrain is a flat platform at the center of the terrain with randomly generated obstacles as pillars
@@ -228,9 +233,7 @@ def discrete_obstacles_terrain(difficulty: float, cfg: hf_terrains_cfg.HfDiscret
         cfg: The configuration for the terrain.
 
     Returns:
-        The height field of the terrain as a 2D numpy array with discretized heights.
-        The shape of the array is (width, length), where width and length are the number of points
-        along the x and y axis, respectively.
+        The discretized height field with shape (width, length) and its origin [m] with shape (3,).
     """
     # resolve terrain configuration
     obs_height = cfg.obstacle_height_range[0] + difficulty * (
@@ -286,11 +289,12 @@ def discrete_obstacles_terrain(difficulty: float, cfg: hf_terrains_cfg.HfDiscret
     y2 = (length_pixels + platform_width) // 2
     hf_raw[x1:x2, y1:y2] = 0
     # round off the heights to the nearest vertical step
-    return np.rint(hf_raw).astype(np.int16)
+    height_field = np.rint(hf_raw).astype(np.int16)
+    return height_field, _terrain_origin(height_field, cfg, 0)
 
 
 @height_field_to_mesh
-def wave_terrain(difficulty: float, cfg: hf_terrains_cfg.HfWaveTerrainCfg) -> np.ndarray:
+def wave_terrain(difficulty: float, cfg: hf_terrains_cfg.HfWaveTerrainCfg) -> tuple[np.ndarray, np.ndarray]:
     r"""Generate a terrain with a wave pattern.
 
     The terrain is a flat platform at the center of the terrain with a wave pattern. The wave pattern
@@ -313,9 +317,7 @@ def wave_terrain(difficulty: float, cfg: hf_terrains_cfg.HfWaveTerrainCfg) -> np
         cfg: The configuration for the terrain.
 
     Returns:
-        The height field of the terrain as a 2D numpy array with discretized heights.
-        The shape of the array is (width, length), where width and length are the number of points
-        along the x and y axis, respectively.
+        The discretized height field with shape (width, length) and its origin [m] with shape (3,).
 
     Raises:
         ValueError: When the number of waves is non-positive.
@@ -347,11 +349,14 @@ def wave_terrain(difficulty: float, cfg: hf_terrains_cfg.HfWaveTerrainCfg) -> np
     # add the waves
     hf_raw += amplitude_pixels * (np.cos(yy * wave_number) + np.sin(xx * wave_number))
     # round off the heights to the nearest vertical step
-    return np.rint(hf_raw).astype(np.int16)
+    height_field = np.rint(hf_raw).astype(np.int16)
+    return height_field, _terrain_origin(height_field, cfg)
 
 
 @height_field_to_mesh
-def stepping_stones_terrain(difficulty: float, cfg: hf_terrains_cfg.HfSteppingStonesTerrainCfg) -> np.ndarray:
+def stepping_stones_terrain(
+    difficulty: float, cfg: hf_terrains_cfg.HfSteppingStonesTerrainCfg
+) -> tuple[np.ndarray, np.ndarray]:
     """Generate a terrain with a stepping stones pattern.
 
     The terrain is a stepping stones pattern which trims to a flat platform at the center of the terrain.
@@ -365,9 +370,7 @@ def stepping_stones_terrain(difficulty: float, cfg: hf_terrains_cfg.HfSteppingSt
         cfg: The configuration for the terrain.
 
     Returns:
-        The height field of the terrain as a 2D numpy array with discretized heights.
-        The shape of the array is (width, length), where width and length are the number of points
-        along the x and y axis, respectively.
+        The discretized height field with shape (width, length) and its origin [m] with shape (3,).
     """
     # resolve terrain configuration
     stone_width = cfg.stone_width_range[1] - difficulty * (cfg.stone_width_range[1] - cfg.stone_width_range[0])
@@ -434,4 +437,28 @@ def stepping_stones_terrain(difficulty: float, cfg: hf_terrains_cfg.HfSteppingSt
     y2 = (length_pixels + platform_width) // 2
     hf_raw[x1:x2, y1:y2] = 0
     # round off the heights to the nearest vertical step
-    return np.rint(hf_raw).astype(np.int16)
+    height_field = np.rint(hf_raw).astype(np.int16)
+    return height_field, _terrain_origin(height_field, cfg, 0)
+
+
+def _terrain_origin(
+    height_field: np.ndarray, cfg: hf_terrains_cfg.HfTerrainBaseCfg, height: int | None = None
+) -> np.ndarray:
+    """Return the origin [m] in the generated height field's local frame."""
+    center_x = height_field.shape[0] // 2
+    center_y = height_field.shape[1] // 2
+    if height is None:
+        radius = max(1, int(1.0 / cfg.horizontal_scale))
+        height = np.max(
+            height_field[
+                max(0, center_x - radius) : center_x + radius,
+                max(0, center_y - radius) : center_y + radius,
+            ]
+        )
+    return np.array(
+        [
+            (height_field.shape[0] - 1) * cfg.horizontal_scale / 2,
+            (height_field.shape[1] - 1) * cfg.horizontal_scale / 2,
+            height * cfg.vertical_scale,
+        ]
+    )

source/isaaclab/isaaclab/terrains/height_field/utils.py

@@ -25,8 +25,8 @@ def height_field_to_mesh(func: Callable) -> Callable:
     Additionally, it adds a border around the terrain to avoid artifacts at the edges.
 
     Args:
-        func: The height field function to convert. The function should return a 2D numpy array
-            with the heights of the terrain.
+        func: The height field function to convert. It should return the height field with discretized heights
+            and the terrain origin [m] relative to the generated height field.
 
     Returns:
         The mesh function. The mesh function returns a tuple containing a list of ``trimesh``
@@ -53,7 +53,7 @@ def wrapper(difficulty: float, cfg: HfTerrainBaseCfg):
         terrain_size = copy.deepcopy(cfg.size)
         cfg.size = tuple(sub_terrain_size)
         # generate the height field
-        z_gen = func(difficulty, cfg)
+        z_gen, origin = func(difficulty, cfg)
         # handle the border for the terrain
         heights[border_pixels:-border_pixels, border_pixels:-border_pixels] = z_gen
         # set terrain size back to config
@@ -64,13 +64,8 @@ def wrapper(difficulty: float, cfg: HfTerrainBaseCfg):
             heights, cfg.horizontal_scale, cfg.vertical_scale, cfg.slope_threshold
         )
         mesh = trimesh.Trimesh(vertices=vertices, faces=triangles)
-        # compute origin
-        x1 = int((cfg.size[0] * 0.5 - 1) / cfg.horizontal_scale)
-        x2 = int((cfg.size[0] * 0.5 + 1) / cfg.horizontal_scale)
-        y1 = int((cfg.size[1] * 0.5 - 1) / cfg.horizontal_scale)
-        y2 = int((cfg.size[1] * 0.5 + 1) / cfg.horizontal_scale)
-        origin_z = np.max(heights[x1:x2, y1:y2]) * cfg.vertical_scale
-        origin = np.array([0.5 * cfg.size[0], 0.5 * cfg.size[1], origin_z])
+        # place the generator's origin in the padded mesh
+        origin[:2] += border_pixels * cfg.horizontal_scale
         return [mesh], origin
 
     return wrapper