From 2dd81df8f31e537ab97bea3b47b8477938ea06e6 Mon Sep 17 00:00:00 2001 From: ooctipus Date: Wed, 23 Sep 2026 20:12:11 -0700 Subject: [PATCH 1/8] [4C/10] Route rigid rendering transforms through SDP (#7941) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Route rigid-body transforms from physics to renderers through `SceneDataProvider` (SDP). Physics publishes its native buffer and increments its version after changes. SDP binds that buffer when format and ordering match, or converts it once per requested layout and reuses the result until the next change. Readers never reset the producer's version. - **OVRTX:** converts physics poses directly into its matrix format, without first copying them into a Newton render state. - **Newton rendering:** borrows SDP's transform buffer instead of copying into a second buffer. Newton physics and rendering continue sharing their existing model/state. - **Isaac RTX:** shares one registry-owned Fabric binding with Kit, preserves authored scale, and updates the transform hierarchy on GPU. PhysX publishes its native Fabric matrices without fetching packed poses. - **Ownership:** physics refreshes native data; SDP borrows or converts arrays. One `FabricBackend` in `isaaclab_physx` owns the stage/hierarchy handles and shared transform bindings, identified by stage/device. Consumers pass SDP explicitly to updates; it is not part of backend identity. Core `RenderContext` has no Fabric methods or state. `FabricMatrix44` contains only matrix storage. - **Rendering updates:** removes repeated renderer-driven physics refreshes. Writes made between physics steps remain visible on the next render. For Newton → Isaac RTX, cached bindings and GPU hierarchy updates replace repeated binding setup and the CPU hierarchy fallback. This is the main runtime saving measured below. Particle/deformable transport is outside this change. OVRTX retains its existing Newton geometry bridge. Counts are SDP output-writing passes, not total SDK-internal copies. Conversion, reordering, and scale are combined in one pass; unchanged data reuses the result. | Physics | Renderer | Published → requested format | SDP passes | | --- | --- | --- | ---: | | Newton | Newton Warp | `Transform` → `Transform` | 0 | | Newton | OVRTX | `Transform` → `TransposedMatrix44d` | 1 | | Newton | Isaac RTX | `Transform` → `FabricMatrix44` | 1 | | OVPhysX | Newton Warp | `Transform` → `Transform` | 0 if ordering matches; otherwise 1 | | OVPhysX | OVRTX | `Transform` → `TransposedMatrix44d` | 1 | | Isaac PhysX | Newton Warp | `Transform` → `Transform` | 0 if ordering matches; otherwise 1 | | Isaac PhysX | Isaac RTX | Native Fabric → borrowed Fabric | 0 with `use_fabric=True` | OVPhysX and OVRTX cannot run with Kit. Fabric hierarchy propagation and OVRTX's native attribute write happen after the SDP pass and are not included in these counts. Kuka Allegro Camera, 4096 environments, 64×64 RGB, Newton MJWarp → Isaac RTX, RTX 5090, no interactive visualizer. Two warm-cache runs per revision; runtime excludes 25 warmup steps and measures 200 synchronized full environment steps, including rendering and observations. | Metric | PR | Develop | | --- | ---: | ---: | | Warm startup | 135.47 s | 133.19 s | | Runtime step | 172.32 ms | 554.49 ms | | Environment frames/s | 23,770 | 7,387 | The measured step time was **68.9% lower (3.22× throughput)**. No startup improvement was measured. All 4096 camera images were finite and nonconstant. Measured revisions: PR `257e54d5f` and develop `53a7f1c0a`, with identical dependencies. Subsequent cleanup has not been rebenchmarked. Custom scene-data backends must initialize `transforms_version=0` and increment it after native pose writes or buffer swaps. The consumer-facing `get_transforms(output)` API binds shared, read-only arrays, including converted outputs. Pass `allow_passthrough=False` for caller-owned writable or preallocated arrays; conversion writes directly into them. Single-format backends keep their existing `transforms` property; the base `get_transforms(output_format)` delegates to it. Multi-format backends may override that method and `native_transform_formats`. - Focused CPU tests and GPU 0 Newton/PhysX Fabric tests cover pointer sharing, publication versions, transform formats, ordering, authored scale, buffer reallocation, same-step writes, and resets. The existing cache regression now also checks that independent SDP readers cannot hide producer changes from one another. - All 11 native Newton/Fabric tests and the native PhysX Fabric test passed after moving bindings out of core. They also check one shared RTX/Kit resource and explicit Newton-to-Fabric updates without an RTX camera or Kit viewer. - Removed obsolete synchronization tests and duplicate mock-only checks; native and numerical regressions remain. - Native OVPhysX → OVRTX rendered scale, pose changes, and camera calibration passed for both legacy and ovstage APIs. Matched OVRTX timings have not been collected. - Refactor and bug fixes - Breaking custom scene-data backend interface change, with migration above - [x] Backport to the active release branch - [x] Contribution guidelines reviewed - [x] Changelog fragments and migration documentation updated - [x] Retained focused tests and formatting checks passed after test cleanup - [ ] Full GPU CI passed for the latest revision - [x] Native OVPhysX → OVRTX rendering validated (cherry picked from commit 22bfa0dba7fcd11431f44f1f736b7866fcbb8515) --- .../developer-tools/scene_data_providers.rst | 63 ++- .../sdp-transform-publication.major.rst | 19 + .../isaaclab/renderers/render_context.py | 30 +- .../isaaclab/scene/interactive_scene.py | 3 +- .../isaaclab/isaaclab/scene_data/__init__.pyi | 7 +- .../isaaclab/scene_data/scene_data_backend.py | 30 +- .../scene_data/scene_data_provider.py | 292 ++++++----- .../isaaclab/sim/simulation_context.py | 12 +- .../assets/_articulation_iface_test_utils.py | 2 + ...igid_object_collection_iface_test_utils.py | 2 + .../assets/_rigid_object_iface_test_utils.py | 2 + .../test/envs/test_direct_marl_env.py | 17 - .../test/envs/test_env_rendering_logic.py | 24 - .../test_simulation_render_context.py | 10 +- .../scene_data/test_scene_data_transforms.py | 203 +++++++- ...test_newton_manager_visualization_state.py | 197 +++++--- .../changelog.d/sdp-transform-transport.rst | 11 + .../isaaclab_newton/physics/newton_manager.py | 353 ++++--------- .../renderers/newton_warp_renderer.py | 19 +- .../physics/test_newton_fabric_body_sync.py | 255 ++++------ .../test_newton_manager_abstraction.py | 98 ++-- .../changelog.d/sdp-transform-transport.rst | 10 + .../assets/articulation/articulation.py | 16 + .../assets/articulation/articulation_data.py | 1 + .../assets/rigid_object/rigid_object.py | 4 + .../rigid_object_collection.py | 4 + .../isaaclab_ov/physics/ovphysx_manager.py | 175 +++---- .../isaaclab_ov/renderers/ovrtx_renderer.py | 183 ++----- .../renderers/ovrtx_renderer_kernels.py | 27 - .../test_ovphysx_scene_data_backend.py | 464 +++++------------- .../isaaclab_ov/test/test_ovrtx_clone_plan.py | 59 +-- .../test/test_ovrtx_deformable_bindings.py | 321 ++++-------- .../test/test_ovrtx_renderer_contract.py | 122 ++--- .../changelog.d/sdp-transform-publication.rst | 8 + .../assets/articulation/articulation.py | 4 + .../assets/articulation/articulation_data.py | 1 + .../assets/rigid_object/rigid_object.py | 2 + .../rigid_object_collection.py | 2 + .../isaaclab_physx/physics/physx_manager.py | 85 +++- .../isaaclab_physx/renderers/fabric.py | 126 +++++ .../renderers/isaac_rtx_renderer.py | 12 +- .../renderers/isaac_rtx_renderer_utils.py | 19 +- .../test_isaac_rtx_renderer_contract.py | 21 +- .../test_isaac_rtx_renderer_utils.py | 151 ++---- .../test/sim/test_physx_scene_data_backend.py | 94 ++++ .../test/sim/test_views_xform_prim_fabric.py | 65 ++- .../changelog.d/sdp-transform-publication.rst | 6 + .../kit/kit_visualizer.py | 18 +- .../test_kit_visualizer_scene_partitioning.py | 22 + .../test/visualizer_golden_utils.py | 12 +- .../test/visualizer_integration_utils.py | 98 +--- 51 files changed, 1693 insertions(+), 2088 deletions(-) create mode 100644 source/isaaclab/changelog.d/sdp-transform-publication.major.rst create mode 100644 source/isaaclab_newton/changelog.d/sdp-transform-transport.rst create mode 100644 source/isaaclab_ov/changelog.d/sdp-transform-transport.rst create mode 100644 source/isaaclab_physx/changelog.d/sdp-transform-publication.rst create mode 100644 source/isaaclab_physx/isaaclab_physx/renderers/fabric.py create mode 100644 source/isaaclab_visualizers/changelog.d/sdp-transform-publication.rst diff --git a/docs/source/developer-tools/scene_data_providers.rst b/docs/source/developer-tools/scene_data_providers.rst index 0b4a5eeb5407..ae20df95dc06 100644 --- a/docs/source/developer-tools/scene_data_providers.rst +++ b/docs/source/developer-tools/scene_data_providers.rst @@ -31,14 +31,19 @@ The system has three layers: 1. :class:`~isaaclab.scene_data.SceneDataBackend`: a small interface implemented by each physics manager. It exposes the backend's transform array directly as one of the :class:`~isaaclab.scene_data.SceneDataFormat` Warp structs, plus the per-transform prim paths - and total count. There is no per-frame "update" call; the property accessors return live - views into the underlying tensor each time they're read. + and total count. Producers increment ``transforms_version`` after native state writes or buffer swaps; + SDP calls ``get_transforms(output_format)`` before reading the version, since resolving the pointer + can itself detect a swap. The default implementation returns the existing ``transforms`` property. + The version never resets, so independent readers cannot hide changes from one another. - - :attr:`SceneDataBackend.transforms`: current transforms as a Warp struct (one of + - :attr:`SceneDataBackend.transforms`: the native data as a Warp struct (one of :class:`SceneDataFormat.Vec3_Quat`, :class:`SceneDataFormat.Transform`, :class:`SceneDataFormat.Matrix44`, :class:`SceneDataFormat.Vec3_Matrix33`). + - :attr:`SceneDataBackend.transforms_version`: monotonic version of the native transforms. - :attr:`SceneDataBackend.transform_count`: number of transforms. - :attr:`SceneDataBackend.transform_paths`: list of USD prim paths, one per transform. + - :attr:`SceneDataBackend.native_transform_formats`: formats published without conversion. + PhysX publishes either packed poses or Fabric matrices and refreshes only the requested representation. - :attr:`SceneDataBackend.points`: flattened deformable nodal positions as :class:`SceneDataFormat.Points` (optional; rigid-only backends return an empty buffer). - :attr:`SceneDataBackend.point_count`: total number of geometry points. @@ -48,11 +53,10 @@ The system has three layers: 2. :class:`~isaaclab.scene_data.SceneDataProvider`: wraps a backend and offers format conversion plus index re-mapping. - - :meth:`SceneDataProvider.get_transforms`: writes the backend's transforms into a - consumer-provided :class:`SceneDataFormat` struct, optionally converting format - (e.g. ``Vec3_Quat`` to ``Transform``) and applying an index mapping. When the backend - format matches the output format and no mapping is provided, the result is a zero-copy - passthrough. + - :meth:`SceneDataProvider.get_transforms`: binds native arrays when format and ordering match, + or SDP-owned buffers converted once per producer version and destination layout. These shared + arrays are read-only, including when they replace preallocated output fields. Pass + ``allow_passthrough=False`` to write directly into caller-owned arrays instead. - :meth:`SceneDataProvider.create_mapping`: builds a remap array from the backend's prim paths to a consumer's desired ordering. Used when a renderer or visualizer wants transforms indexed by its own body list rather than by the physics view order. @@ -86,10 +90,12 @@ When PhysX is the active physics backend, the provider reads transforms directly The transforms are returned as :class:`SceneDataFormat.Transform` (Warp ``transformf`` array), so consumers that want this format get them zero-copy. -Newton-native consumers (Newton visualizer, Rerun, Viser, Newton Warp renderer, OVRTX renderer) -also need a Newton ``Model``/``State`` to render against. To provide that, -:class:`~isaaclab_newton.physics.NewtonManager` builds a **shadow Newton model** from the USD -stage on first access and updates its ``body_q`` from the PhysX backend each render frame. +Newton-native consumers (Newton visualizer, Rerun, Viser, Newton Warp renderer) also need a +Newton ``Model``/``State``. Their declared cloning contexts construct that representation from +the shared clone plan before initialization. Its rigid ``body_q`` binds to SDP's requested +``Transform`` array; no intermediate per-frame copy into a second state buffer is required. +OVRTX requests ``TransposedMatrix44d`` directly from SDP, including destination ordering and +static scale in the same conversion. It no longer reads Newton state for rigid transforms. When the scene has PhysX or OVPhysX deformables, the shadow model also allocates ``particle_q`` render slots for soft/cloth meshes, syncs simulation nodal positions through :meth:`SceneDataProvider.get_points` with ``allow_passthrough=False`` into a separate @@ -99,8 +105,26 @@ barycentric sim-to-visual remap so Newton Warp and OVRTX render the paired visua than tet simulation topology. The shadow deformable registry exposes render-slot offsets and ``particles_per_body`` counts for OVRTX point bindings. -This is hidden behind :meth:`NewtonManager.get_model` / :meth:`NewtonManager.get_state`, so -renderers don't need to know which physics backend is active. +The deformable and cable geometry bridge remains separate from this rigid-transform path. +OVRTX still uses Newton geometry metadata for those features. + +PhysX owns its native Fabric refresh and publishes the resulting matrices through SDP without +fetching packed poses. ``isaaclab_physx.renderers.fabric.FabricBackend`` owns the shared native stage +and hierarchy handles. Its identity is the stage and device, not the SDP source or attribute type. +``SimulationContext`` declares ``fabric_cfg`` when Kit is available. After physics initializes, Kit, +Isaac RTX, and explicit Fabric synchronization obtain the same resource through +``get_or_create_backend(sim.fabric_cfg)``. Transform bindings are state on that resource, not a +separate backend. Consumers pass the simulation's SDP to ``update_transforms(provider)``; for foreign +physics it converts directly into Fabric local matrices, then propagates the GPU hierarchy. +Core ``RenderContext`` owns no Fabric bindings. +It binds rigid destinations as Fabric-only reset-stack roots because +physics publishes absolute poses, including for nested bodies. Visual descendants still inherit +their body's transform; authored USD is unchanged. Native source indices and world scales are +bound once. Fabric's selection reuse API reports scene-wide structural changes; the resource refreshes +array views without repeating path matching or scale capture. Otherwise GPU propagation +reuses the hierarchy topology. Clean requests never acquire writable Fabric arrays. +Renderers do not select a physics-specific synchronization path. +``FabricMatrix44`` contains only matrix storage, not bindings or native engine handles. Newton backend -------------- @@ -109,11 +133,20 @@ When Newton is the active physics backend, the backend wraps the Newton model's directly. No shadow model or per-frame sync is needed: Newton already owns the authoritative model and state, and the provider exposes that state as :class:`SceneDataFormat.Transform`. +Native reads reconcile pending authored state writes once. A new physics publication does not +itself request forward kinematics. Kit/RTX requests current Fabric transforms through SDP +before rendering, without issuing an additional physics ``forward()``. Headless viewport +capture requests these transforms on demand rather than on every visualizer step. + +Externally replayed CUDA graphs do not call Python write hooks. After writes have been captured, +Newton conservatively republishes transforms when read so an unannounced replay cannot leave +rendering stale. Those reads do not benefit from clean-publication caching. + Data requirements ------------------ Visualizers and renderers declare what they need from the scene data path. This is resolved at -simulation-context construction time and is what triggers the shadow-model build for PhysX: +consumer construction time, before the shared clone plan is built: .. list-table:: :header-rows: 1 diff --git a/source/isaaclab/changelog.d/sdp-transform-publication.major.rst b/source/isaaclab/changelog.d/sdp-transform-publication.major.rst new file mode 100644 index 000000000000..36c90c2c759b --- /dev/null +++ b/source/isaaclab/changelog.d/sdp-transform-publication.major.rst @@ -0,0 +1,19 @@ +Changed +^^^^^^^ + +* **Breaking:** Added ``transforms_version`` to scene-data backends. Custom backends must initialize + it to zero and increment it after native pose writes or buffer swaps. SDP reads the existing + ``transforms`` property through ``get_transforms(output_format)`` without resetting the version. Backends + publishing multiple native formats may override that method and ``native_transform_formats``. + ``SceneDataProvider.get_transforms`` bound shared, + read-only arrays by default: matching layouts aliased native data and other layouts converted + once per publication. Callers requiring their own writable or preallocated arrays must pass + ``allow_passthrough=False``; this wrote directly into the supplied arrays without a staging copy. +* Routed rigid Fabric conversion through SDP while the Kit rendering integration owned destination binding and + GPU hierarchy propagation, preserving native PhysX publication and authored scale. Converted rigid destinations + became Fabric-only reset-stack roots so nested bodies retained their absolute physics poses. + Transform freshness no longer depended on the physics-step counter; + ``RenderContext.reset_scene_state_cadence`` remained available for geometry updates. + ``SimulationContext.fabric_cfg`` declared the shared native Fabric stage/device without allocating bindings. +* Used native Warp structs for Fabric transform bindings, relying on the project-managed Warp + dependency selected by Isaac Lab's Kit launch configuration. diff --git a/source/isaaclab/isaaclab/renderers/render_context.py b/source/isaaclab/isaaclab/renderers/render_context.py index 9ca243c1a671..bba6f7a76bf7 100644 --- a/source/isaaclab/isaaclab/renderers/render_context.py +++ b/source/isaaclab/isaaclab/renderers/render_context.py @@ -60,8 +60,8 @@ def _write_material( class RenderContext: """Orchestrate simulation-owned renderers and own flat runtime material buffers. - Renderer instances are borrowed from the simulation's backend registry. Scene state updates - run at most once per physics step, regardless of how many cameras share a renderer. + Renderer instances are borrowed from the simulation's backend registry. SDP owns transform + freshness, including pose writes that do not advance the physics-step counter. """ __slots__ = ( @@ -70,7 +70,7 @@ class RenderContext: "_physics_initialized", "_prepared_renderer_ids", "_prepared_num_envs", - "_last_scene_state_step", + "_last_geometry_update_step", "_visual_materials", "_visual_material_batches", "_visual_material_batches_by_channel", @@ -88,7 +88,7 @@ def __init__(self, backend_registry: list[tuple[BackendCfg, Any]]) -> None: self._physics_initialized: bool = False # Set to True after the first PHYSICS_READY callback fires. self._prepared_renderer_ids: set[int] = set() self._prepared_num_envs: int | None = None - self._last_scene_state_step: int | None = None + self._last_geometry_update_step: int | None = None # Physics step of the last renderer geometry update. self._visual_materials: list[Any] = [] self._visual_material_batches: tuple[VisualMaterialBatch, ...] = () self._visual_material_batches_by_channel: dict[str, VisualMaterialBatch] = {} @@ -126,7 +126,7 @@ def validate_renderer_cfg(self, cfg: RendererCfg) -> None: def register_renderer(self, cfg: RendererCfg, renderer: BaseRenderer) -> None: """Include a newly registry-owned renderer in cloning and post-physics initialization.""" self.clone_contexts.update(cfg.cloning_contexts) - self._last_scene_state_step = None + self._last_geometry_update_step = None if self._physics_initialized: renderer.initialize() @@ -317,19 +317,15 @@ def ensure_prepare_stage(self, stage: Any, num_envs: int) -> None: self._prepared_num_envs = num_envs def update_scene_state(self, physics_step_count: int) -> None: - """Update scene state on all backends (at most once per step). + """Publish physics state and refresh renderers through SDP's producer versions. - Invokes :meth:`BaseRenderer.update_transforms` and then - :meth:`BaseRenderer.update_geometries` on each registered renderer. + Transforms follow SDP freshness; geometry updates retain their once-per-step cadence. """ - if self._last_scene_state_step == physics_step_count: - return - for _cfg, renderer in self._renderer_entries: renderer.update_transforms() - renderer.update_geometries() - - self._last_scene_state_step = physics_step_count + if self._last_geometry_update_step != physics_step_count: + renderer.update_geometries() + self._last_geometry_update_step = physics_step_count def render_into_camera( self, @@ -349,8 +345,8 @@ def reset_stage_prepare_flag(self) -> None: self._prepared_num_envs = None def reset_scene_state_cadence(self) -> None: - """Clear per-step scene state update dedupe (e.g. a long pause with no physics).""" - self._last_scene_state_step = None + """Invalidate geometry updates after resets that do not advance the physics step.""" + self._last_geometry_update_step = None def close(self) -> None: """Release material writers and lifecycle bookkeeping, not registry-owned renderers. @@ -368,7 +364,7 @@ def close(self) -> None: self.clone_contexts.clear() self._prepared_renderer_ids.clear() self._prepared_num_envs = None - self._last_scene_state_step = None + self._last_geometry_update_step = None self._physics_initialized = False self._visual_materials.clear() self._visual_material_batches = () diff --git a/source/isaaclab/isaaclab/scene/interactive_scene.py b/source/isaaclab/isaaclab/scene/interactive_scene.py index 40099db6b875..ca5d61ccd5ae 100644 --- a/source/isaaclab/isaaclab/scene/interactive_scene.py +++ b/source/isaaclab/isaaclab/scene/interactive_scene.py @@ -498,8 +498,7 @@ def update(self, dt: float) -> None: Args: dt: The amount of time passed from last :meth:`update` call. """ - # Scene-wide renderer scene-state sync once per step when all sensors update, - # so per-camera fetches do not own this concern (deduped inside RenderContext). + # Publish transforms before eager sensors read their Fabric-backed poses. if not self.cfg.lazy_sensor_update: self.sim.render_context.update_scene_state(self.sim.get_physics_step_count()) diff --git a/source/isaaclab/isaaclab/scene_data/__init__.pyi b/source/isaaclab/isaaclab/scene_data/__init__.pyi index d4e47a65c698..d43d7e98fd78 100644 --- a/source/isaaclab/isaaclab/scene_data/__init__.pyi +++ b/source/isaaclab/isaaclab/scene_data/__init__.pyi @@ -3,12 +3,7 @@ # # SPDX-License-Identifier: BSD-3-Clause -__all__ = [ - "REQUIRES_STAGE_AND_MODEL", - "SceneDataBackend", - "SceneDataFormat", - "SceneDataProvider", -] +__all__ = ["REQUIRES_STAGE_AND_MODEL", "SceneDataBackend", "SceneDataFormat", "SceneDataProvider"] from .scene_data_backend import SceneDataBackend, SceneDataFormat from .scene_data_provider import REQUIRES_STAGE_AND_MODEL, SceneDataProvider diff --git a/source/isaaclab/isaaclab/scene_data/scene_data_backend.py b/source/isaaclab/isaaclab/scene_data/scene_data_backend.py index af38842da67f..6bbef2b4ccc8 100644 --- a/source/isaaclab/isaaclab/scene_data/scene_data_backend.py +++ b/source/isaaclab/isaaclab/scene_data/scene_data_backend.py @@ -16,6 +16,8 @@ from __future__ import annotations +from typing import Any + import warp as wp # Under Sphinx ``autodoc_mock_imports``, ``wp.struct`` is a ``_MockObject`` @@ -69,6 +71,20 @@ class Matrix44: matrices: wp.array(dtype=wp.mat44f) = None """Per-transform 4x4 homogeneous transform matrices [m].""" + @wp_struct + class TransposedMatrix44d: + """Double-precision row-vector transforms, as consumed by USD renderers.""" + + matrices: wp.array(dtype=wp.mat44d) = None + """World transforms [m], shape [transform_count].""" + + @wp_struct + class FabricMatrix44: + """Double-precision row-vector matrices in native Fabric storage.""" + + matrices: wp.fabricarray(dtype=wp.mat44d) = None + """Transforms [m], shape [transform_count].""" + @wp_struct class Points: """Flat world-space nodal or particle positions.""" @@ -78,13 +94,25 @@ class Points: class SceneDataBackend: + transforms_version: int + """Monotonic producer version, incremented after native writes or buffer swaps; never reset by readers.""" + + @property + def native_transform_formats(self) -> tuple[Any, ...]: + """Formats available without conversion, used when binding consumer destinations.""" + return (self.transforms._cls,) + + def get_transforms(self, output_format: Any) -> Any: + """Publish the requested native format when available, otherwise the primary format.""" + return self.transforms + @property def transforms( self, ) -> ( SceneDataFormat.Vec3_Quat | SceneDataFormat.Transform | SceneDataFormat.Matrix44 | SceneDataFormat.Vec3_Matrix33 ): - """Return the sim backends transforms as one of the SceneDataFormat structs.""" + """Return native transforms without copying; pointer changes must increment ``transforms_version``.""" raise NotImplementedError @property diff --git a/source/isaaclab/isaaclab/scene_data/scene_data_provider.py b/source/isaaclab/isaaclab/scene_data/scene_data_provider.py index 9ebfc0d27031..8ceb1cf95c7c 100644 --- a/source/isaaclab/isaaclab/scene_data/scene_data_provider.py +++ b/source/isaaclab/isaaclab/scene_data/scene_data_provider.py @@ -13,8 +13,7 @@ import numpy as np import warp as wp -import isaaclab.sim as sim_utils - +from .. import sim as sim_utils from .scene_data_backend import SceneDataBackend, SceneDataFormat logger = logging.getLogger(__name__) @@ -37,12 +36,13 @@ def _publication_device(data: Any) -> wp.Device: """Return the common device of a populated scene-data publication.""" - arrays = tuple(array for name in data._cls.vars if (array := getattr(data, name)) is not None) + data_format = data._cls + arrays = tuple(array for name in data_format.vars if (array := getattr(data, name)) is not None) if not arrays: - raise ValueError(f"{data._cls.__name__} contains no published arrays.") + raise ValueError(f"{data_format.__name__} contains no published arrays.") device = arrays[0].device if any(array.device != device for array in arrays[1:]): - raise ValueError(f"{data._cls.__name__} arrays must share one device.") + raise ValueError(f"{data_format.__name__} arrays must share one device.") return device @@ -54,6 +54,8 @@ def _init_output(output: Any, count: int, device: wp.Device) -> None: class SceneDataProvider: + """Borrow or convert published arrays; producers own native refresh, renderers own destination lifecycle.""" + def __init__(self, backend: SceneDataBackend): """Initialize the scene data provider. @@ -63,6 +65,101 @@ def __init__(self, backend: SceneDataBackend): self.backend = backend self._num_envs_cache: int | None = None self._interactive_scene: Any | None = None + self._transform_cache: dict[tuple, tuple[int, Any]] = {} + + def get_transforms( + self, + output: SceneDataFormat.Vec3_Quat + | SceneDataFormat.Transform + | SceneDataFormat.Matrix44 + | SceneDataFormat.Vec3_Matrix33 + | SceneDataFormat.TransposedMatrix44d + | SceneDataFormat.FabricMatrix44, + mapping: wp.array | wp.fabricarray | None = None, + allow_passthrough: bool = True, + *, + count: int | None = None, + scales: wp.array | None = None, + ) -> bool: + """Bind shared transforms or write them directly into caller-owned output arrays. + + With passthrough enabled, matching native arrays are borrowed without a copy; other + layouts share SDP-owned buffers converted once per producer version. Treat these arrays + as read-only. With passthrough disabled, conversion writes directly into ``output``. + Fabric destinations must already be bound by their rendering owner. + + Args: + output: A :class:`SceneDataFormat` struct instance specifying the requested format. + Missing non-Fabric arrays are allocated when passthrough is disabled. + mapping: Native-to-output indices from :meth:`create_mapping`, or identity ordering. + Fabric destinations use their native output-to-source index attribute. + allow_passthrough: Whether to bind shared arrays instead of writing caller-owned arrays. + count: Destination count when remapping, or the native transform count. + scales: Static output scales for ``TransposedMatrix44d``, shape [count], or + source-indexed authored scales for Fabric. Mapping and scales are immutable + for a binding's lifetime; replace their arrays when the layout changes. + + Returns: + True if transforms are available in ``output``, False if no transforms are published + or the format conversion is unsupported. + """ + # Warp exposes the struct's field/type descriptor as _cls, not its Python type. + output_format = output._cls + fabric = output_format is SceneDataFormat.FabricMatrix44 + source = self.backend.get_transforms(output_format) + source_format = source._cls + version = self.backend.transforms_version + native_count = next( + (len(array) for name in source_format.vars if (array := getattr(source, name)) is not None), 0 + ) + if native_count == 0: + return False + count = native_count if count is None else count + if mapping is None and count != native_count: + raise ValueError("A different destination count requires an explicit transform mapping.") + if scales is not None and output_format not in ( + SceneDataFormat.TransposedMatrix44d, + SceneDataFormat.FabricMatrix44, + ): + raise ValueError("Static scales require double-precision row-vector matrix destinations.") + if source_format is output_format and mapping is None and scales is None: + result = source + if not allow_passthrough: + _init_output(output, count, _publication_device(source)) + for name in output_format.vars: + wp.copy(getattr(output, name), getattr(source, name)) + return True + else: + # A Fabric binding keeps its authored scales across selection reallocations. + key = (output_format, scales) if fabric else (output_format, mapping, count, scales) + cached = self._transform_cache.get(key) if allow_passthrough else None + if not allow_passthrough or fabric: + result = output + else: + result = cached[1] if cached is not None else output_format() + if cached is None or cached[0] != version or cached[1] is not result: + # Fabric changes storage and indexing, not the matrix conversion. + format_name = "TransposedMatrix44d" if fabric else output_format.__name__ + kernel = getattr(ConversionKernels, f"convert_{source_format.__name__}_to_{format_name}", None) + if kernel is None: + return False + device = _publication_device(source) + _init_output(result, count, device) + inputs = [source, mapping if mapping is not None else wp.array(dtype=wp.int32)] + if output_format is SceneDataFormat.TransposedMatrix44d or fabric: + inputs.append(scales) + wp.launch( + kernel, + dim=len(result.matrices) if fabric else native_count, + inputs=inputs, + outputs=[result], + device=device, + ) + if allow_passthrough: + self._transform_cache[key] = (version, result) + for name in output_format.vars: + setattr(output, name, getattr(result, name)) + return True def set_interactive_scene(self, scene: Any) -> None: """Attach the active interactive scene for scene-owned sensor discovery.""" @@ -152,65 +249,6 @@ def get_camera_transforms(self) -> dict[str, Any] | None: """ return _walk_camera_prims(self.usd_stage) - def get_transforms( - self, - output: SceneDataFormat.Vec3_Quat - | SceneDataFormat.Transform - | SceneDataFormat.Matrix44 - | SceneDataFormat.Vec3_Matrix33, - mapping: wp.array(dtype=wp.int32) | None = None, - allow_passthrough: bool = True, - ) -> bool: - """Convert sim backend transforms into the requested output format. - - When the backend's native format matches ``output``, data is either passed - through by reference (``allow_passthrough=True``) or deep-copied. Otherwise a - Warp conversion kernel is launched to transform the data, applying ``mapping`` - to reorder the output if provided. - - Args: - output: A pre-allocated :class:`SceneDataFormat` struct that determines the - target format. Uninitialized (``None``) fields are allocated automatically - when a conversion kernel is needed. - mapping: Optional index remapping array produced by - :meth:`create_mapping`. When ``None``, input and output indices are - identical. - allow_passthrough: If ``True`` and the formats already match, the output - struct's fields are set to reference the input arrays directly - (zero-copy). If ``False``, the data is always copied. - - Returns: - ``True`` if the conversion succeeded, ``False`` if no suitable conversion - kernel exists for the input/output format pair. - """ - input = self.backend.transforms - - if mapping is None and type(input) is type(output): - if allow_passthrough: - for field_name in input._cls.vars: - setattr(output, field_name, getattr(input, field_name)) - else: - _init_output(output, self.transform_count, _publication_device(input)) - for field_name in input._cls.vars: - wp.copy(getattr(output, field_name), getattr(input, field_name)) - return True - - conversion_kernel_name = f"convert_{input._cls.__name__}_to_{output._cls.__name__}" - - if conversion_kernel := getattr(ConversionKernels, conversion_kernel_name, None): - device = _publication_device(input) - _init_output(output, self.transform_count, device) - wp.launch( - kernel=conversion_kernel, - dim=self.transform_count, - inputs=[input, mapping], - outputs=[output], - device=device, - ) - return True - - return False - def init_output( self, output: SceneDataFormat.Vec3_Quat @@ -255,7 +293,7 @@ def create_mapping(self, paths: list[str | None]) -> wp.array(dtype=wp.int32) | if out_path not in path_to_out: path_to_out[out_path] = out_idx mapping = [path_to_out.get(path, -1) for path in input_paths] - if not np.array_equal(mapping, np.arange(len(input_paths))): + if len(paths) != len(input_paths) or not np.array_equal(mapping, np.arange(len(input_paths))): input = self.backend.transforms return wp.array(mapping, dtype=wp.int32, device=_publication_device(input)) return None @@ -368,6 +406,66 @@ def get_output_index(tid: wp.int32, mapping: wp.array(dtype=wp.int32)) -> wp.int return mapping[tid] return wp.int32(-1) + @wp.func + def matrix_indices(tid: int, mapping: wp.array(dtype=wp.int32)): + """Return source, destination, and authored-scale indices.""" + index = ConversionKernels.get_output_index(tid, mapping) + return tid, index, index + + @wp.func + def matrix_indices(tid: int, mapping: wp.fabricarray(dtype=wp.int32)): # noqa: F811 - Warp overload + return mapping[tid], tid, mapping[tid] + + @wp.func + def transposed_matrix(matrix: wp.mat44f, scales: wp.array(dtype=wp.vec3f), index: int) -> wp.mat44d: + result = wp.mat44d(wp.transpose(matrix)) + if scales.shape[0]: + scale = scales[index] + for row in range(3): + for column in range(3): + result[row, column] = result[row, column] * wp.float64(scale[row]) + return result + + @wp.kernel(enable_backward=False) + def convert_Transform_to_TransposedMatrix44d( + input: SceneDataFormat.Transform, mapping: Any, scales: wp.array(dtype=wp.vec3f), output: Any + ): + source, index, scale_index = ConversionKernels.matrix_indices(wp.tid(), mapping) + if index > -1: + output.matrices[index] = ConversionKernels.transposed_matrix( + wp.transform_to_matrix(input.transforms[source]), scales, scale_index + ) + + @wp.kernel(enable_backward=False) + def convert_Vec3_Quat_to_TransposedMatrix44d( + input: SceneDataFormat.Vec3_Quat, mapping: Any, scales: wp.array(dtype=wp.vec3f), output: Any + ): + source, index, scale_index = ConversionKernels.matrix_indices(wp.tid(), mapping) + if index > -1: + pose = wp.transformf(input.positions[source], input.orientations[source]) + output.matrices[index] = ConversionKernels.transposed_matrix( + wp.transform_to_matrix(pose), scales, scale_index + ) + + @wp.kernel(enable_backward=False) + def convert_Vec3_Matrix33_to_TransposedMatrix44d( + input: SceneDataFormat.Vec3_Matrix33, mapping: Any, scales: wp.array(dtype=wp.vec3f), output: Any + ): + source, index, scale_index = ConversionKernels.matrix_indices(wp.tid(), mapping) + if index > -1: + pose = wp.transformf(input.positions[source], wp.quat_from_matrix(input.orientations[source])) + output.matrices[index] = ConversionKernels.transposed_matrix( + wp.transform_to_matrix(pose), scales, scale_index + ) + + @wp.kernel(enable_backward=False) + def convert_Matrix44_to_TransposedMatrix44d( + input: SceneDataFormat.Matrix44, mapping: Any, scales: wp.array(dtype=wp.vec3f), output: Any + ): + source, index, scale_index = ConversionKernels.matrix_indices(wp.tid(), mapping) + if index > -1: + output.matrices[index] = ConversionKernels.transposed_matrix(input.matrices[source], scales, scale_index) + @wp.kernel def convert_Vec3_Quat_to_Vec3_Quat( input: SceneDataFormat.Vec3_Quat, mapping: wp.array(dtype=wp.int32), output: SceneDataFormat.Vec3_Quat @@ -634,67 +732,3 @@ def _walk_camera_prims(stage: Usd.Stage | None) -> dict[str, Any] | None: orientations.append(per_world_ori) return {"order": shared_paths, "positions": positions, "orientations": orientations, "num_envs": num_envs} - - -############################ -## Example - -if __name__ == "__main__": - - class ExampleSceneDataBackend(SceneDataBackend): - def __init__(self): - self.__transforms = SceneDataFormat.Transform() - self.__transforms.transforms = wp.array(np.hstack([np.arange(10).reshape(10, 1)] * 7), dtype=wp.transformf) - - @property - def transforms(self) -> SceneDataFormat.Transform: - return self.__transforms - - @property - def transform_count(self) -> int: - return self.__transforms.transforms.shape[0] - - @property - def transform_paths(self): - return [ - "/world/shape_01", - "/world/shape_02", - "/world/shape_03", - "/world/shape_04", - "/world/shape_05", - "/world/shape_06", - "/world/shape_07", - "/world/shape_08", - "/world/shape_09", - "/world/shape_10", - ] - - sim = ExampleSceneDataBackend() - sdp = SceneDataProvider(sim) - - output_data = SceneDataFormat.Vec3_Matrix33() - output_data.positions = wp.empty(sdp.transform_count, dtype=wp.vec3f) - output_data.orientations = wp.empty(sdp.transform_count, dtype=wp.mat33f) - - print(sim.transforms.transforms) - mapping = sdp.create_mapping( - [ - "/world/shape_02", - "/world/shape_01", - "/world/shape_03", - "/world/shape_04", - "/world/shape_05", - None, - None, - "/world/shape_10", - None, - None, - ] - ) - print(mapping) - if sdp.get_transforms(output_data, mapping): - print(output_data.positions) - else: - print("Failed to get transforms!") - - wp.synchronize() diff --git a/source/isaaclab/isaaclab/sim/simulation_context.py b/source/isaaclab/isaaclab/sim/simulation_context.py index 1f294f61b65a..a9bb7d566833 100644 --- a/source/isaaclab/isaaclab/sim/simulation_context.py +++ b/source/isaaclab/isaaclab/sim/simulation_context.py @@ -195,6 +195,12 @@ def __init__(self, cfg: SimulationCfg | None = None): # Construct visualizers before cloning; initialize their runtime bindings after physics is ready. self._scene_data_provider = SceneDataProvider(self.physics_manager.get_scene_data_backend()) + self.fabric_cfg: BackendCfg | None = None + """Native Fabric stage/device configuration, or None without Kit.""" + if use_isaac_sim: + from isaaclab_physx.renderers.fabric import FabricBackendCfg # noqa: PLC0415 + + self.fabric_cfg = FabricBackendCfg(stage=self.stage, device=self.device) self._visualizers: list[BaseVisualizer] = [] self._pending_visualizers: list[BaseVisualizer] = [] self._reset_requested: bool = False @@ -620,11 +626,7 @@ def _resolve_visualizer_cfgs(self) -> list[Any]: f"{install_hints}" ) - # XR auto-start: auto-inject a KitVisualizer when XR is active and no - # Kit visualizer is already present. The KitVisualizer pumps - # app.update() and triggers forward() (via requires_forward_before_step) - # to sync Fabric data so the XR runtime receives up-to-date hand/joint - # transforms each frame. + # XR auto-start needs a Kit visualizer to publish SDP transforms before pumping the app. if self._xr_enabled and bool(self.get_setting("/isaaclab/xr/auto_start")): has_kit = any(getattr(cfg, "visualizer_type", None) == "kit" for cfg in resolved) if not has_kit: diff --git a/source/isaaclab/test/assets/_articulation_iface_test_utils.py b/source/isaaclab/test/assets/_articulation_iface_test_utils.py index a3896243ce9c..8dfa6d01a685 100644 --- a/source/isaaclab/test/assets/_articulation_iface_test_utils.py +++ b/source/isaaclab/test/assets/_articulation_iface_test_utils.py @@ -26,6 +26,7 @@ from isaaclab_physx.assets.articulation.articulation import Articulation as PhysXArticulation from isaaclab_physx.assets.articulation.articulation_data import ArticulationData as PhysXArticulationData from isaaclab_physx.physics import PhysxManager as SimulationManager + from isaaclab_physx.physics.physx_manager import PhysxSceneDataBackend from isaaclab_physx.test.fixtures.views import MockArticulationViewWarp as PhysXMockArticulationViewWarp except ImportError as error: BACKEND_UNAVAILABLE_REASONS["physx"] = f"{type(error).__name__}: {error}" @@ -34,6 +35,7 @@ _mock_physics_sim_view = MagicMock() _mock_physics_sim_view.get_gravity.return_value = (0.0, 0.0, -9.81) SimulationManager.get_physics_sim_view = MagicMock(return_value=_mock_physics_sim_view) + SimulationManager._scene_data_backend = PhysxSceneDataBackend() BACKENDS.append("physx") diff --git a/source/isaaclab/test/assets/_rigid_object_collection_iface_test_utils.py b/source/isaaclab/test/assets/_rigid_object_collection_iface_test_utils.py index 92bda8ac518c..271b3dec8b93 100644 --- a/source/isaaclab/test/assets/_rigid_object_collection_iface_test_utils.py +++ b/source/isaaclab/test/assets/_rigid_object_collection_iface_test_utils.py @@ -29,6 +29,7 @@ RigidObjectCollectionData as PhysXRigidObjectCollectionData, ) from isaaclab_physx.physics import PhysxManager as SimulationManager + from isaaclab_physx.physics.physx_manager import PhysxSceneDataBackend from isaaclab_physx.test.fixtures.views import MockRigidBodyViewWarp as PhysXMockRigidBodyViewWarp except ImportError: pass @@ -37,6 +38,7 @@ _mock_physics_sim_view = MagicMock() _mock_physics_sim_view.get_gravity.return_value = (0.0, 0.0, -9.81) SimulationManager.get_physics_sim_view = MagicMock(return_value=_mock_physics_sim_view) + SimulationManager._scene_data_backend = PhysxSceneDataBackend() BACKENDS.append("physx") diff --git a/source/isaaclab/test/assets/_rigid_object_iface_test_utils.py b/source/isaaclab/test/assets/_rigid_object_iface_test_utils.py index 3c6d11de4f28..1d9149f2deb2 100644 --- a/source/isaaclab/test/assets/_rigid_object_iface_test_utils.py +++ b/source/isaaclab/test/assets/_rigid_object_iface_test_utils.py @@ -24,6 +24,7 @@ from isaaclab_physx.assets.rigid_object.rigid_object import RigidObject as PhysXRigidObject from isaaclab_physx.assets.rigid_object.rigid_object_data import RigidObjectData as PhysXRigidObjectData from isaaclab_physx.physics import PhysxManager as SimulationManager + from isaaclab_physx.physics.physx_manager import PhysxSceneDataBackend from isaaclab_physx.test.fixtures.views import MockRigidBodyViewWarp as PhysXMockRigidBodyViewWarp except ImportError: pass @@ -32,6 +33,7 @@ _mock_physics_sim_view = MagicMock() _mock_physics_sim_view.get_gravity.return_value = (0.0, 0.0, -9.81) SimulationManager.get_physics_sim_view = MagicMock(return_value=_mock_physics_sim_view) + SimulationManager._scene_data_backend = PhysxSceneDataBackend() BACKENDS.append("physx") diff --git a/source/isaaclab/test/envs/test_direct_marl_env.py b/source/isaaclab/test/envs/test_direct_marl_env.py index 935f2ec9bcbd..b1e3ac0abf07 100644 --- a/source/isaaclab/test/envs/test_direct_marl_env.py +++ b/source/isaaclab/test/envs/test_direct_marl_env.py @@ -45,20 +45,3 @@ def test_initialization_and_close(device): assert env._is_closed assert sim_utils.SimulationContext.instance() is None - - -def test_reset_invalidates_renderer_scene_state_cadence(): - """A same-step multi-agent reset must republish renderer scene state.""" - env = None - try: - sim_utils.create_new_stage() - env = DirectMARLEnv(cfg=make_empty_direct_marl_env_cfg()) - env._get_observations = lambda: {} - env.sim.render_context._last_scene_state_step = 7 - - env.reset() - - assert env.sim.render_context._last_scene_state_step is None - finally: - if env is not None: - env.close() diff --git a/source/isaaclab/test/envs/test_env_rendering_logic.py b/source/isaaclab/test/envs/test_env_rendering_logic.py index 8aaf1bf72014..7fb6b5134f49 100644 --- a/source/isaaclab/test/envs/test_env_rendering_logic.py +++ b/source/isaaclab/test/envs/test_env_rendering_logic.py @@ -253,30 +253,6 @@ def wrapped_step(dt): SimulationContext.clear_instance() -@pytest.mark.parametrize("env_type", ["manager_based_env", "manager_based_rl_env", "direct_rl_env"]) -def test_env_reset_invalidates_renderer_scene_state_cadence(env_type): - """A same-step reset must force the next camera read to republish scene state.""" - env = None - try: - sim_utils.create_new_stage() - if env_type == "manager_based_env": - env = create_manager_based_env(render_interval=1) - elif env_type == "manager_based_rl_env": - env = create_manager_based_rl_env(render_interval=1) - else: - env = create_direct_rl_env(render_interval=1) - - env.sim.render_context._last_scene_state_step = 7 - env.reset() - - assert env.sim.render_context._last_scene_state_step is None - finally: - if env is not None: - env.close() - else: - SimulationContext.clear_instance() - - @pytest.mark.parametrize("env_type", ["manager_based_env", "manager_based_rl_env", "direct_rl_env"]) def test_env_render_false_skips_rendering(env_type, physics_callback, render_callback): """Test that setting render_enabled=False skips all rendering while physics continues.""" diff --git a/source/isaaclab/test/renderers/test_simulation_render_context.py b/source/isaaclab/test/renderers/test_simulation_render_context.py index 84a60c13dfda..3ada5ae132d9 100644 --- a/source/isaaclab/test/renderers/test_simulation_render_context.py +++ b/source/isaaclab/test/renderers/test_simulation_render_context.py @@ -142,6 +142,7 @@ def test_close_backend_removes_renderer_from_orchestration(sim): sim.render_context.update_scene_state(2) replacement.prepare_stage.assert_called_once_with(None, 4) replacement.update_transforms.assert_called_once_with() + replacement.update_geometries.assert_called_once_with() sim.render_context.close() renderer.close.assert_called_once_with() replacement.close.assert_not_called() @@ -160,15 +161,17 @@ def test_prepare_stage_is_idempotent_and_checks_env_count_until_reset(sim): assert renderer.prepare_stage.call_args_list == [call(None, 4), call(None, 8)] -def test_scene_state_updates_once_per_step_until_cadence_reset(sim): +def test_scene_state_does_not_skip_writes_within_a_physics_step(sim): renderer = sim.get_or_create_backend(RendererCfg(class_type=_renderer)) for step in (1, 1, 2): sim.render_context.update_scene_state(step) - assert renderer.update_transforms.call_count == renderer.update_geometries.call_count == 2 + assert renderer.update_transforms.call_count == 3 + assert renderer.update_geometries.call_count == 2 sim.render_context.reset_scene_state_cadence() sim.render_context.update_scene_state(2) - assert renderer.update_transforms.call_count == renderer.update_geometries.call_count == 3 + assert renderer.update_transforms.call_count == 4 + assert renderer.update_geometries.call_count == 3 @pytest.mark.parametrize("profile", [False, True]) @@ -186,6 +189,7 @@ def test_render_into_camera_call_order_and_profile_output(sim, capsys, profile): call.update_geometries(), call.render(data), call.read_output(data, camera), + call.update_transforms(), call.render(data), call.read_output(data, camera), ] diff --git a/source/isaaclab/test/scene_data/test_scene_data_transforms.py b/source/isaaclab/test/scene_data/test_scene_data_transforms.py index 94539be89ccd..b7b8ef8e36b1 100644 --- a/source/isaaclab/test/scene_data/test_scene_data_transforms.py +++ b/source/isaaclab/test/scene_data/test_scene_data_transforms.py @@ -8,13 +8,21 @@ from __future__ import annotations from types import SimpleNamespace +from unittest.mock import Mock import numpy as np import pytest import warp as wp -from isaaclab.scene_data.scene_data_backend import SceneDataFormat +from isaaclab.scene_data.scene_data_backend import SceneDataBackend, SceneDataFormat from isaaclab.scene_data.scene_data_provider import SceneDataProvider +from isaaclab.test.utils import test_devices + + +class _Backend(SimpleNamespace, SceneDataBackend): + transforms = None + transform_count = 0 + transform_paths = () @pytest.mark.skipif( @@ -27,8 +35,9 @@ def test_get_transforms_matches_backend_device_when_warp_default_is_cuda(): [[x, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0] for x in range(3)], dtype=wp.transformf, device="cpu" ) provider = SceneDataProvider( - SimpleNamespace( + _Backend( transforms=transforms, + transforms_version=0, transform_count=3, transform_paths=["/World/a", "/World/b", "/World/c"], ) @@ -45,3 +54,193 @@ def test_get_transforms_matches_backend_device_when_warp_default_is_cuda(): assert str(output.positions.device) == "cpu" assert str(output.orientations.device) == "cpu" assert np.allclose(output.positions.numpy()[:, 0], [2.0, 0.0, 1.0]) + + +def test_publication_aliases_native_pointer_and_converts_once_per_write(monkeypatch): + """Clean reads share conversions; no provider can hide a publication from another.""" + data = SceneDataFormat.Transform() + data.transforms = wp.array([[1, 2, 3, 0, 0, 0, 1]], dtype=wp.transformf, device="cpu") + backend = _Backend(transforms=data, transforms_version=0, transform_count=1) + provider = SceneDataProvider(backend) + native = SceneDataFormat.Transform() + converted = SceneDataFormat.Vec3_Quat() + other = SceneDataFormat.Vec3_Quat() + with pytest.raises(ValueError, match="destination count"): + provider.get_transforms(native, count=2) + launch = Mock(wraps=wp.launch) + monkeypatch.setattr(wp, "launch", launch) + assert provider.get_transforms(native) + assert native.transforms is data.transforms + launch.assert_not_called() + assert provider.get_transforms(converted) + assert provider.get_transforms(other) + assert other.positions is converted.positions + assert launch.call_count == 1 + np.testing.assert_array_equal(converted.positions.numpy(), [[1, 2, 3]]) + # A consumer can rebind its wrapper without changing another consumer's arrays. + converted.positions = None + assert provider.get_transforms(converted) + assert converted.positions is other.positions + + data.transforms.assign([[4, 5, 6, 0, 0, 0, 1]]) + backend.transforms_version += 1 + assert provider.get_transforms(converted) + assert converted.positions is other.positions + assert launch.call_count == 2 + np.testing.assert_array_equal(converted.positions.numpy(), [[4, 5, 6]]) + + data.transforms = wp.array([[7, 8, 9, 0, 0, 0, 1]], dtype=wp.transformf, device="cpu") + backend.transforms_version += 1 + assert provider.get_transforms(native) + assert native.transforms is data.transforms + assert provider.get_transforms(converted) + assert converted.positions is other.positions + assert launch.call_count == 3 + np.testing.assert_array_equal(converted.positions.numpy(), [[7, 8, 9]]) + + peer = SceneDataProvider(backend) + peer_output = SceneDataFormat.Vec3_Quat() + assert peer.get_transforms(peer_output) + for position in ([10, 11, 12], [13, 14, 15]): + data.transforms.assign([position + [0, 0, 0, 1]]) + backend.transforms_version += 1 + assert provider.get_transforms(converted) + assert peer.get_transforms(peer_output) + np.testing.assert_array_equal(converted.positions.numpy(), [position]) + np.testing.assert_array_equal(peer_output.positions.numpy(), [position]) + + +@pytest.mark.parametrize("format_name", ["Transform", "Vec3_Quat"]) +def test_owned_transform_buffers_are_written_directly_and_do_not_alias_cache(format_name, monkeypatch): + data = SceneDataFormat.Transform() + data.transforms = wp.array([[1, 2, 3, 0, 0, 0, 1]], dtype=wp.transformf, device="cpu") + backend = _Backend(transforms=data, transforms_version=0, transform_count=1) + provider = SceneDataProvider(backend) + shared, owned = (getattr(SceneDataFormat, format_name)() for _ in range(2)) + assert provider.get_transforms(shared) + provider.init_output(owned) + arrays = [getattr(owned, name) for name in owned._cls.vars] + launch, copy = Mock(wraps=wp.launch), Mock(wraps=wp.copy) + monkeypatch.setattr(wp, "launch", launch) + monkeypatch.setattr(wp, "copy", copy) + for x in (4, 7): + data.transforms.assign([[x, 5, 6, 0, 0, 0, 1]]) + backend.transforms_version += 1 + launch.reset_mock() + copy.reset_mock() + assert provider.get_transforms(owned, allow_passthrough=False) + assert launch.call_count == int(format_name != "Transform") + assert copy.call_count == int(format_name == "Transform") + assert provider.get_transforms(shared) + for name, array in zip(owned._cls.vars, arrays): + assert getattr(owned, name) is array + assert array is not getattr(shared, name) + np.testing.assert_array_equal(array.numpy(), getattr(shared, name).numpy()) + + +def test_mapping_preserves_unmapped_destination_slots(): + data = SceneDataFormat.Transform() + data.transforms = wp.array([[1, 2, 3, 0, 0, 0, 1], [4, 5, 6, 0, 0, 0, 1]], dtype=wp.transformf, device="cpu") + provider = SceneDataProvider( + _Backend(transforms=data, transforms_version=0, transform_count=2, transform_paths=["/a", "/b"]) + ) + mapping = provider.create_mapping(["/a", "/b", None]) + output = SceneDataFormat.Transform() + output.transforms = wp.zeros(3, dtype=wp.transformf, device="cpu") + assert provider.get_transforms(output, mapping, allow_passthrough=False, count=3) + np.testing.assert_array_equal(output.transforms.numpy()[:2], data.transforms.numpy()) + np.testing.assert_array_equal(output.transforms.numpy()[2], np.zeros(7)) + + +@pytest.mark.parametrize("format_name", ["Transform", "Vec3_Quat", "Vec3_Matrix33", "Matrix44"]) +@pytest.mark.parametrize("scaled", [False, True]) +def test_transposed_matrices_fuse_format_mapping_and_scale(format_name, scaled): + """All native formats produce the same row-vector matrices, with output-indexed scale.""" + poses = np.array([[1, 2, 3, 0, 0, 0, 1], [4, 5, 6, 1, 2, 2, 2]], dtype=np.float32) + poses[1, 3:] /= np.sqrt(13) + # Independent quaternion-to-matrix reference; a non-axis rotation exposes transpose/scale ordering errors. + rotations = np.array([np.eye(3), np.array([[-3, -4, 12], [12, 3, 4], [-4, 12, 3]]) / 13], dtype=np.float32) + matrices = np.broadcast_to(np.eye(4), (2, 4, 4)).copy() + matrices[:, :3, :3] = rotations + matrices[:, :3, 3] = poses[:, :3] + data = getattr(SceneDataFormat, format_name)() + if format_name == "Transform": + data.transforms = wp.array(poses, dtype=wp.transformf, device="cpu") + elif format_name == "Matrix44": + data.matrices = wp.array(matrices, dtype=wp.mat44f, device="cpu") + else: + data.positions = wp.array(poses[:, :3], dtype=wp.vec3f, device="cpu") + data.orientations = wp.array( + poses[:, 3:] if format_name == "Vec3_Quat" else rotations, + dtype=wp.quatf if format_name == "Vec3_Quat" else wp.mat33f, + device="cpu", + ) + provider = SceneDataProvider(_Backend(transforms=data, transforms_version=0, transform_count=2)) + mapping = wp.array([1, 0], dtype=wp.int32, device="cpu") + scales = wp.array([[2, 3, 4], [5, 6, 7]], dtype=wp.vec3f, device="cpu") if scaled else None + output = SceneDataFormat.TransposedMatrix44d() + assert provider.get_transforms(output, mapping, scales=scales) + expected = matrices[::-1].transpose(0, 2, 1).copy() + if scaled: + expected[:, :3, :3] *= scales.numpy()[:, :, None] + np.testing.assert_allclose(output.matrices.numpy(), expected, rtol=1.0e-6, atol=1.0e-6) + matrices = output.matrices + assert provider.get_transforms(output, mapping, scales=scales) + assert output.matrices is matrices + + +@pytest.mark.parametrize("format_name", ["Transform", "Vec3_Quat", "Vec3_Matrix33", "Matrix44"]) +@pytest.mark.parametrize("device", test_devices()) +def test_fabric_conversion_preserves_scale_and_refreshes_reallocated_destinations(format_name, device, monkeypatch): + """Fabric conversion skips solver-only bodies and preserves scales across buffer reallocations.""" + assert set(SceneDataFormat.FabricMatrix44.vars) == {"matrices"} + poses = np.array([[1, 2, 3, 0, 0, 0, 1], [7, 8, 9, 0, 0, 0, 1], [4, 5, 6, 1, 2, 2, 2]], dtype=np.float32) + poses[2, 3:] /= np.sqrt(13) + data = SceneDataFormat.Transform() + data.transforms = wp.array(poses, dtype=wp.transformf, device=device) + native = SceneDataProvider(_Backend(transforms=data, transforms_version=0, transform_count=len(poses))) + source = getattr(SceneDataFormat, format_name)() + assert native.get_transforms(source) + provider = SceneDataProvider(_Backend(transforms=source, transforms_version=0, transform_count=len(poses))) + authored_scales = np.array([[5, 6, 7], [1, 1, 1], [2, 3, 4]], dtype=np.float32) + scales = wp.array(authored_scales, dtype=wp.vec3f, device=device) + expected = np.array([np.eye(4), np.diag([5, 6, 7, 1])], dtype=np.float64) + rotation = np.array([[-3, -4, 12], [12, 3, 4], [-4, 12, 3]]) / 13 + expected[0, :3, :3] = np.diag([2, 3, 4]) @ rotation.T + expected[:, 3, :3] = [[4, 5, 6], [1, 2, 3]] + indices = wp.array([len(poses) - 1, 0], dtype=wp.int32, device=device) + launch = Mock(wraps=wp.launch) + monkeypatch.setattr(wp, "launch", launch) + for _ in range(2): + matrices = wp.empty(2, dtype=wp.mat44d, device=device) + interface = { + "version": 1, + "device": device, + "attribs": { + "mapping": { + "type": (True, "i4", 1, 0, ""), + "access": 1, + "pointers": [indices.ptr], + "counts": [2], + }, + "matrices": { + "type": (True, "f8", 16, 0, "matrix"), + "access": 2, + "pointers": [matrices.ptr], + "counts": [2], + }, + }, + } + storage = SimpleNamespace(__fabric_arrays_interface__=interface) + output = SceneDataFormat.FabricMatrix44() + output.matrices = wp.fabricarray(storage, "matrices") + mapping = wp.fabricarray(storage, "mapping") + destination = output.matrices + launch.reset_mock() + assert provider.get_transforms(output, mapping, scales=scales) + assert provider.get_transforms(output, mapping, scales=scales) + assert output.matrices is destination + launch.assert_called_once() + assert len(provider._transform_cache) == 1 + np.testing.assert_allclose(matrices.numpy(), expected, rtol=1.0e-6, atol=1.0e-6) + np.testing.assert_array_equal(scales.numpy(), authored_scales) diff --git a/source/isaaclab/test/sim/test_newton_manager_visualization_state.py b/source/isaaclab/test/sim/test_newton_manager_visualization_state.py index f5ca56085da3..9b5be2fa473a 100644 --- a/source/isaaclab/test/sim/test_newton_manager_visualization_state.py +++ b/source/isaaclab/test/sim/test_newton_manager_visualization_state.py @@ -236,6 +236,7 @@ def test_visualization_model_is_built_during_clone_and_allocated_on_physics_read monkeypatch, body_count, particle_count ): """Cloning owns parsing; READY owns native allocation; getters never discover or allocate.""" + import warp as wp from isaaclab_newton.cloner import NewtonReplicateContext from isaaclab_newton.cloner import replicate as replicate_module from isaaclab_newton.physics import NewtonManager @@ -244,6 +245,7 @@ def test_visualization_model_is_built_during_clone_and_allocated_on_physics_read from pxr import Usd, UsdGeom from isaaclab.physics import PhysicsEvent, PhysicsManager + from isaaclab.scene_data import SceneDataFormat, SceneDataProvider from isaaclab.sim import SimulationContext class ForeignPhysicsManager(PhysicsManager): @@ -265,15 +267,30 @@ class ForeignPhysicsManager(PhysicsManager): UsdGeom.Xform.Define(sim.stage, "/Scene/Source") sim.physics_manager = ForeignPhysicsManager sim._backend_registry = [] - sim._scene_data_provider = SimpleNamespace(backend=object(), point_count=0) + body_paths = [f"/Scene/Body_{index}" for index in range(body_count)] + transforms = SceneDataFormat.Transform() + transforms.transforms = wp.zeros(body_count, dtype=wp.transformf, device="cpu") + sim._scene_data_provider = SceneDataProvider( + SimpleNamespace( + transforms=transforms, + get_transforms=lambda _format: transforms, + transforms_version=0, + transform_paths=body_paths, + transform_count=body_count, + point_count=0, + ) + ) monkeypatch.setattr(SimulationContext, "_instance", sim) finalize = Mock( side_effect=lambda device: SimpleNamespace( body_count=body_count, + body_label=body_paths, particle_count=particle_count, world_count=2, - state=lambda: SimpleNamespace(body_q=None, particle_q=None), + state=lambda: SimpleNamespace( + body_q=wp.empty(body_count, dtype=wp.transformf, device="cpu") if body_count else None, particle_q=None + ), ) ) monkeypatch.setattr(ModelBuilder, "finalize", finalize) @@ -291,6 +308,8 @@ class ForeignPhysicsManager(PhysicsManager): assert not sim._backend_registry ForeignPhysicsManager.dispatch_event(PhysicsEvent.PHYSICS_READY) + if body_count: + assert NewtonManager.get_state_0().body_q is transforms.transforms first_model = NewtonManager.get_model() first_state = NewtonManager.get_state() ForeignPhysicsManager.dispatch_event(PhysicsEvent.PHYSICS_READY) @@ -310,67 +329,65 @@ class ForeignPhysicsManager(PhysicsManager): ForeignPhysicsManager.dispatch_event(PhysicsEvent.STOP) -def test_update_visualization_state_noop_when_backend_is_newton(monkeypatch): - """When sim backend is Newton, update_visualization_state is a no-op.""" - from isaaclab_newton.physics import NewtonManager - - _reset_newton_manager_state() - monkeypatch.setattr(NewtonManager, "_backend_is_newton", classmethod(lambda cls, scene_data_provider=None: True)) - monkeypatch.setattr(NewtonManager, "get_scene_data_provider", classmethod(lambda cls: SimpleNamespace())) - - # Pre-set sentinel values to ensure update doesn't touch them. - monkeypatch.setattr(NewtonManager, "backend", SimpleNamespace(model="live-model", state_0="live-state")) - NewtonManager.update_visualization_state() - assert NewtonManager.backend.model == "live-model" - assert NewtonManager.backend.state_0 == "live-state" - - -@pytest.mark.parametrize("newton_active", [True, False]) -def test_get_state_forwards_only_for_live_newton_state(monkeypatch, newton_active): - """PhysX shadow state keeps its visualization update without entering Newton FK.""" - from isaaclab_newton.physics import NewtonManager - - events: list[str] = [] - state = object() - monkeypatch.setattr(NewtonManager, "_fk_reset_mask", object(), raising=False) - monkeypatch.setattr( - NewtonManager, - "_backend_is_newton", - classmethod(lambda cls, provider=None: newton_active), - ) - monkeypatch.setattr(NewtonManager, "forward", classmethod(lambda cls: events.append("forward"))) - monkeypatch.setattr( - NewtonManager, - "update_visualization_state", - classmethod(lambda cls, provider=None: events.append("visualization")), - ) - monkeypatch.setattr(NewtonManager, "get_state_0", classmethod(lambda cls: state)) - - assert NewtonManager.get_state() is state - expected = ["forward", "visualization"] if newton_active else ["visualization"] - assert events == expected - - -def test_scene_data_reads_through_public_state_boundary(monkeypatch): - """SceneData does not bypass the coherent Newton state accessor.""" +@pytest.mark.parametrize("invalidate", ["invalidate_body_state", "invalidate_fk"]) +def test_native_publication_reuses_clean_fk_and_refreshes_writes_and_swaps(monkeypatch, invalidate): + """Clean native reads reuse FK and conversions; writes and solver-buffer swaps refresh their values.""" import warp as wp - from isaaclab_newton.physics import NewtonManager - from isaaclab_newton.physics import newton_manager as nm + from isaaclab_newton.physics import NewtonManager, NewtonXPBDManager + from isaaclab_newton.physics.newton_manager import NewtonSceneDataBackend + + from isaaclab.physics import PhysicsManager + from isaaclab.scene_data import SceneDataFormat, SceneDataProvider - events: list[str] = [] - body_q = wp.zeros(1, dtype=wp.transformf, device="cpu") - state = SimpleNamespace(body_q=body_q) - backend = nm.NewtonSceneDataBackend() + _reset_newton_manager_state() + monkeypatch.setattr(PhysicsManager, "_device", "cpu") + state = SimpleNamespace(body_q=wp.array([[0, 0, 0, 0, 0, 0, 1]], dtype=wp.transformf, device="cpu")) + backend = NewtonSceneDataBackend() + provider = SceneDataProvider(backend) monkeypatch.setattr( - NewtonManager, - "get_state", - classmethod(lambda cls, provider=None: events.append("state") or state), + NewtonManager, "backend", SimpleNamespace(model=SimpleNamespace(body_count=1, world_count=1), state_0=state) ) - - transforms = backend.transforms - - assert events == ["state"] - assert transforms.transforms is body_q + monkeypatch.setattr(NewtonManager, "_scene_data_backend", backend) + monkeypatch.setattr(NewtonManager, "_world_reset_mask", wp.zeros(2, dtype=wp.bool, device="cpu")) + monkeypatch.setattr(NewtonManager, "_fk_reset_mask", wp.zeros(1, dtype=wp.bool, device="cpu")) + # Fabric may bind between native allocation and the solver's FK-hook initialization. + assert backend.transforms.transforms is state.body_q + monkeypatch.setattr(NewtonManager, "_eval_fk", Mock()) + monkeypatch.setattr(NewtonManager, "_reset_solver_internals_delegate", Mock()) + monkeypatch.setattr(wp, "launch", Mock(wraps=wp.launch)) + + output = SceneDataFormat.Matrix44() + assert provider.get_transforms(output) + matrices = output.matrices + NewtonManager.pre_render() + NewtonManager._eval_fk.assert_not_called() + NewtonManager.get_state(provider) + assert provider.get_transforms(output) + assert output.matrices is matrices + assert wp.launch.call_count == 1 + NewtonManager._eval_fk.assert_not_called() + + state.body_q.assign([[1, 2, 3, 0, 0, 0, 1]]) + getattr(NewtonXPBDManager, invalidate)() + assert provider.get_transforms(output) + assert output.matrices is matrices + np.testing.assert_allclose(output.matrices.numpy()[0, :3, 3], [1, 2, 3]) + NewtonManager._eval_fk.assert_called_once() + assert provider.get_transforms(output) + assert output.matrices is matrices + NewtonManager.pre_render() + NewtonManager._eval_fk.assert_called_once() + assert wp.launch.call_count == 2 + + replacement = wp.array([[3, 2, 1, 0, 0, 0, 1]], dtype=wp.transformf, device="cpu") + NewtonManager.backend.state_0 = SimpleNamespace(body_q=replacement) + native = SceneDataFormat.Transform() + assert provider.get_transforms(native) + assert native.transforms is replacement + assert provider.get_transforms(output) + assert output.matrices is matrices + assert wp.launch.call_count == 3 + np.testing.assert_allclose(output.matrices.numpy()[0, :3, 3], [3, 2, 1]) def test_resolve_scene_data_body_paths_uses_joint_body_targets(): @@ -392,8 +409,9 @@ def test_resolve_scene_data_body_paths_uses_joint_body_targets(): assert resolved_paths == ["/World/envs/env_0/Robot/robot0_forearm"] -def test_update_visualization_state_copies_identity_mapped_transforms(monkeypatch): - """Identity-mapped transforms update the persistent Newton shadow buffer.""" +@pytest.mark.parametrize("layout", ["identity", "reordered", "missing", "duplicate"]) +def test_update_visualization_state_shares_sdp_transforms(monkeypatch, layout): + """Native and reordered layouts bind shared output once and refresh only on publication.""" import numpy as np import warp as wp from isaaclab_newton.physics import NewtonManager @@ -404,6 +422,12 @@ def test_update_visualization_state_copies_identity_mapped_transforms(monkeypatc monkeypatch.setattr(NewtonManager, "_backend_is_newton", classmethod(lambda cls, provider=None: False)) body_paths = ["/World/envs/env_0/Object", "/World/envs/env_1/Object"] + render_paths = { + "identity": body_paths, + "reordered": body_paths[::-1], + "missing": [body_paths[0], "/World/Missing"], + "duplicate": [body_paths[0], body_paths[0]], + }[layout] source_transforms = wp.array( [ [1.0, 2.0, 3.0, 0.0, 0.0, 0.0, 1.0], @@ -414,31 +438,46 @@ def test_update_visualization_state_copies_identity_mapped_transforms(monkeypatc ) source_data = SceneDataFormat.Transform() source_data.transforms = source_transforms - provider_impl = SceneDataProvider( - SimpleNamespace(transforms=source_data, transform_paths=body_paths, transform_count=len(body_paths)) - ) - provider = SimpleNamespace( - usd_stage=None, - create_mapping=provider_impl.create_mapping, - get_transforms=provider_impl.get_transforms, - point_count=0, + provider = SceneDataProvider( + SimpleNamespace( + transforms=source_data, + get_transforms=lambda _format: source_data, + transforms_version=0, + transform_paths=body_paths, + transform_count=len(body_paths), + point_count=0, + ) ) + monkeypatch.setattr(SceneDataProvider, "usd_stage", property(lambda self: None)) destination = wp.zeros(len(body_paths), dtype=wp.transformf, device="cpu") monkeypatch.setattr( NewtonManager, "backend", SimpleNamespace( - model=SimpleNamespace(body_label=body_paths, body_count=len(body_paths)), + model=SimpleNamespace(body_label=render_paths, body_count=len(body_paths)), state_0=SimpleNamespace(body_q=destination, particle_q=None), ), ) + if layout in ("missing", "duplicate"): + with pytest.raises(ValueError, match="one unique SDP transform path"): + NewtonManager.update_visualization_state(provider) + return + + remapped = layout == "reordered" NewtonManager.update_visualization_state(provider) + shared = NewtonManager.get_state(provider).body_q + assert (shared is source_transforms) is not remapped + np.testing.assert_allclose(shared.numpy(), source_transforms.numpy()[:: -1 if remapped else 1]) + assert NewtonManager.get_state(provider).body_q is shared - assert NewtonManager.backend.state_0.body_q is destination - assert NewtonManager._scene_data.transforms is destination - np.testing.assert_allclose(destination.numpy(), source_transforms.numpy()) + source_data.transforms = wp.array(source_transforms.numpy() + 1.0, dtype=wp.transformf, device="cpu") + provider.backend.transforms_version += 1 + NewtonManager.update_visualization_state(provider) + np.testing.assert_allclose( + NewtonManager.backend.state_0.body_q.numpy(), source_data.transforms.numpy()[:: -1 if remapped else 1] + ) def test_update_visualization_state_syncs_shadow_particle_q(monkeypatch): @@ -460,14 +499,13 @@ def test_update_visualization_state_syncs_shadow_particle_q(monkeypatch): [2], ) ) - monkeypatch.setattr(provider, "get_transforms", lambda output, mapping=None, allow_passthrough=True: True) particle_q = wp.zeros(2, dtype=wp.vec3f, device="cpu") NewtonManager = _prepare_physx_shadow_sync( monkeypatch, provider, model=SimpleNamespace(body_label=["/World/envs/env_0/Robot"]), - state_0=SimpleNamespace(body_q=wp.zeros(1, dtype=wp.transformf, device="cpu"), particle_q=particle_q), + state_0=SimpleNamespace(body_q=None, particle_q=particle_q), entities=[_make_shadow_entity(cloth_path, sim_particle_count=2)], sim_particle_count=2, ) @@ -504,14 +542,13 @@ def test_update_visualization_state_remaps_volume_vis_positions(monkeypatch): [4], ) ) - monkeypatch.setattr(provider, "get_transforms", lambda output, mapping=None, allow_passthrough=True: True) particle_q = wp.zeros(1, dtype=wp.vec3f, device="cpu") NewtonManager = _prepare_physx_shadow_sync( monkeypatch, provider, model=SimpleNamespace(body_label=["/World/envs/env_0/Robot"]), - state_0=SimpleNamespace(body_q=wp.zeros(1, dtype=wp.transformf, device="cpu"), particle_q=particle_q), + state_0=SimpleNamespace(body_q=None, particle_q=particle_q), entities=[ _make_shadow_entity( soft_path, @@ -542,7 +579,6 @@ def test_sync_skips_unmapped_deformable_rest_pose(monkeypatch): [2], ) ) - monkeypatch.setattr(provider, "get_transforms", lambda output, mapping=None, allow_passthrough=True: True) particle_q = wp.array( [ @@ -558,7 +594,7 @@ def test_sync_skips_unmapped_deformable_rest_pose(monkeypatch): monkeypatch, provider, model=SimpleNamespace(body_label=["/World/envs/env_0/Robot"]), - state_0=SimpleNamespace(body_q=wp.zeros(1, dtype=wp.transformf, device="cpu"), particle_q=particle_q), + state_0=SimpleNamespace(body_q=None, particle_q=particle_q), entities=[ _make_shadow_entity("/World/envs/env_0/ClothA", sim_particle_count=2), _make_shadow_entity( @@ -596,7 +632,6 @@ def test_sync_skips_mismatched_volume_without_remap(monkeypatch): [4], ) ) - monkeypatch.setattr(provider, "get_transforms", lambda output, mapping=None, allow_passthrough=True: True) # Vis-sized render buffer initialized to a sentinel rest pose. particle_q = wp.array([wp.vec3(9.0, 9.0, 9.0)], dtype=wp.vec3f, device="cpu") @@ -604,7 +639,7 @@ def test_sync_skips_mismatched_volume_without_remap(monkeypatch): monkeypatch, provider, model=SimpleNamespace(body_label=["/World/envs/env_0/Robot"]), - state_0=SimpleNamespace(body_q=wp.zeros(1, dtype=wp.transformf, device="cpu"), particle_q=particle_q), + state_0=SimpleNamespace(body_q=None, particle_q=particle_q), entities=[ _make_shadow_entity( soft_path, diff --git a/source/isaaclab_newton/changelog.d/sdp-transform-transport.rst b/source/isaaclab_newton/changelog.d/sdp-transform-transport.rst new file mode 100644 index 000000000000..a31eaeb5f264 --- /dev/null +++ b/source/isaaclab_newton/changelog.d/sdp-transform-transport.rst @@ -0,0 +1,11 @@ +Changed +^^^^^^^ + +* Shared Newton rigid-body transforms through SceneDataProvider publications, including solver state-buffer + swaps, and moved Fabric bindings into the Kit rendering integration. Explicit Fabric synchronization + continued to work without a Kit viewer or RTX camera. Newton render-only states under foreign + physics now reference shared SDP transforms instead of copying them; consumers must treat their + ``body_q`` arrays as read-only. Particle and cable synchronization remained unchanged. +* Reconciled authored state writes only while pending, instead of re-running forward kinematics for + every new transform publication. Rendering requested rigid Fabric updates through SDP rather + than the physics pre-render hook. Captured external writes retained conservative reconciliation. diff --git a/source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py b/source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py index 217acca0084c..94007a30365c 100644 --- a/source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py +++ b/source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py @@ -142,53 +142,6 @@ def _compile_label_pattern(expr: str | list[str] | None) -> re.Pattern[str] | No # _LocalSite: (None, [[env0_idx, ...], ...]) — per-world site indices -@wp.kernel(enable_backward=False) -def _capture_fabric_scales( - fabric_transforms: wp.fabricarray(dtype=wp.mat44d), - newton_indices: wp.fabricarray(dtype=wp.uint32), - body_scales: wp.array(dtype=wp.vec3f), -): - """Capture initialized Fabric world scales by Newton body index.""" - i = int(wp.tid()) - idx = int(newton_indices[i]) - matrix = wp.mat44f(fabric_transforms[i]) - body_scales[idx] = wp.vec3f( - wp.length(wp.vec3f(matrix[0, 0], matrix[0, 1], matrix[0, 2])), - wp.length(wp.vec3f(matrix[1, 0], matrix[1, 1], matrix[1, 2])), - wp.length(wp.vec3f(matrix[2, 0], matrix[2, 1], matrix[2, 2])), - ) - - -@wp.kernel(enable_backward=False) -def _set_fabric_transforms( - fabric_transforms: wp.fabricarray(dtype=wp.mat44d), - newton_indices: wp.fabricarray(dtype=wp.uint32), - newton_body_q: wp.array(ndim=1, dtype=wp.transformf), - body_scales: wp.array(dtype=wp.vec3f), -): - """Write Newton body poses to Fabric world matrices with their initialized scale. - - For each Fabric prim at thread ``i``, reads the Newton body transform at - ``newton_body_q[newton_indices[i]]`` and combines its translation and rotation - with the corresponding scale captured from the initialized Fabric world matrix. - Newton transforms do not carry scale, so reapplying the captured value prevents - authored USD scale from being overwritten with unit scale during rendering sync. - """ - i = int(wp.tid()) - idx = int(newton_indices[i]) - transform = newton_body_q[idx] - scale = body_scales[idx] - fabric_transforms[i] = wp.mat44d( - wp.transpose( - wp.transform_compose( - wp.transform_get_translation(transform), - wp.transform_get_rotation(transform), - scale, - ) - ) - ) - - @wp.kernel(enable_backward=False) def _sync_particle_points( fabric_points: wp.fabricarrayarray(dtype=wp.vec3f), @@ -351,13 +304,17 @@ class NewtonSceneDataBackend(SceneDataBackend): """ def __init__(self): - self._scene_data = SceneDataFormat.Transform() + self._transforms = SceneDataFormat.Transform() + self.transforms_version = 0 @property def transforms(self) -> SceneDataFormat.Transform: - """Return the current Newton rigid body transforms as :class:`SceneDataFormat.Transform`.""" - self._scene_data.transforms = self.state.body_q - return self._scene_data + """Publish the authoritative native pointer, including solver state-buffer swaps.""" + transforms = self.state.body_q + if self._transforms.transforms is not transforms: + self._transforms.transforms = transforms + self.transforms_version += 1 + return self._transforms @property def transform_count(self) -> int: @@ -377,8 +334,13 @@ def model(self) -> Model: @property def state(self) -> State: - """Return Newton state after applying pending forward kinematics.""" - return NewtonManager.get_state() + """Return native physics state without entering the rendering consumer path.""" + if NewtonManager._transforms_may_change_on_graph_replay: + # Raw external graph replays bypass Python invalidation, so these reads must stay conservative. + self.transforms_version += 1 + if NewtonManager._eval_fk is not _eval_fk_unbound: + NewtonManager.forward() + return NewtonManager.get_state_0() def _eval_fk_unbound(world_reset_mask: wp.array | None, fk_mask: wp.array | None) -> None: @@ -465,6 +427,7 @@ class NewtonManager(PhysicsManager): # Newton reserves the final slot for global entities in world -1. _world_reset_mask: wp.array | None = None # (num_envs + 1,) wp.bool _fk_reset_mask: wp.array | None = None # (articulation_count,) wp.bool — for eval_fk(mask=...) + _reconciliation_pending: bool = False # Solver-specialized FK delegate. Bound in initialize_solver() to the active subclass's choice of FK implementation. _eval_fk: Callable[[wp.array | None, wp.array | None], None] = _eval_fk_unbound # Solver-specialized reset delegate. Like _eval_fk, this must dispatch correctly through the base manager. @@ -500,13 +463,8 @@ class NewtonManager(PhysicsManager): _sensor_bvh_shape_flags: ShapeFlags = ShapeFlags.VISIBLE # USD/Fabric sync - _newton_stage_path = None _usdrt_stage = None - _newton_index_attr = "newton:index" - # Body-indexed world scales captured before Newton first overwrites Fabric transforms. - _fabric_body_scales: wp.array | None = None _clone_physics_only = False - _transforms_dirty: bool = False _transforms_may_change_on_graph_replay: bool = False _particles_dirty: bool = False _cables_dirty: bool = False @@ -518,15 +476,6 @@ class NewtonManager(PhysicsManager): _newton_particle_count_attr = "newton:particleCount" _particle_visual_prims: dict[str, _ParticleVisualPrim] = {} - # Cached after the first fabric sync that probes IFabricHierarchy GPU APIs. - _use_fabric_gpu_hierarchy: bool | None = None - - # Set to True after sync_transforms_to_fabric() successfully writes body positions for - # the first time in each simulation session. Reset to False in clear(). Polled by - # test drain helpers to know when the GPU has propagated the newton:index Fabric - # attribute and body_q values are valid. - _newton_fabric_ready: bool = False - # Model changes (callbacks use unified system from PhysicsManager) _model_changes: set[int] = set() @@ -536,8 +485,8 @@ class NewtonManager(PhysicsManager): # Visualization-only state used when the sim backend is PhysX. Populated # from the clone plan in :meth:`_initialize_visualization_model` and updated each render # frame in :meth:`update_visualization_state`. - _scene_data: SceneDataFormat.Transform | None = None _scene_data_mapping: wp.array | None = None + _scene_data_version: int | None = None _scene_data_points: SceneDataFormat.Points | None = None _scene_data_geometry_mapping: wp.array | None = None _shadow_deformable_entities: list | None = None @@ -606,7 +555,7 @@ def initialize(cls, sim_context: SimulationContext) -> None: cameras_enabled = bool(get_settings_manager().get("/isaaclab/cameras_enabled", False)) cls._clone_physics_only = not has_kit() or ("kit" not in requested and not cameras_enabled) - cls._scene_data_backend = NewtonSceneDataBackend() + NewtonManager._scene_data_backend = NewtonSceneDataBackend() @classmethod def reset(cls, soft: bool = False) -> None: @@ -633,6 +582,8 @@ def reset(cls, soft: bool = False) -> None: NewtonManager._collision_pipeline = None NewtonManager._contacts = None NewtonManager._solver = None + NewtonManager._eval_fk = _eval_fk_unbound + NewtonManager._reset_solver_internals_delegate = _reset_solver_internals_unbound NewtonManager._adapter = None cls._invalidate_sensor_graph() NewtonManager._sensor_state = None @@ -675,12 +626,17 @@ def forward(cls) -> None: data layer invokes ``NewtonManager.forward()`` on the base class, where ``cls`` is the base ``NewtonManager``; the bound delegate dispatches to the concrete subclass override. """ + if cls._eval_fk is not _eval_fk_unbound and not ( + cls._reconciliation_pending or cls._transforms_may_change_on_graph_replay + ): + return cls._reset_solver_internals_delegate(cls._world_reset_mask) cls._eval_fk(cls._world_reset_mask, cls._fk_reset_mask) if cls._fk_reset_mask is not None: cls._fk_reset_mask.zero_() if cls._world_reset_mask is not None: cls._world_reset_mask.zero_() + NewtonManager._reconciliation_pending = False cls._mark_sensor_state_dirty() @classmethod @@ -690,142 +646,17 @@ def video_capture_backend(cls) -> str: @classmethod def pre_render(cls) -> None: - """Refresh derived Newton state before cameras and visualizers read it.""" - if cls._fk_reset_mask is not None: - cls.forward() - if NewtonManager._transforms_may_change_on_graph_replay: - cls._mark_transforms_dirty() - cls.sync_transforms_to_fabric() + """Refresh legacy cable and particle geometry; rigid transforms are requested through SDP.""" cls.sync_cables_to_usd() cls.sync_particles_to_usd() @classmethod def sync_transforms_to_fabric(cls) -> None: - """Write Newton body_q to Fabric world matrices for Kit viewport / RTX rendering. - - The write lands in Fabric only. Authored USD attributes are left untouched, so the - poses are visible to the RTX renderer but absent from a stage export or save. - - No-op when ``_usdrt_stage`` is None (i.e. Kit visualizer is not active) - or when transforms have not changed since the last sync. - - Called at render cadence by :meth:`pre_render` (via - :meth:`~isaaclab.sim.SimulationContext.render`). - Physics stepping marks transforms dirty via :meth:`_mark_transforms_dirty` - so that the expensive Fabric hierarchy update only runs once per render - frame rather than after every physics step. - - Uses ``wp.fabricarray`` directly (no ``isaacsim.physics.newton`` extension needed). - On the first successful sync, a Warp kernel captures each initialized Fabric - world scale by Newton body index. The pose kernel then combines that scale with - ``state_0.body_q[newton_index[i]]`` and writes the corresponding ``mat44d`` to - ``omni:fabric:worldMatrix`` for each prim. - - When ``IFabricHierarchy.update_world_xforms_gpu_with_options`` is - available the method mirrors PhysX's ``DirectGpuHelper`` pattern: pause - Fabric change tracking, write transforms, resume tracking, then run the - GPU hierarchy update with ``RIGID_BODY | FORCE_UPDATE`` so Newton-authored - world matrices stay authoritative on rigid-body prims. Otherwise it - falls back to the CPU ``update_world_xforms()`` path. - """ + """Publish rigid-body poses through SDP to Fabric, leaving authored USD untouched.""" if cls._usdrt_stage is None or cls.backend is None: return - if not cls._transforms_dirty: - return - try: - import usdrt - - fabric_hierarchy = None - gpu_opts_cls = None - if hasattr(usdrt, "hierarchy"): - fabric_hierarchy = usdrt.hierarchy.IFabricHierarchy().get_fabric_hierarchy( - cls._usdrt_stage.GetFabricId(), cls._usdrt_stage.GetStageIdAsStageId() - ) - gpu_opts_cls = getattr(usdrt.hierarchy, "FabricHierarchyGpuUpdateOptions", None) - - if cls._use_fabric_gpu_hierarchy is None and hasattr(usdrt, "hierarchy"): - # Probe the pybind class once so a transient null hierarchy handle does - # not permanently disable the GPU path for the session. - NewtonManager._use_fabric_gpu_hierarchy = gpu_opts_cls is not None and hasattr( - usdrt.hierarchy.IFabricHierarchy, "update_world_xforms_gpu_with_options" - ) - if cls._use_fabric_gpu_hierarchy: - logger.info("Fabric GPU transform hierarchy enabled via IFabricHierarchy") - else: - logger.info("Fabric GPU transform hierarchy unavailable; falling back to update_world_xforms()") - - use_gpu_hierarchy = bool( - cls._use_fabric_gpu_hierarchy and fabric_hierarchy is not None and gpu_opts_cls is not None - ) - - # Pause hierarchy change tracking BEFORE SelectPrims. - # SelectPrims with ReadWrite access calls getAttributeArrayGpu - # internally, which marks Fabric buffers dirty. If tracking is - # still active at that point the hierarchy records the change and - # Kit's updateWorldXforms will do an expensive connectivity - # rebuild every frame. PhysX avoids this via ScopedUSDRT which - # pauses tracking before any Fabric writes. - if use_gpu_hierarchy: - fabric_hierarchy.track_world_xform_changes(False) - fabric_hierarchy.track_local_xform_changes(False) - - try: - selection = cls._usdrt_stage.SelectPrims( - require_attrs=[ - (usdrt.Sdf.ValueTypeNames.Matrix4d, "omni:fabric:worldMatrix", usdrt.Usd.Access.ReadWrite), - (usdrt.Sdf.ValueTypeNames.UInt, cls._newton_index_attr, usdrt.Usd.Access.Read), - ], - device=str(PhysicsManager._device), - ) - if selection.GetCount() == 0: - # The newton:index attribute is written CPU-side by start_simulation() but - # GPU propagation is deferred. Keep _transforms_dirty=True so the next - # pre_render() retries once initialize_solver() has completed (FK delegate - # bound) and body_q holds valid values. - if cls._eval_fk is _eval_fk_unbound: - NewtonManager._transforms_dirty = False - return - - fabric_transforms = wp.fabricarray(selection, "omni:fabric:worldMatrix") - newton_indices = wp.fabricarray(selection, cls._newton_index_attr) - if cls._fabric_body_scales is None: - NewtonManager._fabric_body_scales = wp.empty( - cls.backend.model.body_count, - dtype=wp.vec3f, - device=PhysicsManager._device, - ) - wp.launch( - _capture_fabric_scales, - dim=newton_indices.shape[0], - inputs=[fabric_transforms, newton_indices, cls._fabric_body_scales], - device=PhysicsManager._device, - ) - wp.launch( - _set_fabric_transforms, - dim=newton_indices.shape[0], - inputs=[fabric_transforms, newton_indices, cls.backend.state_0.body_q, cls._fabric_body_scales], - device=PhysicsManager._device, - ) - wp.synchronize_device(PhysicsManager._device) - - NewtonManager._newton_fabric_ready = True - NewtonManager._transforms_dirty = False - - if use_gpu_hierarchy: - # RIGID_BODY: inverse-propagate on PhysicsRigidBodyAPI buckets - # (keep Newton world matrices, derive local). FORCE_UPDATE: - # bypass the change-listener dirty check after tracking pause. - fabric_hierarchy.update_world_xforms_gpu_with_options( - gpu_opts_cls.RIGID_BODY | gpu_opts_cls.FORCE_UPDATE - ) - elif fabric_hierarchy is not None: - fabric_hierarchy.update_world_xforms() - finally: - if use_gpu_hierarchy: - fabric_hierarchy.track_world_xform_changes(True) - fabric_hierarchy.track_local_xform_changes(True) - except Exception: - logger.exception("[NewtonManager] sync_transforms_to_fabric FAILED") + sim = PhysicsManager._sim + sim.get_or_create_backend(sim.fabric_cfg).update_transforms(sim.get_scene_data_provider()) @classmethod def sync_transforms_to_usd(cls) -> None: @@ -844,7 +675,7 @@ def sync_transforms_to_usd(cls) -> None: @classmethod def sync_cables_to_usd(cls) -> None: """Write Newton cable segment endpoints to Fabric curve points.""" - if not cls._cables_dirty: + if not (cls._cables_dirty or cls._transforms_may_change_on_graph_replay): return if cls._usdrt_stage is None or cls._cable_shape_ids is None: NewtonManager._cables_dirty = False @@ -865,7 +696,7 @@ def sync_cables_to_usd(cls) -> None: NewtonManager._cables_dirty = False return _, _, body_q, _, _ = cls._cable_sync_cpu_buffers - wp.copy(body_q, cls.backend.state_0.body_q) + wp.copy(body_q, cls._scene_data_backend.state.body_q) wp.launch( _sync_cable_points, dim=selection.GetCount(), @@ -960,15 +791,11 @@ def _sync_particle_points_prims(cls) -> bool: return len(due) < len(cls._particle_visual_prims) @classmethod - def _mark_transforms_dirty(cls) -> None: - """Flag that rigid-body transforms have changed and Fabric needs re-sync. - - The actual sync is deferred to :meth:`sync_transforms_to_fabric`, - which runs at render cadence via :meth:`pre_render`. - """ - NewtonManager._transforms_dirty = True + def _mark_transforms_changed(cls) -> None: + """Publish authored rigid-body changes and invalidate cable geometry.""" + if NewtonManager._scene_data_backend is not None: + NewtonManager._scene_data_backend.transforms_version += 1 NewtonManager._cables_dirty = True - device = PhysicsManager._device if device is not None: device = wp.get_device(device) @@ -984,16 +811,6 @@ def _mark_particles_dirty(cls) -> None: """ NewtonManager._particles_dirty = True - @classmethod - def _mark_state_dirty(cls) -> None: - """Flag that all physics state has changed and Fabric needs re-sync. - - Convenience method that marks both transforms and particles dirty. - Called by :meth:`_simulate` after stepping. - """ - cls._mark_transforms_dirty() - cls._mark_particles_dirty() - @classmethod def register_particle_visual_prim( cls, prim_path: str, particle_offset: int, particle_count: int, sync_frequency: int = 1 @@ -1117,9 +934,8 @@ def step(cls) -> None: cls._simulate_physics_only() PhysicsManager._sim_time += physics_dt - if cls._usdrt_stage is not None: - cls._mark_state_dirty() - elif cls._particle_visual_prims: + cls._mark_transforms_changed() + if cls._usdrt_stage is not None or cls._particle_visual_prims: cls._mark_particles_dirty() cls._mark_sensor_state_dirty() @@ -1168,8 +984,6 @@ def clear(cls): NewtonManager._visualization_stop_callback = None if callback is not None: callback.deregister() - NewtonManager._use_fabric_gpu_hierarchy = None - NewtonManager._newton_fabric_ready = False NewtonManager._num_envs = None NewtonManager._builder = None NewtonManager._solver = None @@ -1200,6 +1014,7 @@ def clear(cls): # Per-world reset masks NewtonManager._world_reset_mask = None NewtonManager._fk_reset_mask = None + NewtonManager._reconciliation_pending = False NewtonManager._graph = None NewtonManager._graph_capture_pending = False NewtonManager._sensor_tasks = {} @@ -1207,12 +1022,8 @@ def clear(cls): NewtonManager._invalidate_sensor_graph() NewtonManager._sensor_state = None NewtonManager._sensor_state_dirty = True - NewtonManager._sensor_graph_capture_failed = False NewtonManager._sensor_bvh_shape_flags = ShapeFlags.VISIBLE - NewtonManager._newton_stage_path = None NewtonManager._usdrt_stage = None - NewtonManager._fabric_body_scales = None - NewtonManager._transforms_dirty = False NewtonManager._transforms_may_change_on_graph_replay = False NewtonManager._particles_dirty = False NewtonManager._cables_dirty = False @@ -1223,8 +1034,8 @@ def clear(cls): NewtonManager._deformable_registry = [] NewtonManager._per_world_builder_hooks = [] NewtonManager._up_axis = "Z" - NewtonManager._scene_data = None NewtonManager._scene_data_mapping = None + NewtonManager._scene_data_version = None NewtonManager._scene_data_points = None NewtonManager._scene_data_geometry_mapping = None NewtonManager._shadow_deformable_entities = None @@ -1509,10 +1320,11 @@ def invalidate_fk( index. Shape ``(world_count, count_per_world)``. Obtained from ``ArticulationView.articulation_ids``. """ - cls._mark_transforms_dirty() + cls._mark_transforms_changed() if cls._world_reset_mask is None or cls._fk_reset_mask is None: return + NewtonManager._reconciliation_pending = True if articulation_ids is not None and env_mask is not None: wp.launch( @@ -1547,9 +1359,10 @@ def invalidate_body_state( env_ids: Integer indices of dirtied environments. Used by index write methods. env_mask: Boolean mask of dirtied environments. Used by mask write methods. """ - cls._mark_transforms_dirty() + cls._mark_transforms_changed() if cls._world_reset_mask is None: return + NewtonManager._reconciliation_pending = True if env_mask is not None: wp.launch( _or_world_reset_mask_from_mask, @@ -1685,14 +1498,10 @@ def start_simulation(cls) -> None: NewtonManager._world_reset_mask = wp.zeros(cls.backend.model.world_count + 1, dtype=wp.bool, device=device) NewtonManager._fk_reset_mask = wp.zeros(cls.backend.model.articulation_count, dtype=wp.bool, device=device) - logger.info("Dispatching PHYSICS_READY callbacks") - cls.dispatch_event(PhysicsEvent.PHYSICS_READY) - # Setup USD/Fabric sync for Kit viewport rendering if not cls._clone_physics_only: import usdrt - NewtonManager._fabric_body_scales = None body_paths = list(cls.backend.model.body_label) NewtonManager._usdrt_stage = get_current_stage(fabric=True) body_bindings = NewtonManager._cl_fabric_body_bindings @@ -1706,6 +1515,12 @@ def start_simulation(cls) -> None: NewtonManager._initialize_fabric_body_prims(cls._usdrt_stage, fabric_hierarchy, usdrt, body_bindings) NewtonManager._initialize_fabric_cable_prims(cls._usdrt_stage, fabric_hierarchy, usdrt) + + logger.info("Dispatching PHYSICS_READY callbacks") + cls.dispatch_event(PhysicsEvent.PHYSICS_READY) + + # MPM assets register their particle visualizations during PHYSICS_READY. + if not cls._clone_physics_only: NewtonManager._initialize_fabric_particle_prims( cls._usdrt_stage, fabric_hierarchy, @@ -1713,15 +1528,15 @@ def start_simulation(cls) -> None: NewtonManager._particle_visual_prims, ) - cls._mark_state_dirty() - cls.sync_transforms_to_fabric() + cls._mark_transforms_changed() + cls._mark_particles_dirty() cls.sync_cables_to_usd() cls.sync_particles_to_usd() @staticmethod def _initialize_fabric_body_prims(stage, fabric_hierarchy, usdrt, body_bindings: Sequence[tuple[str, int]]) -> None: """Initialize Fabric body prims used by Newton transform sync.""" - for prim_path, body_index in body_bindings: + for prim_path, _ in body_bindings: prim = stage.GetPrimAtPath(prim_path) if prim.IsValid(): xformable_prim = usdrt.Rt.Xformable(prim) @@ -1731,11 +1546,7 @@ def _initialize_fabric_body_prims(stage, fabric_hierarchy, usdrt, body_bindings: xformable_prim = usdrt.Rt.Xformable(prim) xformable_prim.CreateFabricHierarchyWorldMatrixAttr() - prim.CreateAttribute(NewtonManager._newton_index_attr, usdrt.Sdf.ValueTypeNames.UInt, custom=True) - prim.GetAttribute(NewtonManager._newton_index_attr).Set(body_index) - # Tag with PhysicsRigidBodyAPI so FabricHierarchyGpuUpdateOptions.RIGID_BODY - # applies Inverse propagation (preserves Newton's world transforms and derives - # local) instead of Forward. + # Include native bodies absent from USD in the SDP rigid-transform binding. prim.AddAppliedSchema("PhysicsRigidBodyAPI") fabric_hierarchy.update_world_xforms() @@ -2403,7 +2214,7 @@ def initialize_solver(cls) -> None: # solver-specialized FK delegate, now that the solver and the delegate both exist. # Runs before graph capture below so the capture warmup sees a valid body_q. cls._eval_fk(None, None) - cls._mark_transforms_dirty() + cls._mark_transforms_changed() # Fully graphable Newton actuators defer capture until ``set_decimation`` # provides the environment's final decimation value. Other paths capture @@ -2743,9 +2554,12 @@ def get_state(cls, scene_data_provider: SceneDataProvider | None = None) -> Stat observe stale transforms. Under the Newton sim backend, pending forward kinematics is applied before returning the live state. """ - if cls._fk_reset_mask is not None and cls._backend_is_newton(scene_data_provider): - cls.forward() - cls.update_visualization_state(scene_data_provider) + if scene_data_provider is None: + scene_data_provider = cls.get_scene_data_provider() + if cls._backend_is_newton(scene_data_provider): + scene_data_provider.get_transforms(SceneDataFormat.Transform()) + else: + cls.update_visualization_state(scene_data_provider) return cls.get_state_0() @classmethod @@ -2936,6 +2750,7 @@ def _initialize_visualization_model(cls, cfg: NewtonBackendCfg, geometry: tuple[ NewtonManager._num_envs = cls.backend.model.num_envs shadow_entities, registry_groups = geometry NewtonManager._scene_data_mapping = None + NewtonManager._scene_data_version = None NewtonManager._shadow_deformable_entities = shadow_entities NewtonManager._scene_data_geometry_mapping = None NewtonManager._mapped_sim_particle_offsets = None @@ -2947,6 +2762,7 @@ def _initialize_visualization_model(cls, cfg: NewtonBackendCfg, geometry: tuple[ NewtonManager._sim_particle_q = None NewtonManager._deformable_registry = [] populate_shadow_deformable_registry(cls, registry_groups) + cls.update_visualization_state() NewtonManager._visualization_stop_callback = sim.physics_manager.register_callback( lambda _payload: NewtonManager.clear(), PhysicsEvent.STOP, @@ -2968,17 +2784,10 @@ def update_visualization_state(cls, scene_data_provider: SceneDataProvider | Non Newton sim backend: no-op — ``_state_0`` is the live, authoritative state already advanced by :meth:`step` / forward kinematics. - PhysX / OVPhysX sim backend: pull rigid-body transforms and deformable - nodal positions from the :class:`~isaaclab.scene_data.SceneDataProvider` - and write them into the shadow ``_state_0.body_q`` / ``particle_q`` so - Newton-native consumers (Newton renderer, Newton/Rerun/Viser visualizers, - OVRTX renderer, Newton GL video) see fresh poses and mesh points. - - Calls use ``allow_passthrough=False`` so identity mappings still copy into - the pre-bound shadow buffers. Passthrough would rebind the temporary - :class:`~isaaclab.scene_data.SceneDataFormat` fields away from - ``_state_0``, leaving OVRTX and other ``get_state()`` consumers on stale - rest-pose particle / body state. + PhysX / OVPhysX sim backend: bind shared SDP rigid-body transforms to + ``state_0.body_q`` without copying. Consumers must treat this array as + read-only. Deformable points still copy into the shadow ``particle_q`` + buffer, including simulation-to-visual mesh remapping. Invoked lazily from :meth:`get_state` so consumers do not need to coordinate the sync explicitly. @@ -2996,23 +2805,25 @@ def update_visualization_state(cls, scene_data_provider: SceneDataProvider | Non return if cls.backend.state_0.body_q is not None: - if cls._scene_data is None: - cls._scene_data = SceneDataFormat.Transform() - - # Invalidate stale mapping when the model's body count changed (e.g. tiled → viewport - # test within the same process where _model was rebuilt from a different stage). - if cls._scene_data_mapping is not None and cls._scene_data_mapping.shape[0] != cls.backend.model.body_count: - cls._scene_data_mapping = None - - if cls._scene_data_mapping is None: + if cls._scene_data_version is None: body_labels = list(cls.backend.model.body_label) body_paths = cls._resolve_scene_data_body_paths(body_labels, scene_data_provider.usd_stage) + if len(set(body_paths)) != cls.backend.model.body_count or not set(body_paths).issubset( + scene_data_provider.backend.transform_paths + ): + raise ValueError("Every Newton render body must have one unique SDP transform path.") cls._scene_data_mapping = scene_data_provider.create_mapping(body_paths) - cls._scene_data.transforms = cls.backend.state_0.body_q - scene_data_provider.get_transforms( - cls._scene_data, mapping=cls._scene_data_mapping, allow_passthrough=False - ) + transforms = SceneDataFormat.Transform() + if scene_data_provider.get_transforms( + transforms, mapping=cls._scene_data_mapping, count=cls.backend.model.body_count + ): + if cls.backend.state_0.body_q is not transforms.transforms: + cls.backend.state_0.body_q = transforms.transforms + cls._invalidate_sensor_graph() + if cls._scene_data_version != scene_data_provider.backend.transforms_version: + cls._mark_sensor_state_dirty() + cls._scene_data_version = scene_data_provider.backend.transforms_version if cls.backend.state_0.particle_q is not None and scene_data_provider.point_count > 0: if cls._scene_data_points is None: @@ -3058,7 +2869,7 @@ def update_visualization_state(cls, scene_data_provider: SceneDataProvider | Non allow_passthrough=False, ) - cls._mark_sensor_state_dirty() + cls._mark_sensor_state_dirty() @classmethod def _geometry_mapped_sim_offsets(cls, scene_data_provider: SceneDataProvider) -> set[int]: diff --git a/source/isaaclab_newton/isaaclab_newton/renderers/newton_warp_renderer.py b/source/isaaclab_newton/isaaclab_newton/renderers/newton_warp_renderer.py index aa46968f24e8..f83e286e36a9 100644 --- a/source/isaaclab_newton/isaaclab_newton/renderers/newton_warp_renderer.py +++ b/source/isaaclab_newton/isaaclab_newton/renderers/newton_warp_renderer.py @@ -481,12 +481,12 @@ def __init__(self, cfg: NewtonWarpRendererCfg): def initialize(self) -> None: """Post-physics setup: read the built Newton model and construct the sensor.""" - self._newton_model = NewtonManager.get_model() - if self._newton_model is None: + model = NewtonManager.get_model() + if model is None: raise RuntimeError("NewtonWarpRenderer requires a clone-built model before initialization.") self.newton_sensor = newton.sensors.SensorTiledCamera( - self._newton_model, + model, default_render_config=newton.sensors.SensorTiledCamera.RenderConfig( enable_textures=self.cfg.enable_textures, enable_shadows=self.cfg.enable_shadows, @@ -560,7 +560,7 @@ def create_render_data(self, spec: CameraRenderSpec) -> RenderData: ): if self._seg_mapper is None: clone_plan = SimulationContext.instance().get_clone_plan() - self._seg_mapper = NewtonSegmentationMapper(self._newton_model, self._stage, self.cfg, clone_plan) + self._seg_mapper = NewtonSegmentationMapper(self.newton_sensor.model, self._stage, self.cfg, clone_plan) if RenderBufferKind.SEMANTIC_SEGMENTATION in spec.cfg.data_types: self._seg_mapper.build_mapping( RenderBufferKind.SEMANTIC_SEGMENTATION, bool(self.cfg.colorize_semantic_segmentation) @@ -577,12 +577,9 @@ def set_outputs(self, render_data: RenderData, output_data: dict[str, ProxyArray """Store output buffers. See :meth:`~isaaclab.renderers.base_renderer.BaseRenderer.set_outputs`.""" render_data.set_outputs(output_data) - def update_transforms(self): - """Sync Newton scene state before rendering. - See :meth:`~isaaclab.renderers.base_renderer.BaseRenderer.update_transforms`.""" - sim = SimulationContext.instance() - sim.physics_manager.forward() - NewtonManager.update_visualization_state() + def update_transforms(self) -> None: + """No-op: the shared sensor pipeline refreshes transforms immediately before rendering.""" + pass def update_geometries(self) -> None: """No-op for Newton Warp - geometry is read directly from Newton state during render. @@ -629,7 +626,7 @@ def render(self, render_data: RenderData): if render_data.sensor_task_name is None: render_data.sensor_task_name = f"newton_warp_render:{id(render_data)}" - tri_indices = self._newton_model.tri_indices + tri_indices = self.newton_sensor.model.tri_indices # Warp mesh refits allocate graph nodes and are not supported inside a conditional graph body. graph_capturable = tri_indices is None or tri_indices.shape[0] == 0 NewtonManager._register_sensor_task( diff --git a/source/isaaclab_newton/test/physics/test_newton_fabric_body_sync.py b/source/isaaclab_newton/test/physics/test_newton_fabric_body_sync.py index 8f89fdda4088..2123fd766a64 100644 --- a/source/isaaclab_newton/test/physics/test_newton_fabric_body_sync.py +++ b/source/isaaclab_newton/test/physics/test_newton_fabric_body_sync.py @@ -18,15 +18,20 @@ import torch import warp as wp from isaaclab_newton.physics import NewtonCfg, NewtonManager, VBDSolverCfg, XPBDSolverCfg +from isaaclab_newton.renderers import NewtonWarpRendererCfg +from isaaclab_physx.renderers import IsaacRtxRendererCfg +from isaaclab_physx.renderers.fabric import FabricBackend, FabricBackendCfg from isaaclab_physx.sim.schemas import PhysxRigidBodyCfg +from isaaclab_visualizers.kit import KitVisualizerCfg from pxr import Gf as UsdGf from pxr import UsdGeom from usdrt import Gf, Rt import isaaclab.sim as sim_utils -from isaaclab.assets import CableObjectCfg, RigidObjectCfg +from isaaclab.assets import AssetBaseCfg, CableObjectCfg, RigidObjectCfg from isaaclab.scene import InteractiveScene, InteractiveSceneCfg +from isaaclab.sensors import CameraCfg from isaaclab.sim import SimulationCfg, build_simulation_context from isaaclab.sim.spawners.materials import CableMaterialCfg from isaaclab.sim.spawners.shapes import CableCfg @@ -36,6 +41,14 @@ @configclass class _RenderSceneCfg(InteractiveSceneCfg): + camera = CameraCfg( + prim_path="{ENV_REGEX_NS}/Camera", + height=16, + width=16, + data_types=["rgb"], + spawn=sim_utils.PinholeCameraCfg(), + renderer_cfg=IsaacRtxRendererCfg(), + ) cube: RigidObjectCfg = RigidObjectCfg( prim_path="{ENV_REGEX_NS}/Cube", spawn=sim_utils.CuboidCfg( @@ -133,127 +146,6 @@ def _expected_cable_points_world(cable, env_id: int = 0) -> torch.Tensor: return torch.stack(points) -class _FakeAttribute: - def __init__(self, value_type, custom): - self.value_type = value_type - self.custom = custom - self.value = None - - def Set(self, value): - self.value = value - - -class _FakePrim: - def __init__(self, valid=True): - self.valid = valid - self.attributes = {} - self.applied_schemas = [] - self.created_world_matrix_attrs = 0 - self.set_world_xform_from_usd = 0 - - def IsValid(self): - return self.valid - - def CreateAttribute(self, name, value_type, custom=False): - self.attributes[name] = _FakeAttribute(value_type, custom) - return self.attributes[name] - - def GetAttribute(self, name): - return self.attributes[name] - - def AddAppliedSchema(self, schema): - self.applied_schemas.append(schema) - - -class _FakeStage: - def __init__(self, prims=None): - self.prims = prims or {} - self.defined_prims = [] - - def GetPrimAtPath(self, path): - return self.prims.get(path, _FakePrim(valid=False)) - - def DefinePrim(self, path, prim_type): - prim = _FakePrim() - self.prims[path] = prim - self.defined_prims.append((path, prim_type)) - return prim - - -class _FakeXformable: - def __init__(self, prim): - self.prim = prim - - def SetWorldXformFromUsd(self): - self.prim.set_world_xform_from_usd += 1 - - def CreateFabricHierarchyWorldMatrixAttr(self): - self.prim.created_world_matrix_attrs += 1 - - -class _FakeFabricHierarchy: - def __init__(self): - self.update_world_xforms_count = 0 - - def update_world_xforms(self): - self.update_world_xforms_count += 1 - - -class _FakeRt: - Xformable = _FakeXformable - - -class _FakeValueTypeNames: - UInt = "UInt" - - -class _FakeSdf: - ValueTypeNames = _FakeValueTypeNames - - -class _FakeUsdrt: - Rt = _FakeRt - Sdf = _FakeSdf - - -def test_initialize_fabric_body_prims_uses_existing_fabric_prim(): - prim = _FakePrim() - stage = _FakeStage({"/World/envs/env_0/Robot/base": prim}) - fabric_hierarchy = _FakeFabricHierarchy() - - NewtonManager._initialize_fabric_body_prims( - stage, fabric_hierarchy, _FakeUsdrt, [("/World/envs/env_0/Robot/base", 3)] - ) - - assert stage.defined_prims == [] - assert prim.set_world_xform_from_usd == 1 - assert prim.created_world_matrix_attrs == 0 - assert prim.GetAttribute("newton:index").value_type == "UInt" - assert prim.GetAttribute("newton:index").custom is True - assert prim.GetAttribute("newton:index").value == 3 - assert prim.applied_schemas == ["PhysicsRigidBodyAPI"] - assert fabric_hierarchy.update_world_xforms_count == 1 - - -def test_initialize_fabric_body_prims_creates_missing_body_as_xform(): - stage = _FakeStage() - fabric_hierarchy = _FakeFabricHierarchy() - - NewtonManager._initialize_fabric_body_prims( - stage, fabric_hierarchy, _FakeUsdrt, [("/World/envs/env_1/Robot/joints/forearm", 7)] - ) - - prim = stage.prims["/World/envs/env_1/Robot/joints/forearm"] - assert stage.defined_prims == [("/World/envs/env_1/Robot/joints/forearm", "Xform")] - assert prim.set_world_xform_from_usd == 0 - assert prim.created_world_matrix_attrs == 1 - assert prim.GetAttribute("newton:index").value_type == "UInt" - assert prim.GetAttribute("newton:index").custom is True - assert prim.GetAttribute("newton:index").value == 7 - assert prim.applied_schemas == ["PhysicsRigidBodyAPI"] - assert fabric_hierarchy.update_world_xforms_count == 1 - - @pytest.mark.isaacsim_ci @pytest.mark.skipif(not wp.get_cuda_device_count(), reason="CUDA is unavailable") def test_root_pose_write_is_visible_on_next_render_without_step(): @@ -269,6 +161,7 @@ def test_root_pose_write_is_visible_on_next_render_without_step(): device=device, gravity=(0.0, 0.0, 0.0), physics=NewtonCfg(solver_cfg=XPBDSolverCfg(), use_cuda_graph=False), + visualizer_cfgs=[KitVisualizerCfg(headless=True)], ) with build_simulation_context(sim_cfg=sim_cfg) as sim: @@ -278,7 +171,11 @@ def test_root_pose_write_is_visible_on_next_render_without_step(): try: sim.reset() scene.reset() - sim.render() + _render(sim, scene) + + fabric = sim.get_or_create_backend(FabricBackendCfg(stage=sim.stage, device=sim.device)) + assert sim.visualizers[0]._fabric is scene["camera"]._renderer._fabric is fabric + assert sum(isinstance(resource, FabricBackend) for _, resource in sim._backend_registry) == 1 cube = scene["cube"] body_path = "/World/envs/env_0/Cube" @@ -291,8 +188,7 @@ def test_root_pose_write_is_visible_on_next_render_without_step(): cube.write_root_link_pose_to_sim_index(root_pose=target_pose) physics_steps = sim.get_physics_step_count() - sim.render() - wp.synchronize_device(device) + _render(sim, scene) assert sim.get_physics_step_count() == physics_steps torch.testing.assert_close( @@ -309,13 +205,13 @@ def test_root_pose_write_is_visible_on_next_render_without_step(): env_mask = wp.ones(1, dtype=wp.bool, device=device) pose_buffer = target_pose.clone() cube.write_root_link_pose_to_sim_mask(root_pose=pose_buffer, env_mask=env_mask) - sim.render() + _render(sim, scene) torch.cuda.synchronize(device) with wp.ScopedCapture(device=device) as capture: cube.write_root_link_pose_to_sim_mask(root_pose=pose_buffer, env_mask=env_mask) - sim.render() + _render(sim, scene) replay_targets = ( torch.tensor([2.5, 0.5, 1.25, 0.0, 0.0, 0.0, 1.0], device=device), @@ -328,8 +224,7 @@ def test_root_pose_write_is_visible_on_next_render_without_step(): wp.synchronize_device(device) physics_steps = sim.get_physics_step_count() - sim.render() - wp.synchronize_device(device) + _render(sim, scene) assert sim.get_physics_step_count() == physics_steps torch.testing.assert_close( @@ -344,9 +239,13 @@ def test_root_pose_write_is_visible_on_next_render_without_step(): @pytest.mark.isaacsim_ci @pytest.mark.skipif(not wp.get_cuda_device_count(), reason="CUDA is unavailable") -def test_root_pose_sync_preserves_authored_scale(): +@pytest.mark.parametrize( + ("device", "renderer_cfg"), + [("cpu", IsaacRtxRendererCfg()), ("cuda:0", IsaacRtxRendererCfg()), ("cuda:0", NewtonWarpRendererCfg())], + ids=["rtx-cpu", "rtx-cuda", "newton-warp"], +) +def test_root_pose_sync_preserves_authored_scale(device, renderer_cfg): """Newton body pose synchronization must preserve authored USD scale in Kit/RTX.""" - device = "cuda:0" sim_cfg = SimulationCfg( device=device, gravity=(0.0, 0.0, 0.0), @@ -355,7 +254,9 @@ def test_root_pose_sync_preserves_authored_scale(): with build_simulation_context(sim_cfg=sim_cfg) as sim: sim._app_control_on_stop_handle = None - scene = InteractiveScene(_RenderSceneCfg(num_envs=1, env_spacing=2.0)) + scene_cfg = _RenderSceneCfg(num_envs=1, env_spacing=2.0) + scene_cfg.camera.renderer_cfg = renderer_cfg + scene = InteractiveScene(scene_cfg) sim.register_interactive_scene(scene) try: body_path = "/World/envs/env_0/Cube" @@ -365,8 +266,7 @@ def test_root_pose_sync_preserves_authored_scale(): sim.reset() scene.reset() - sim.render() - wp.synchronize_device(device) + _render(sim, scene) torch.testing.assert_close(_fabric_scale(body_path), authored_scale, rtol=0.0, atol=1.0e-5) @@ -376,8 +276,10 @@ def test_root_pose_sync_preserves_authored_scale(): device=device, ) scene["cube"].write_root_link_pose_to_sim_index(root_pose=target_pose) - sim.render() - wp.synchronize_device(device) + if isinstance(renderer_cfg, NewtonWarpRendererCfg): + assert not sim.visualizers + NewtonManager.sync_transforms_to_fabric() + _render(sim, scene) torch.testing.assert_close(_fabric_position(body_path), target_pose[0, :3].cpu(), rtol=0.0, atol=1.0e-4) torch.testing.assert_close(_fabric_scale(body_path), authored_scale, rtol=0.0, atol=1.0e-5) @@ -385,6 +287,40 @@ def test_root_pose_sync_preserves_authored_scale(): sim.register_interactive_scene(None) +@pytest.mark.isaacsim_ci +@pytest.mark.skipif(not wp.get_cuda_device_count(), reason="CUDA is unavailable") +def test_nested_bodies_keep_independent_world_poses(): + """A nested rigid body must not inherit its parent's independently published motion.""" + sim_cfg = SimulationCfg( + device="cuda:0", + gravity=(0.0, 0.0, 0.0), + physics=NewtonCfg(solver_cfg=XPBDSolverCfg(), use_cuda_graph=False), + ) + scene_cfg = _RenderSceneCfg(num_envs=1, env_spacing=2.0) + scene_cfg.child = scene_cfg.cube.replace(prim_path="{ENV_REGEX_NS}/Cube/Child") + scene_cfg.cube = AssetBaseCfg(prim_path=scene_cfg.cube.prim_path, spawn=scene_cfg.cube.spawn) + with build_simulation_context(sim_cfg=sim_cfg) as sim: + sim._app_control_on_stop_handle = None + scene = InteractiveScene(scene_cfg) + sim.register_interactive_scene(scene) + try: + sim.reset() + scene.reset() + _render(sim, scene) + paths = ["/World/envs/env_0/Cube", "/World/envs/env_0/Cube/Child"] + targets = torch.tensor([[1.5, -0.75, 2.0], [-0.25, 1.0, 3.0]], device=sim.device) + state = wp.to_torch(NewtonManager.get_state_0().body_q) + indices = [NewtonManager.get_model().body_label.index(path) for path in paths] + state[indices, :3] = targets + NewtonManager.invalidate_body_state() + _render(sim, scene) + for path, target in zip(paths, targets.cpu()): + _assert_position(_fabric_position(path), target) + assert not UsdGeom.Xformable(sim.stage.GetPrimAtPath(path)).GetResetXformStack() + finally: + sim.register_interactive_scene(None) + + @pytest.mark.isaacsim_ci @pytest.mark.skipif(not wp.get_cuda_device_count(), reason="CUDA is unavailable") def test_periodic_cable_is_skipped_by_fabric_sync(): @@ -498,16 +434,17 @@ def _frame_scene(frame_path: str, translation, device: str = "cuda:0"): view = FrameView(frame_path, device=device) sim.reset() scene.reset() - _render(sim, device) + _render(sim, scene) yield sim, scene, view finally: sim.register_interactive_scene(None) -def _render(sim, device: str = "cuda:0") -> None: - """Render and wait for the write to land in Fabric.""" +def _render(sim, scene) -> None: + """Render through the camera's public data path and wait for Fabric writes.""" sim.render() - wp.synchronize_device(device) + scene["camera"].update(sim.get_rendering_dt(), force_recompute=True) + wp.synchronize_device(sim.device) def _world_pose(position: torch.Tensor) -> tuple[wp.array, wp.array]: @@ -535,25 +472,6 @@ def _assert_position(actual: torch.Tensor, expected: torch.Tensor) -> None: torch.testing.assert_close(actual, expected, rtol=0.0, atol=1.0e-4) -@pytest.mark.isaacsim_ci -@pytest.mark.skipif(not wp.get_cuda_device_count(), reason="CUDA is unavailable") -def test_frame_view_pose_write_reaches_fabric(): - """A world-attached FrameView pose write reaches the transform Kit/RTX renders.""" - device = "cuda:0" - frame_path = "/World/Frame" - spawn_position = torch.tensor([0.0, 0.0, 2.0]) - target_position = torch.tensor([1.0, -0.5, 8.0]) - - with _frame_scene(frame_path, tuple(spawn_position.tolist()), device) as (sim, _, view): - _assert_position(_fabric_position(frame_path), spawn_position) - - _write_frame_world_position(view, target_position.to(device)) - _render(sim, device) - - _assert_position(_reported_position(view), target_position) - _assert_position(_fabric_position(frame_path), target_position) - - @pytest.mark.isaacsim_ci @pytest.mark.skipif(not wp.get_cuda_device_count(), reason="CUDA is unavailable") def test_frame_view_pose_write_reaches_fabric_when_the_scope_raises(): @@ -562,13 +480,14 @@ def test_frame_view_pose_write_reaches_fabric_when_the_scope_raises(): frame_path = "/World/Frame" target_position = torch.tensor([1.0, -0.5, 8.0]) - with _frame_scene(frame_path, (0.0, 0.0, 2.0), device) as (sim, _, view): + with _frame_scene(frame_path, (0.0, 0.0, 2.0), device) as (sim, scene, view): with pytest.raises(RuntimeError, match="boom"): # noqa: PT012 -- the raise is the scenario with view.xform_world_space_writer() as writer: writer.set_poses(*_world_pose(target_position.to(device))) raise RuntimeError("boom") - _render(sim, device) + _render(sim, scene) + _assert_position(_reported_position(view), target_position) _assert_position(_fabric_position(frame_path), target_position) @@ -584,13 +503,13 @@ def test_frame_view_pose_write_on_body_child_survives_body_motion(): body_start = torch.tensor([0.0, 0.0, 1.0]) written_position = body_start + torch.tensor([0.5, 0.0, 0.0]) _write_frame_world_position(view, written_position.to(device)) - _render(sim, device) + _render(sim, scene) _assert_position(_fabric_position(frame_path), written_position) body_pose = torch.tensor([[1.5, -0.75, 2.0, 0.0, 0.0, 0.0, 1.0]], dtype=torch.float32, device=device) scene["cube"].write_root_link_pose_to_sim_index(root_pose=body_pose) - _render(sim, device) + _render(sim, scene) expected = body_pose[0, :3].cpu() + (written_position - body_start) _assert_position(_reported_position(view), expected) @@ -609,15 +528,19 @@ def test_first_frame_pose_write_after_body_move_leaves_the_body_rendered(): body_pose = torch.tensor([[1.5, -0.75, 2.0, 0.0, 0.0, 0.0, 1.0]], dtype=torch.float32, device=device) body_target = body_pose[0, :3].cpu() scene["cube"].write_root_link_pose_to_sim_index(root_pose=body_pose) - _render(sim, device) + _render(sim, scene) _assert_position(_fabric_position(body_path), body_target) + late_child = f"{frame_path}/LateChild" + offset = torch.tensor([0.0, 0.0, 0.25]) + sim_utils.create_prim(late_child, "Xform", translation=tuple(offset.tolist())) written_position = body_target + torch.tensor([0.5, 0.0, 0.0]) _write_frame_world_position(view, written_position.to(device)) - _render(sim, device) + _render(sim, scene) _assert_position(_fabric_position(body_path), body_target) _assert_position(_fabric_position(frame_path), written_position) + _assert_position(_fabric_position(late_child), written_position + offset) @pytest.mark.isaacsim_ci @@ -638,7 +561,7 @@ def test_frame_view_pose_write_after_unrendered_steps_reaches_fabric(): target_position = torch.tensor([0.0, 0.0, 1.5]) _write_frame_world_position(view, target_position.to(device)) - _render(sim, device) + _render(sim, scene) _assert_position(_reported_position(view), target_position) _assert_position(_fabric_position(frame_path), target_position) diff --git a/source/isaaclab_newton/test/physics/test_newton_manager_abstraction.py b/source/isaaclab_newton/test/physics/test_newton_manager_abstraction.py index 82b9ae9e1c4d..f2731a773e41 100644 --- a/source/isaaclab_newton/test/physics/test_newton_manager_abstraction.py +++ b/source/isaaclab_newton/test/physics/test_newton_manager_abstraction.py @@ -33,6 +33,7 @@ import textwrap from inspect import signature from types import SimpleNamespace +from unittest.mock import Mock import isaaclab_newton.physics.newton_manager as newton_manager_module import numpy as np @@ -66,12 +67,14 @@ XPBDSolverCfg, ) from isaaclab_newton.physics.mpm_manager import _make_solver_config +from isaaclab_newton.renderers.newton_warp_renderer import NewtonWarpRenderer from newton import JointTargetMode, JointType, ModelBuilder, ShapeFlags from newton.selection import ArticulationView from newton.solvers import SolverFeatherstone, SolverImplicitMPM, SolverKamino, SolverMuJoCo, SolverVBD, SolverXPBD from isaaclab.actuators import ImplicitActuatorCfg -from isaaclab.physics import PhysicsManager +from isaaclab.physics import PhysicsEvent, PhysicsManager +from isaaclab.scene_data import SceneDataFormat from isaaclab.sim import SimulationCfg, build_simulation_context # --------------------------------------------------------------------------- @@ -299,16 +302,17 @@ def test_refit_sensor_bvh_rejects_missing_sensor_state(monkeypatch): def test_sensor_task_builds_and_refits_bvhs_before_rendering(monkeypatch): - """Shape and particle BVHs are built and refit before a render task runs.""" + """One state refresh precedes BVH refits and rendering, including explicit transform updates.""" state = object() - status = {"state_refreshed": False, "shape_refit": False, "particle_refit": False, "rendered": False} + status = {"state_refreshes": 0, "shape_refit": False, "particle_refit": False, "rendered": False} class FakeModel: shape_count = 1 particle_count = 1 bvh_shapes = None bvh_particles = None + tri_indices = None def bvh_build_shapes(self, current_state): assert current_state is state @@ -329,7 +333,7 @@ def bvh_refit_particles(self, current_state): model = FakeModel() def render(): - assert status["state_refreshed"] + assert status["state_refreshes"] == 1 assert model.bvh_shapes is not None assert model.bvh_particles is not None assert status["shape_refit"] @@ -337,7 +341,7 @@ def render(): status["rendered"] = True def get_state(cls): - status["state_refreshed"] = True + status["state_refreshes"] += 1 return state monkeypatch.setattr(NewtonManager, "get_model", classmethod(lambda cls: model)) @@ -353,16 +357,21 @@ def get_state(cls): monkeypatch.setattr(NewtonManager, "_sensor_graph_capture_failed", False, raising=False) monkeypatch.setattr(PhysicsManager, "_cfg", SimpleNamespace(use_cuda_graph=False), raising=False) - NewtonManager._register_sensor_task("render", render) - NewtonManager._update_sensor_tasks("render") + renderer = object.__new__(NewtonWarpRenderer) + renderer.newton_sensor = SimpleNamespace(model=model) + monkeypatch.setattr(renderer, "_launch_render", lambda _data: render()) + renderer.update_transforms() + renderer.render(SimpleNamespace(sensor_task_name=None, ppisp_pipeline=None)) assert status["rendered"] -def test_non_graph_capturable_sensor_task_runs_eagerly(monkeypatch): - """Sensor tasks with allocation-backed work should not attempt CUDA graph capture.""" +def test_newton_warp_renderer_runs_triangle_mesh_refit_eagerly(monkeypatch): + """Allocation-backed triangle-mesh rendering runs without attempting CUDA graph capture.""" state = object() - model = SimpleNamespace(shape_count=0, particle_count=0, bvh_shapes=None, bvh_particles=None) + model = SimpleNamespace( + shape_count=0, particle_count=0, bvh_shapes=None, bvh_particles=None, tri_indices=SimpleNamespace(shape=(1, 3)) + ) calls: list[str] = [] monkeypatch.setattr(NewtonManager, "get_model", classmethod(lambda cls: model)) @@ -385,44 +394,12 @@ def test_non_graph_capturable_sensor_task_runs_eagerly(monkeypatch): classmethod(lambda cls: pytest.fail("Non-graph-capturable task attempted CUDA graph capture.")), ) - NewtonManager._register_sensor_task("render", lambda: calls.append("render"), graph_capturable=False) - NewtonManager._update_sensor_tasks("render") - - assert calls == ["render"] - assert NewtonManager._sensor_graph is None - assert NewtonManager._sensor_graph_capture_failed is False - - -@pytest.mark.parametrize( - ("triangle_count", "expected_graph_capturable"), - [ - pytest.param(None, True, id="no-triangle-array"), - pytest.param(0, True, id="empty-triangle-array"), - pytest.param(1, False, id="deformable-triangle-mesh"), - ], -) -def test_newton_warp_renderer_marks_triangle_mesh_refit_as_eager( - monkeypatch, triangle_count, expected_graph_capturable -): - """Deformable triangle-mesh rendering should opt out of conditional CUDA graph capture.""" - from isaaclab_newton.renderers.newton_warp_renderer import NewtonWarpRenderer - - registration: dict[str, object] = {} - - def register_task(cls, name, update_fn, *, graph_capturable=True): - registration.update(name=name, update_fn=update_fn, graph_capturable=graph_capturable) - - monkeypatch.setattr(NewtonManager, "_register_sensor_task", classmethod(register_task)) - monkeypatch.setattr(NewtonManager, "_update_sensor_tasks", classmethod(lambda cls, *names: None)) - - tri_indices = None if triangle_count is None else SimpleNamespace(shape=(triangle_count, 3)) renderer = object.__new__(NewtonWarpRenderer) - renderer._newton_model = SimpleNamespace(tri_indices=tri_indices) - render_data = SimpleNamespace(sensor_task_name=None, ppisp_pipeline=None) - - renderer.render(render_data) + renderer.newton_sensor = SimpleNamespace(model=model) + monkeypatch.setattr(renderer, "_launch_render", lambda _data: calls.append("render")) + renderer.render(SimpleNamespace(sensor_task_name=None, ppisp_pipeline=None)) - assert registration["graph_capturable"] is expected_graph_capturable + assert calls == ["render"] def test_sensor_bvh_shape_flags_are_fixed_before_builder_creation(monkeypatch): @@ -1298,7 +1275,7 @@ def test_fixed_root_pose_write_updates_solver(monkeypatch, asset_class, writer, def test_forward_consumes_existing_reset_masks(monkeypatch): - """The existing device masks are the complete input to masked FK and the solver reset hook.""" + """Authored-state masks are consumed once, without rerunning clean FK or solver reset.""" world_mask = wp.array([False, True], dtype=wp.bool, device="cpu") fk_mask = wp.array([True, False], dtype=wp.bool, device="cpu") observed: list[tuple[list[bool], list[bool]]] = [] @@ -1313,6 +1290,8 @@ def reset(self, state, world_mask=None, flags=0): monkeypatch.setattr(NewtonManager, "_world_reset_mask", world_mask, raising=False) monkeypatch.setattr(NewtonManager, "_fk_reset_mask", fk_mask, raising=False) + monkeypatch.setattr(NewtonManager, "_reconciliation_pending", True, raising=False) + monkeypatch.setattr(NewtonManager, "_transforms_may_change_on_graph_replay", False) monkeypatch.setattr(NewtonManager, "_eval_fk", record_fk, raising=False) monkeypatch.setattr(NewtonManager, "backend", SimpleNamespace(state_0=object())) monkeypatch.setattr(NewtonManager, "_solver", _RecordingSolver(), raising=False) @@ -1323,6 +1302,7 @@ def reset(self, state, world_mask=None, flags=0): raising=False, ) + NewtonManager.forward() NewtonManager.forward() assert observed == [([False, True], [True, False])] @@ -1342,6 +1322,7 @@ def reset(self, state, world_mask=None, flags=0): monkeypatch.setattr(NewtonManager, "_world_reset_mask", world_mask, raising=False) monkeypatch.setattr(NewtonManager, "_fk_reset_mask", fk_mask, raising=False) + monkeypatch.setattr(NewtonManager, "_reconciliation_pending", True, raising=False) monkeypatch.setattr(NewtonManager, "_eval_fk", lambda worlds, articulations: None, raising=False) monkeypatch.setattr(NewtonManager, "_solver", _RejectingSolver(), raising=False) monkeypatch.setattr( @@ -1447,7 +1428,7 @@ def count_actuator_resolutions(name_keys, names, *args, **kwargs): def test_initialize_solver_prepares_picking_before_graph_capture( monkeypatch, native_path_active, native_graphable, expected_events ): - """Viewer setup precedes initial capture, which only graphable native actuators defer.""" + """Initial and hard resets realize native layouts before consumers, then prepare picking and capture.""" events: list[str] = [] sim_cfg = SimulationCfg( dt=1.0 / 120.0, @@ -1457,6 +1438,19 @@ def test_initialize_solver_prepares_picking_before_graph_capture( with build_simulation_context(sim_cfg=sim_cfg) as sim: build_solver = NewtonMJWarpManager._build_solver + monkeypatch.setitem(sys.modules, "usdrt", Mock()) + monkeypatch.setattr(NewtonMJWarpManager, "_clone_physics_only", False) + monkeypatch.setattr(newton_manager_module, "get_current_stage", lambda **kwargs: Mock()) + for kind in ("body", "cable", "particle"): + monkeypatch.setattr( + NewtonManager, + f"_initialize_fabric_{kind}_prims", + staticmethod(lambda *args, kind=kind: events.append(kind)), + ) + + def on_physics_ready(_): + events.append("ready") + sim.get_scene_data_provider().get_transforms(SceneDataFormat.Transform()) def build_solver_with_actuator_mode(cls, model, solver_cfg): build_solver(model, solver_cfg) @@ -1478,10 +1472,16 @@ def build_solver_with_actuator_mode(cls, model, solver_cfg): "_capture_or_defer_graph", classmethod(lambda cls: events.append("capture")), ) + sim.physics_manager.register_callback( + on_physics_ready, + PhysicsEvent.PHYSICS_READY, + wrap_weak_ref=False, + ) sim.reset() + sim.reset() - assert events == expected_events + assert events == ["body", "cable", "ready", "particle", *expected_events] * 2 def test_abstract_build_solver_raises(): diff --git a/source/isaaclab_ov/changelog.d/sdp-transform-transport.rst b/source/isaaclab_ov/changelog.d/sdp-transform-transport.rst new file mode 100644 index 000000000000..6a36d439a88a --- /dev/null +++ b/source/isaaclab_ov/changelog.d/sdp-transform-transport.rst @@ -0,0 +1,10 @@ +Changed +^^^^^^^ + +* Published OVPhysX rigid poses directly into shared scene-data storage and invalidated cached transforms after + physics steps and manual pose writes. Binding failures were surfaced instead of publishing incomplete poses. +* Routed OVRTX rigid transforms through cached SDP matrix requests, preserving authored scales without a Newton + rigid-state intermediary. Existing renderer configurations remained valid; Newton-backed deformable, particle, + and cable geometry transport remained unchanged. +* Captured OVRTX authored scales from clone-plan prototypes and shared roots, including bodies outside the + default environment namespace. diff --git a/source/isaaclab_ov/isaaclab_ov/assets/articulation/articulation.py b/source/isaaclab_ov/isaaclab_ov/assets/articulation/articulation.py index 2df2dade3aee..6559048006e4 100644 --- a/source/isaaclab_ov/isaaclab_ov/assets/articulation/articulation.py +++ b/source/isaaclab_ov/isaaclab_ov/assets/articulation/articulation.py @@ -536,6 +536,8 @@ def write_root_link_pose_to_sim_index( self._root_view.set_attribute( TT.ROOT_POSE, self.data._root_link_pose_w.data.view(wp.float32), indices=sim_env_ids ) + OvPhysxManager._kinematics_dirty = True + OvPhysxManager._scene_data_backend.transforms_version += 1 def write_root_link_pose_to_sim_mask( self, @@ -575,6 +577,8 @@ def write_root_link_pose_to_sim_mask( if not skip_forward: self.data._reset_pose() self._root_view.set_attribute(TT.ROOT_POSE, self.data._root_link_pose_w.data.view(wp.float32), mask=env_mask_wp) + OvPhysxManager._kinematics_dirty = True + OvPhysxManager._scene_data_backend.transforms_version += 1 def write_root_com_pose_to_sim_index( self, @@ -618,6 +622,8 @@ def write_root_com_pose_to_sim_index( self._root_view.set_attribute( TT.ROOT_POSE, self.data._root_link_pose_w.data.view(wp.float32), indices=sim_env_ids ) + OvPhysxManager._kinematics_dirty = True + OvPhysxManager._scene_data_backend.transforms_version += 1 def write_root_com_pose_to_sim_mask( self, @@ -658,6 +664,8 @@ def write_root_com_pose_to_sim_mask( if not skip_forward: self.data._reset_pose(from_link=False) self._root_view.set_attribute(TT.ROOT_POSE, self.data._root_link_pose_w.data.view(wp.float32), mask=env_mask_wp) + OvPhysxManager._kinematics_dirty = True + OvPhysxManager._scene_data_backend.transforms_version += 1 def write_root_velocity_to_sim_index( self, @@ -973,6 +981,8 @@ def write_joint_state_to_sim_index( self._data._reset_pose() self._data._reset_velocity() self._root_view.set_attribute(TT.DOF_POSITION, joint_pos_backend, indices=sim_env_ids) + OvPhysxManager._kinematics_dirty = True + OvPhysxManager._scene_data_backend.transforms_version += 1 self._root_view.set_attribute(TT.DOF_VELOCITY, joint_vel_backend, indices=sim_env_ids) def write_joint_position_to_sim_index( @@ -1023,6 +1033,8 @@ def write_joint_position_to_sim_index( self._data._reset_pose() self._data._reset_velocity() self._root_view.set_attribute(TT.DOF_POSITION, joint_pos_backend, indices=sim_env_ids) + OvPhysxManager._kinematics_dirty = True + OvPhysxManager._scene_data_backend.transforms_version += 1 def write_joint_position_to_sim_mask( self, @@ -1074,6 +1086,8 @@ def write_joint_position_to_sim_mask( self._data._reset_pose() self._data._reset_velocity() self._root_view.set_attribute(TT.DOF_POSITION, joint_pos_backend, mask=env_mask_wp) + OvPhysxManager._kinematics_dirty = True + OvPhysxManager._scene_data_backend.transforms_version += 1 def write_joint_velocity_to_sim_index( self, @@ -1244,6 +1258,8 @@ def write_joint_state_to_sim_mask( self._data._reset_pose() self._data._reset_velocity() self._root_view.set_attribute(TT.DOF_POSITION, joint_pos_backend, mask=env_mask_wp) + OvPhysxManager._kinematics_dirty = True + OvPhysxManager._scene_data_backend.transforms_version += 1 self._root_view.set_attribute(TT.DOF_VELOCITY, joint_vel_backend, mask=env_mask_wp) """ diff --git a/source/isaaclab_ov/isaaclab_ov/assets/articulation/articulation_data.py b/source/isaaclab_ov/isaaclab_ov/assets/articulation/articulation_data.py index ae87c4fdefab..4dea95271e4c 100644 --- a/source/isaaclab_ov/isaaclab_ov/assets/articulation/articulation_data.py +++ b/source/isaaclab_ov/isaaclab_ov/assets/articulation/articulation_data.py @@ -200,6 +200,7 @@ def _ensure_fk_fresh(self) -> None: physx_instance = OvPhysxManager.get_physx_instance() if physx_instance is not None: physx_instance.update_articulations_kinematic() + OvPhysxManager._kinematics_dirty = False self._fk_timestamp = self._sim_timestamp def _reset_pose(self, from_link: bool = True) -> None: diff --git a/source/isaaclab_ov/isaaclab_ov/assets/rigid_object/rigid_object.py b/source/isaaclab_ov/isaaclab_ov/assets/rigid_object/rigid_object.py index 262c2f57c214..468ec45722f3 100644 --- a/source/isaaclab_ov/isaaclab_ov/assets/rigid_object/rigid_object.py +++ b/source/isaaclab_ov/isaaclab_ov/assets/rigid_object/rigid_object.py @@ -375,6 +375,7 @@ def write_root_link_pose_to_sim_index( self._root_view.set_attribute( TT.RIGID_BODY_POSE, self.data._root_link_pose_w.data.view(wp.float32), indices=sim_env_ids ) + OvPhysxManager._scene_data_backend.transforms_version += 1 def write_root_link_pose_to_sim_mask( self, @@ -415,6 +416,7 @@ def write_root_link_pose_to_sim_mask( self._root_view.set_attribute( TT.RIGID_BODY_POSE, self.data._root_link_pose_w.data.view(wp.float32), mask=env_mask_wp ) + OvPhysxManager._scene_data_backend.transforms_version += 1 def write_root_com_pose_to_sim_index( self, @@ -457,6 +459,7 @@ def write_root_com_pose_to_sim_index( self._root_view.set_attribute( TT.RIGID_BODY_POSE, self.data._root_link_pose_w.data.view(wp.float32), indices=sim_env_ids ) + OvPhysxManager._scene_data_backend.transforms_version += 1 def write_root_com_pose_to_sim_mask( self, @@ -498,6 +501,7 @@ def write_root_com_pose_to_sim_mask( self._root_view.set_attribute( TT.RIGID_BODY_POSE, self.data._root_link_pose_w.data.view(wp.float32), mask=env_mask_wp ) + OvPhysxManager._scene_data_backend.transforms_version += 1 def write_root_com_velocity_to_sim_index( self, diff --git a/source/isaaclab_ov/isaaclab_ov/assets/rigid_object_collection/rigid_object_collection.py b/source/isaaclab_ov/isaaclab_ov/assets/rigid_object_collection/rigid_object_collection.py index e57ffedfc181..8b4d34d43bf6 100644 --- a/source/isaaclab_ov/isaaclab_ov/assets/rigid_object_collection/rigid_object_collection.py +++ b/source/isaaclab_ov/isaaclab_ov/assets/rigid_object_collection/rigid_object_collection.py @@ -418,6 +418,7 @@ def write_body_link_pose_to_sim_index( self.data._reset_pose() # set into simulation self._binding_write(TT.LINK_POSE, self.data._body_link_pose_w.data, env_ids=env_ids) + OvPhysxManager._scene_data_backend.transforms_version += 1 def write_body_link_pose_to_sim_mask( self, @@ -469,6 +470,7 @@ def write_body_link_pose_to_sim_mask( self.data._reset_pose() # set into simulation self._binding_write(TT.LINK_POSE, self.data._body_link_pose_w.data, env_ids=env_ids) + OvPhysxManager._scene_data_backend.transforms_version += 1 def write_body_com_pose_to_sim_index( self, @@ -515,6 +517,7 @@ def write_body_com_pose_to_sim_index( self.data._reset_pose(from_link=False) # set into simulation (OVPhysX only exposes the link frame) self._binding_write(TT.LINK_POSE, self.data._body_link_pose_w.data, env_ids=env_ids) + OvPhysxManager._scene_data_backend.transforms_version += 1 def write_body_com_pose_to_sim_mask( self, @@ -569,6 +572,7 @@ def write_body_com_pose_to_sim_mask( self.data._reset_pose(from_link=False) # set into simulation (OVPhysX only exposes the link frame) self._binding_write(TT.LINK_POSE, self.data._body_link_pose_w.data, env_ids=env_ids) + OvPhysxManager._scene_data_backend.transforms_version += 1 def write_body_com_velocity_to_sim_index( self, diff --git a/source/isaaclab_ov/isaaclab_ov/physics/ovphysx_manager.py b/source/isaaclab_ov/isaaclab_ov/physics/ovphysx_manager.py index a412a0a313ac..661272c0da6b 100644 --- a/source/isaaclab_ov/isaaclab_ov/physics/ovphysx_manager.py +++ b/source/isaaclab_ov/isaaclab_ov/physics/ovphysx_manager.py @@ -90,39 +90,15 @@ def _prepare_default_cache_dir(cache_dir: str) -> str: class OvPhysxSceneDataBackend(SceneDataBackend): """Scene-data backend for the OVPhysX physics manager. - Mirrors the contract of ``PhysxSceneDataBackend`` but adapts to the - ovphysx wheel's one-pattern-per-binding API: each distinct env-wildcard - rigid-body prim path produces its own ``TT.RIGID_BODY_POSE`` binding. - :attr:`transforms` reads each binding into its pre-allocated float32 - staging buffer and concatenates them into a single ``wp.transformf`` - array. - - The merged-buffer + staging-buffer separation is required because the - wheel's ``TensorBinding.read(dst)`` writes into ``dst`` only when - ``dst.shape == binding.shape``, so we cannot read directly into a slice - of the merged buffer. - - Unlike PhysX -- which receives a live :class:`omni.physics.tensors.SimulationView` - via a ``simulation_view`` property setter and discovers prims lazily -- - OVPhysX wires bindings through an explicit :meth:`setup` call that - takes the live ``ovphysx.PhysX`` handle and the USD stage. The wheel - exposes a ``physx + stage`` pair rather than a single ``SimulationView``, - so a property setter would have to either bundle the two or fire on the - second assignment; the explicit call keeps the lifecycle obvious. + Each rigid-body binding reads directly into its portion of one native pose + buffer. Pointer aliases preserve the binding shape without staging or merging. """ def __init__(self): - self._physx = None - # Each entry: ``{"pattern": str, "pose": TensorBinding, - # "pose_buf": wp.array (float32, (N, 7)), - # "pose_buf_transformf": wp.array (transformf, (N,)), - # "row_offset": int, "row_count": int}``. - # The ``pose_buf_transformf`` view aliases ``pose_buf`` via zero-copy - # ``wp.array(ptr=...)``; cached at setup time so per-step reads in - # :attr:`transforms` don't churn Python allocations. - self._rigid_bindings: list[dict[str, Any]] = [] - self._merged_transforms: wp.array | None = None - self._scene_data = SceneDataFormat.Transform() + self._rigid_bindings: list[tuple[OvPhysxView, wp.array]] = [] + self._transforms = SceneDataFormat.Transform() + self.transforms_version = 0 + self._poses_version = -1 self._points_data = SceneDataFormat.Points() self._deformable_bindings: list[dict[str, Any]] = [] self._geometry_paths: list[str] = [] @@ -131,16 +107,14 @@ def __init__(self): @property def transform_count(self) -> int: - """Sum of per-binding row counts.""" - return sum(int(entry["row_count"]) for entry in self._rigid_bindings) + """Number of poses in the native publication.""" + poses = self._transforms.transforms + return 0 if poses is None else len(poses) @property def transform_paths(self) -> list[str]: """Concatenated ``prim_paths`` across all bindings, in registration order.""" - paths: list[str] = [] - for entry in self._rigid_bindings: - paths.extend(list(entry["pose"].prim_paths)) - return paths + return [path for view, _ in self._rigid_bindings for path in view.prim_paths] def setup(self, physx, stage, device: str) -> None: """Discover RigidBodyAPI prims, dedup by env-wildcard form, create one binding per pattern. @@ -148,13 +122,13 @@ def setup(self, physx, stage, device: str) -> None: Args: physx: Live ``ovphysx.PhysX`` instance (the wheel handle). stage: USD stage to traverse for RigidBodyAPI prims. - device: Warp device string used to allocate the staging and merged buffers. + device: Warp device string used to allocate the published buffers. """ from isaaclab_ov import tensor_types as TT # local: keep heavy ovphysx out of module load - self._physx = physx self._rigid_bindings = [] - self._merged_transforms = None + self._transforms.transforms = None + self.transforms_version += 1 self._deformable_bindings = [] self._geometry_paths = [] self._geometry_counts = [] @@ -169,49 +143,30 @@ def setup(self, physx, stage, device: str) -> None: if prim.HasAPI(UsdPhysics.RigidBodyAPI): patterns.add(re.sub(r"/World/envs/env_\d+", "/World/envs/env_*", prim.GetPath().pathString)) - # Rigid discovery may be empty for deformable-only scenes; still set up - # deformable nodal bindings so SceneData geometry export stays available. - if patterns: - # One pose binding per distinct pattern. - total_count = 0 - for pattern in sorted(patterns): - try: - view = OvPhysxView(physx, pattern=pattern, device=device) - pose_binding = view.binding_for(TT.RIGID_BODY_POSE) - except Exception as exc: - logger.warning("Failed to create RIGID_BODY_POSE binding for %s: %s", pattern, exc) - continue - row_count = int(pose_binding.shape[0]) - if row_count == 0: - logger.debug("Pattern %s matched 0 rigid bodies; skipping.", pattern) - view.close() - continue - pose_buf = wp.zeros(pose_binding.shape, dtype=wp.float32, device=device) - # Zero-copy reinterpret of the (N, 7) float32 staging buffer as (N,) wp.transformf. - # Same pointer + layout; transformf is 7 float32s (pos.xyz + quat.xyzw). Cached - # so per-step ``transforms`` reads don't reallocate the view object. - pose_buf_transformf = wp.array( - ptr=pose_buf.ptr, - shape=(row_count,), + views = [] + for pattern in sorted(patterns): + view = OvPhysxView(physx, pattern=pattern, device=device) + view.binding_for(TT.RIGID_BODY_POSE) + if view.count == 0: + logger.debug("Pattern %s matched 0 rigid bodies; skipping.", pattern) + view.close() + continue + views.append(view) + + if views: + poses = wp.empty(sum(view.count for view in views), dtype=wp.transformf, device=device) + self._transforms.transforms = poses + offset = 0 + for view in views: + buffer = wp.array( + ptr=poses.ptr + offset * wp.types.type_size_in_bytes(wp.transformf), + shape=(view.count,), dtype=wp.transformf, - device=str(pose_buf.device), + device=device, copy=False, ) - self._rigid_bindings.append( - { - "pattern": pattern, - "view": view, - "pose": pose_binding, - "pose_buf": pose_buf, - "pose_buf_transformf": pose_buf_transformf, - "row_offset": total_count, - "row_count": row_count, - } - ) - total_count += row_count - - if total_count > 0: - self._merged_transforms = wp.zeros((total_count,), dtype=wp.transformf, device=device) + self._rigid_bindings.append((view, buffer)) + offset += view.count self._setup_deformable_bindings(physx, stage, device) @@ -355,40 +310,13 @@ def geometry_counts(self) -> list[int]: @property def transforms(self) -> SceneDataFormat.Transform: - """Read all bindings into the merged buffer; return as ``SceneDataFormat.Transform``. - - Each binding's float32 ``(N, 7)`` read buffer is reinterpreted as ``(N,)`` of - ``wp.transformf`` (zero-copy via ``wp.array(ptr=..., dtype=wp.transformf)``, - cached on the entry at setup time) and copied into the merged buffer at the - binding's ``row_offset``. - - Returns: - ``SceneDataFormat.Transform`` whose ``transforms`` field is a - ``wp.array(dtype=wp.transformf)`` of length :attr:`transform_count`. - Each ``wp.transformf`` row carries position [m] followed by - quaternion (xyzw, unit). ``transforms`` is ``None`` when no - bindings are wired. - """ - if self._merged_transforms is None or not self._rigid_bindings: - self._scene_data.transforms = self._merged_transforms - return self._scene_data - - for entry in self._rigid_bindings: - try: - entry["view"].read_into("rigid_body_pose", entry["pose_buf"]) - except Exception as exc: - logger.warning("RIGID_BODY_POSE read failed for %s: %s", entry["pattern"], exc) - continue - wp.copy( - self._merged_transforms, - entry["pose_buf_transformf"], - dest_offset=int(entry["row_offset"]), - src_offset=0, - count=int(entry["row_count"]), - ) - - self._scene_data.transforms = self._merged_transforms - return self._scene_data + """Publish native rigid-body poses [m, xyzw].""" + if self._poses_version != self.transforms_version: + OvPhysxManager.pre_render() + for view, buffer in self._rigid_bindings: + view.read_into("rigid_body_pose", buffer) + self._poses_version = self.transforms_version + return self._transforms class OvPhysxBackend: @@ -493,6 +421,7 @@ class OvPhysxManager(PhysicsManager): _pending_clones: ClassVar[list[tuple[str, list[str], list[CloneTransform]]]] = [] _atexit_registered: ClassVar[bool] = False _scene_data_backend: ClassVar[OvPhysxSceneDataBackend | None] = None + _kinematics_dirty: ClassVar[bool] = False # Gravity currently applied to the running scene [m/s^2]. Seeded from ``SimulationCfg.gravity`` # in :meth:`initialize` and refreshed by :meth:`set_gravity`. ``cfg.gravity`` stays the nominal # value that randomization terms resample from, so live updates must not be written back to it. @@ -624,6 +553,7 @@ def initialize(cls, sim_context: SimulationContext) -> None: # and the USD stage are live. Matches PhysX's pattern of constructing # the backend during ``initialize()``. cls._scene_data_backend = OvPhysxSceneDataBackend() + cls._kinematics_dirty = False @classmethod def reset(cls, soft: bool = False) -> None: @@ -645,11 +575,23 @@ def reset(cls, soft: bool = False) -> None: cls.dispatch_event(PhysicsEvent.STOP, payload={}) cls._warmup_and_load() cls.dispatch_event(PhysicsEvent.PHYSICS_READY, payload={}) + cls._kinematics_dirty = True + cls._scene_data_backend.transforms_version += 1 @classmethod def forward(cls) -> None: - """No-op -- ovphysx does not have a fabric/rendering pipeline.""" - pass + """Evaluate and publish state changes made without stepping physics.""" + if cls.backend is not None and cls.backend.physx is not None: + cls.backend.physx.update_articulations_kinematic() + cls._kinematics_dirty = False + cls._scene_data_backend.transforms_version += 1 + + @classmethod + def pre_render(cls) -> None: + """Finish native kinematics before SDP publishes manually written joint poses.""" + if cls._kinematics_dirty and cls.backend is not None and cls.backend.physx is not None: + cls.backend.physx.update_articulations_kinematic() + cls._kinematics_dirty = False @classmethod def step(cls) -> None: @@ -659,6 +601,8 @@ def step(cls) -> None: dt = cls.get_physics_dt() cls.backend.physx.step_sync(dt=dt) cls.backend.physx.update_articulations_kinematic() + cls._kinematics_dirty = False + cls._scene_data_backend.transforms_version += 1 PhysicsManager._sim_time += dt @staticmethod @@ -694,6 +638,7 @@ def close(cls) -> None: # belong to the runtime instance just released. The next # SimulationContext re-creates it in initialize(). cls._scene_data_backend = None + cls._kinematics_dirty = False cls._next_control_ordinal = 2 @classmethod diff --git a/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py b/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py index 8c026265a9ed..729bac191a22 100644 --- a/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py +++ b/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py @@ -69,6 +69,7 @@ from isaaclab.cloner import ClonePlan from isaaclab.cloner import query as clone_query from isaaclab.renderers import BaseRenderer, RenderBufferKind, RenderBufferSpec +from isaaclab.scene_data import SceneDataFormat from isaaclab.sim import SimulationContext from isaaclab.utils.warp.warp_math import convert_camera_frame_orientation_convention_wp @@ -86,7 +87,6 @@ create_camera_transforms_kernel, extract_all_tiles_kernel, generate_random_colors_from_ids_kernel, - sync_newton_transforms_kernel, ) from isaaclab_ov.renderers.ovrtx_shader_cache import redirect_shader_cache from isaaclab_ov.renderers.ovrtx_usd import ( @@ -389,7 +389,8 @@ def __init__(self, cfg: OVRTXRendererCfg): # Shared by both paths. The legacy-only binding handles that pair with these live in # _init_fields_legacy instead; the ovstage path drives the same offsets and counts # through its stage queries. - self._object_newton_indices: wp.array | None = None + self._sdp = SimulationContext.instance().get_scene_data_provider() + self._transform_version = -1 self._object_scales: wp.array | None = None self._object_scales_by_path: dict[str, tuple[float, float, float]] = {} self._deformable_particle_offsets: list[int] = [] @@ -503,10 +504,10 @@ def _clone_targets_env_roots(self) -> bool: return any(destination.format(0) == "/World/envs/env_0" for destination in self._clone_plan.destinations) def _capture_object_scales(self, stage: Any, plan: ClonePlan) -> None: - """Record composed world scales of scaled environment prims before the stage is exported. + """Record composed world scales beneath the plan's prototypes and shared roots before export. - The per-frame object transform write rebuilds each body's matrix from a Newton - ``transformf``, which carries only translation and rotation, so any scale authored on the + The per-frame object transform write rebuilds each body's matrix from an SDP + pose, which carries only translation and rotation, so any scale authored on the USD prim is lost once that write lands. Capturing the composed scale here, while the full stage is still live, lets :meth:`_create_object_scale_array` fold it back in. @@ -522,18 +523,15 @@ def _capture_object_scales(self, stage: Any, plan: ClonePlan) -> None: from pxr import Gf, Usd, UsdGeom - envs_prim = stage.GetPrimAtPath("/World/envs") - if not envs_prim.IsValid(): - return - xform_cache = UsdGeom.XformCache() - for prim in Usd.PrimRange(envs_prim): - if not prim.IsA(UsdGeom.Xformable): - continue - scale = Gf.Transform(xform_cache.GetLocalToWorldTransform(prim)).GetScale() - scale = (float(scale[0]), float(scale[1]), float(scale[2])) - if not all(math.isclose(axis, 1.0, rel_tol=1e-6, abs_tol=1e-6) for axis in scale): - self._object_scales_by_path[str(prim.GetPath())] = scale + for root in (*plan.sources, *plan.global_paths): + for prim in Usd.PrimRange(stage.GetPrimAtPath(root)): + if not prim.IsA(UsdGeom.Xformable): + continue + scale = Gf.Transform(xform_cache.GetLocalToWorldTransform(prim)).GetScale() + scale = (float(scale[0]), float(scale[1]), float(scale[2])) + if not all(math.isclose(axis, 1.0, rel_tol=1e-6, abs_tol=1e-6) for axis in scale): + self._object_scales_by_path[str(prim.GetPath())] = scale # OVRTX creates non-source rows after this stage is exported, so those destination prims # cannot be traversed above. Clone queries retain the plan's nearest-owner semantics. @@ -544,10 +542,10 @@ def _capture_object_scales(self, stage: Any, plan: ClonePlan) -> None: self._object_scales_by_path.setdefault(clone_path, scale) def _create_object_scale_array(self, object_paths: list[str]) -> wp.array: - """Build the device scale array aligned with the Newton body binding order. + """Build the device scale array aligned with the published body binding order. Args: - object_paths: Bound body prim paths, ordered to match the Newton index array. + object_paths: Bound body prim paths, ordered to match the SDP publication. Returns: Per-body scale factors, shape ``[len(object_paths)]``, unit where no scale was authored. @@ -558,14 +556,10 @@ def _create_object_scale_array(self, object_paths: list[str]) -> wp.array: def _init_fields_legacy(self) -> None: """Initialize the legacy-path instance fields. - Counterpart to :meth:`_init_fields_ovstage`. Only fields the ovstage path never touches live - here: the ``bind_attribute``/``bind_array_attribute`` handles and the caller-owned object - transform buffer. State shared by both paths (``_object_newton_indices``, the particle - offset/count lists) stays in :meth:`__init__`. + Only binding handles live here; geometry offsets and counts are shared with ovstage. """ self._camera_xform_binding = None self._object_xform_binding = None - self._object_transform_buffer: wp.array | None = None self._deformable_points_binding = None self._particle_points_binding = None self._particle_workaround_applied = False @@ -716,36 +710,9 @@ def _update_scene_partitions_after_clone(self, num_envs: int): logger.info("Written omni:scenePartition to %d cameras", num_envs) def _setup_xform_bindings_legacy(self): - """Setup OVRTX bindings for scene objects to sync with Newton physics.""" - try: - from isaaclab_newton.physics import NewtonManager - except ImportError: - logger.debug("NewtonManager not available, skipping object bindings") - return - - if SimulationContext.instance() is None: - logger.info("No active simulation context, will not set up ovrtx object bindings for newton") - return - - newton_model = NewtonManager.get_model() - if newton_model is None: - logger.debug("Newton model not available, skipping object bindings") - return - - all_body_paths = getattr(newton_model, "body_label", None) - if all_body_paths is None: - logger.info("Newton model has no body_label, skipping object bindings") - return - - object_paths = [] - newton_indices = [] - for idx, path in enumerate(all_body_paths): - if "/World/envs/" in path and self._camera_rel_path not in path and "GroundPlane" not in path: - object_paths.append(path) - newton_indices.append(idx) - - if len(object_paths) == 0: - logger.info("No dynamic objects found for binding") + """Bind the body paths published through SDP.""" + object_paths = self._sdp.backend.transform_paths + if not object_paths: return self._object_xform_binding = self.backend.renderer.bind_attribute( @@ -764,9 +731,7 @@ def _setup_xform_bindings_legacy(self): if self._object_xform_binding is None: raise RuntimeError("Failed to create OVRTX object bindings") - self._object_newton_indices = wp.array(newton_indices, dtype=wp.int32, device=self._device) self._object_scales = self._create_object_scale_array(object_paths) - self._object_transform_buffer = wp.zeros(len(newton_indices), dtype=wp.mat44d, device=self._device) def _setup_deformable_bindings_legacy(self, num_envs: int): """Setup OVRTX bindings for Newton deformable bodies. @@ -1111,41 +1076,23 @@ def set_outputs(self, render_data: OVRTXCameraRenderData, output_data: dict[str, ) def _update_transforms_legacy(self) -> None: - """Sync transforms to OVRTX.""" - if ( - self._object_xform_binding is None - or self._object_newton_indices is None - or self._object_scales is None - or self._object_transform_buffer is None - ): + """Write SDP's requested matrix layout without another conversion.""" + if self._object_xform_binding is None: return - - # If self._object_newton_indices is not None, then Newton's the current physics backend - - from isaaclab_newton.physics import NewtonManager - - newton_state = NewtonManager.get_state() - if newton_state is None: - raise RuntimeError("Newton state should not be None") - - body_q = getattr(newton_state, "body_q", None) - if body_q is None: + transforms = SceneDataFormat.TransposedMatrix44d() + if not self._sdp.get_transforms(transforms, scales=self._object_scales): + return + if self._transform_version == self._sdp.backend.transforms_version: return - - wp.launch( - kernel=sync_newton_transforms_kernel, - dim=len(self._object_newton_indices), - inputs=[self._object_transform_buffer, self._object_newton_indices, body_q, self._object_scales], - device=self._device, - ) # Blocking ``write()`` so the buffer stays valid until OVRTX finishes reading it. # ``DataAccess.ASYNC`` + the Warp CUDA stream let OVRTX read in place and wait # on-GPU for the kernel; ``SYNC`` is rejected for GPU buffers. self._object_xform_binding.write( - self._object_transform_buffer, + transforms.matrices, data_access=DataAccess.ASYNC, cuda_stream=self._warp_device.stream.cuda_stream, ) + self._transform_version = self._sdp.backend.transforms_version def _update_geometries_legacy(self) -> None: """Sync geometries to OVRTX.""" @@ -1711,7 +1658,6 @@ def _safe_unbind(binding, name: str) -> None: self._camera_xform_binding = None _safe_unbind(self._object_xform_binding, "object transforms") self._object_xform_binding = None - self._object_transform_buffer = None _safe_unbind(self._deformable_points_binding, "deformable points") self._deformable_points_binding = None _safe_unbind(self._particle_points_binding, "particle points") @@ -2122,36 +2068,9 @@ def _update_scene_partitions_after_clone_ovstage(self, num_envs: int): logger.info("Written omni:scenePartition to %d cameras", num_envs) def _setup_xform_bindings_ovstage(self) -> None: - """Setup OVRTX bindings for scene objects to sync with Newton physics (ovstage path).""" - try: - from isaaclab_newton.physics import NewtonManager - except ImportError: - logger.debug("NewtonManager not available, skipping object bindings") - return - - if SimulationContext.instance() is None: - logger.info("No active simulation context, will not set up ovrtx object bindings for newton") - return - - newton_model = NewtonManager.get_model() - if newton_model is None: - logger.debug("Newton model not available, skipping object bindings") - return - - all_body_paths = getattr(newton_model, "body_label", None) - if all_body_paths is None: - logger.info("Newton model has no body_label, skipping object bindings") - return - - object_paths = [] - newton_indices = [] - for idx, path in enumerate(all_body_paths): - if "/World/envs/" in path and self._camera_rel_path not in path and "GroundPlane" not in path: - object_paths.append(path) - newton_indices.append(idx) - - if len(object_paths) == 0: - logger.info("No dynamic objects found for binding") + """Bind the body paths published through SDP.""" + object_paths = self._sdp.backend.transform_paths + if not object_paths: return self._object_paths_list = self.backend.paths.create_path_list_from_strings(object_paths) @@ -2168,7 +2087,6 @@ def _setup_xform_bindings_ovstage(self) -> None: if self._object_xform_query is None: raise RuntimeError("Failed to create OVRTX object bindings") - self._object_newton_indices = wp.array(newton_indices, dtype=wp.int32, device=self._device) self._object_scales = self._create_object_scale_array(object_paths) def _setup_deformable_bindings_ovstage(self, num_envs: int) -> None: @@ -2360,45 +2278,25 @@ def _setup_particle_bindings_ovstage(self) -> None: raise RuntimeError("Failed to create OVRTX particle point bindings") def _update_transforms_ovstage(self) -> None: - if self._object_xform_query is None or self._object_newton_indices is None or self._object_scales is None: + """Write SDP's matrix layout through the active ovstage ordinal.""" + if self._object_xform_query is None: return - - # If self._object_newton_indices is not None, then Newton's the current physics backend - - from isaaclab_newton.physics import NewtonManager - - newton_state = NewtonManager.get_state() - if newton_state is None: - raise RuntimeError("Newton state should not be None") - - body_q = getattr(newton_state, "body_q", None) - if body_q is None: + transforms = SceneDataFormat.TransposedMatrix44d() + if not self._sdp.get_transforms(transforms, scales=self._object_scales): return - - num_objects = len(self._object_newton_indices) - object_transforms = wp.empty(num_objects, dtype=wp.mat44d, device=self._device) - wp.launch( - kernel=sync_newton_transforms_kernel, - dim=num_objects, - inputs=[object_transforms, self._object_newton_indices, body_q, self._object_scales], - device=self._device, - ) - # The tensor is handed over zero-copy, so ovstage reads ``object_transforms`` in place and - # must not do so until the kernel above has landed. Passing the producing Warp stream as - # ``cuda_stream`` gives producer ordering: ovstage drains the work already queued on that - # stream before it touches the tensor. That replaces the device-wide - # ``wp.synchronize_device()`` with stream-scoped ordering and removes the host copy; it is - # not a nonblocking handoff, and the ``.wait()`` below can still block the calling thread. - # A GPU-side wait would need the event-based API instead. + if self._transform_version == self._sdp.backend.transforms_version: + return + # Stream-ordered zero-copy handoff; wait until OVStage has consumed the shared buffer. self.backend.stage.write_attribute( self._object_xform_query, "omni:xform", ordinal=self._current_ordinal, - tensors=xform_tensor_from_warp(object_transforms), + tensors=xform_tensor_from_warp(transforms.matrices), is_array=False, semantic=ovstage.AttributeSemantic.MATRIX, cuda_stream=self._warp_device.stream.cuda_stream, ).wait() + self._transform_version = self._sdp.backend.transforms_version def _update_geometries_ovstage(self) -> None: if self._deformable_points_query is not None or self._particle_points_query is not None: @@ -2608,7 +2506,6 @@ def _safe_destroy_path_list(path_list, name: str) -> None: _safe_destroy_path_list(self._cable_paths_list, "cable paths") self._cable_paths_list = None - self._object_newton_indices = None self._object_scales = None self._object_scales_by_path = {} # Descriptors alias ``_cable_points``; drop them before the buffer so no cached diff --git a/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer_kernels.py b/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer_kernels.py index 9d83de2e4757..d5e598ebb5ef 100644 --- a/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer_kernels.py +++ b/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer_kernels.py @@ -175,33 +175,6 @@ def generate_random_colors_from_ids_kernel( output_colors[i, j, k] = random_color_from_id_wp(input_ids[i, j, k]) -@wp.kernel -def sync_newton_transforms_kernel( - ovrtx_transforms: wp.array(dtype=wp.mat44d), # type: ignore - newton_body_indices: wp.array(dtype=wp.int32), # type: ignore - newton_body_q: wp.array(dtype=wp.transformf), # type: ignore - object_scales: wp.array(dtype=wp.vec3f), # type: ignore -): - """Sync Newton physics body transforms to OVRTX 4x4 column-major matrices. - - A Newton ``transformf`` holds only translation and rotation, so the authored USD scale is - reapplied here to keep it from being overwritten with unit scale. - """ - i = wp.tid() - body_idx = newton_body_indices[i] - transform = newton_body_q[body_idx] - scale = object_scales[i] - ovrtx_transforms[i] = wp.mat44d( - wp.transpose( - wp.transform_compose( - wp.transform_get_translation(transform), - wp.transform_get_rotation(transform), - scale, - ) - ) - ) - - @wp.func def _cable_capsule_endpoint_world( shape_id: int, diff --git a/source/isaaclab_ov/test/physics/test_ovphysx_scene_data_backend.py b/source/isaaclab_ov/test/physics/test_ovphysx_scene_data_backend.py index aee9f584ac93..1787f108cb13 100644 --- a/source/isaaclab_ov/test/physics/test_ovphysx_scene_data_backend.py +++ b/source/isaaclab_ov/test/physics/test_ovphysx_scene_data_backend.py @@ -23,12 +23,14 @@ @pytest.fixture(autouse=True) def _native_backend(monkeypatch): - from isaaclab_ov.physics.ovphysx_manager import OvPhysxBackend, OvPhysxManager + from isaaclab_ov.physics.ovphysx_manager import OvPhysxBackend, OvPhysxManager, OvPhysxSceneDataBackend backend = OvPhysxBackend.__new__(OvPhysxBackend) backend.physx = None backend.stage = None monkeypatch.setattr(OvPhysxManager, "backend", backend) + monkeypatch.setattr(OvPhysxManager, "_scene_data_backend", OvPhysxSceneDataBackend()) + monkeypatch.setattr(OvPhysxManager, "_kinematics_dirty", False) @pytest.fixture(autouse=True) @@ -371,9 +373,12 @@ def test_manager_forced_rewarm_invalidates_bindings_before_loading(monkeypatch): lambda event, payload=None: calls.append(event), ) + version = OvPhysxManager._scene_data_backend.transforms_version OvPhysxManager.reset() assert calls == [PhysicsEvent.STOP, "warmup", PhysicsEvent.PHYSICS_READY] + assert OvPhysxManager._scene_data_backend.transforms_version > version + assert OvPhysxManager._kinematics_dirty @pytest.mark.parametrize( @@ -431,6 +436,7 @@ def pinned_config(*, num_threads=None, cooked_collider_cache_dir=None, carbonite OvPhysxManager.backend.physx = physx monkeypatch.setattr(OvPhysxManager, "get_physics_dt", lambda: 0.02) monkeypatch.setattr(PhysicsManager, "_sim_time", 0.0) + version = OvPhysxManager._scene_data_backend.transforms_version OvPhysxManager.step() OvPhysxManager._prepare_physx_for_stage_reuse() @@ -440,6 +446,36 @@ def pinned_config(*, num_threads=None, cooked_collider_cache_dir=None, carbonite assert physx.constructor["config"].cooked_collider_cache_dir == cache_dir assert physx.calls == [("step_sync", 0.02), ("update_articulations_kinematic",), ("reset_stage",), ("wait_op", 23)] assert PhysicsManager._sim_time == 0.02 + assert OvPhysxManager._scene_data_backend.transforms_version > version + assert not OvPhysxManager._kinematics_dirty + + +def test_transforms_finish_dirty_kinematics_before_native_reads(monkeypatch): + """Direct SDP consumers refresh pending FK once, before reading native poses.""" + import warp as wp + from isaaclab_ov.physics import OvPhysxManager + + from isaaclab.scene_data import SceneDataFormat, SceneDataProvider + + calls = [] + OvPhysxManager.backend.physx = SimpleNamespace(update_articulations_kinematic=lambda: calls.append("fk")) + backend = OvPhysxManager._scene_data_backend + poses = wp.zeros(1, dtype=wp.transformf, device="cpu") + backend._transforms.transforms = poses + backend._rigid_bindings = [(SimpleNamespace(read_into=lambda *args: calls.append("read")), poses)] + sdp = SceneDataProvider(backend) + monkeypatch.setattr(OvPhysxManager, "_kinematics_dirty", True) + sdp.get_transforms(SceneDataFormat.Transform()) + sdp.get_transforms(SceneDataFormat.Transform()) + assert calls == ["fk", "read"] + assert not OvPhysxManager._kinematics_dirty + + version = backend.transforms_version + OvPhysxManager.forward() + assert backend.transforms_version > version + sdp.get_transforms(SceneDataFormat.Transform()) + sdp.get_transforms(SceneDataFormat.Transform()) + assert calls == ["fk", "read", "fk", "read"] def test_manager_serializes_env0_only_stage_in_memory(caplog): @@ -839,352 +875,98 @@ def _stop_at_stage_creation(): assert SimulationContext.instance() is None -def _make_stub_binding(prim_paths: list[str]) -> SimpleNamespace: - """Stub an ovphysx ``TensorBinding`` exposing ``shape``, ``count``, ``prim_paths``, and ``read(dst)``.""" - n = len(prim_paths) - return SimpleNamespace( - shape=(n, 7), - count=n, - prim_paths=list(prim_paths), - read=lambda dst: None, # no-op write; transform_count/paths don't trigger reads. - ) - - -def _bare_backend(): - """Construct an ``OvPhysxSceneDataBackend`` instance bypassing the live-wheel ``__init__``. - - Tests seed ``_rigid_bindings`` and the merged buffer directly, mirroring the - bypass-init pattern used in ``test_newton_manager_visualization_state.py``. - """ - from isaaclab_ov.physics.ovphysx_manager import OvPhysxSceneDataBackend - - return object.__new__(OvPhysxSceneDataBackend) - - -def test_transform_count_sums_across_bindings(): - """``transform_count`` returns the sum of each binding's row count.""" - b = _bare_backend() - b._rigid_bindings = [ - { - "pose": _make_stub_binding(["/World/envs/env_0/Cube", "/World/envs/env_1/Cube"]), - "pose_buf": None, - "row_offset": 0, - "row_count": 2, - }, - {"pose": _make_stub_binding(["/World/envs/env_0/Pole"]), "pose_buf": None, "row_offset": 2, "row_count": 1}, - ] - assert b.transform_count == 3 - - -def test_transform_paths_concatenates_prim_paths(): - """``transform_paths`` concatenates each binding's ``prim_paths`` in registration order.""" - b = _bare_backend() - b._rigid_bindings = [ - { - "pose": _make_stub_binding(["/World/envs/env_0/Cube", "/World/envs/env_1/Cube"]), - "pose_buf": None, - "row_offset": 0, - "row_count": 2, - }, - {"pose": _make_stub_binding(["/World/envs/env_0/Pole"]), "pose_buf": None, "row_offset": 2, "row_count": 1}, - ] - assert b.transform_paths == [ - "/World/envs/env_0/Cube", - "/World/envs/env_1/Cube", - "/World/envs/env_0/Pole", - ] - - -def test_transform_count_zero_when_no_bindings(): - """``transform_count`` returns 0 when the bindings list is empty.""" - b = _bare_backend() - b._rigid_bindings = [] - assert b.transform_count == 0 - - -def test_transform_paths_empty_when_no_bindings(): - """``transform_paths`` returns an empty list when the bindings list is empty.""" - b = _bare_backend() - b._rigid_bindings = [] - assert b.transform_paths == [] - - -def test_setup_creates_one_binding_per_distinct_pattern(monkeypatch): - """``setup(physx, stage, device)`` buckets RigidBodyAPI prims by env-wildcard form. - - For cartpole-shaped scenes (``cart``, ``pole``), expect 2 bindings — one - per distinct env-relative prim path. - """ - from isaaclab_ov.physics.ovphysx_manager import OvPhysxSceneDataBackend - - b = OvPhysxSceneDataBackend() - - # Stage stub: traversal yields four RigidBodyAPI prims (cart/pole across two envs). - paths = [ - "/World/envs/env_0/Robot/cart", - "/World/envs/env_0/Robot/pole", - "/World/envs/env_1/Robot/cart", - "/World/envs/env_1/Robot/pole", - ] - - def fake_traverse(): - for p in paths: - yield _fake_rigid_body_prim(p) +def test_transforms_read_native_slices_only_when_dirty(monkeypatch): + """Native bindings fill one shared pose buffer directly and skip clean publications.""" + import isaaclab_ov.physics.ovphysx_manager as module + import numpy as np + import warp as wp - stage = SimpleNamespace(Traverse=fake_traverse) + from isaaclab.scene_data import SceneDataFormat, SceneDataProvider - created: list[SimpleNamespace] = [] + expected = np.array([[1, 2, 3, 0, 0, 0, 1], [4, 5, 6, 0, 0, 0, 1], [7, 8, 9, 0, 0, 0, 1]], dtype=np.float32) + paths = ["/World/envs/env_0/Cart", "/World/envs/env_1/Cart", "/World/envs/env_0/Pole"] + reads = [] class FakePhysX: def create_tensor_binding(self, pattern, tensor_type): - shape = (2, 7) # 2 envs match each pattern - b = SimpleNamespace( - pattern=pattern, - tensor_type=tensor_type, - shape=shape, - count=2, - prim_paths=[], - read=lambda dst: None, - destroy=lambda: None, - ) - created.append(b) - return b - - # Patch UsdPhysics so HasAPI in the test doesn't depend on the real PXR module. - import isaaclab_ov.physics.ovphysx_manager as om_mod - - monkeypatch.setattr(om_mod, "UsdPhysics", SimpleNamespace(RigidBodyAPI=object())) - # Rigid-body setup uses a SimpleNamespace stage stub; skip deformable discovery. - monkeypatch.setattr(om_mod, "discover_deformables_on_stage", lambda stage: []) - - b.setup(FakePhysX(), stage, "cpu") - - # Cartpole = 2 distinct env-wildcard patterns -> 2 bindings. - assert len(created) == 2 - assert {c.pattern for c in created} == { - "/World/envs/env_*/Robot/cart", - "/World/envs/env_*/Robot/pole", - } - # Per-binding row counts sum to 4. - assert b.transform_count == 4 - + start, end = (0, 2) if pattern.endswith("/Cart") else (2, 3) -def test_transforms_reads_each_binding_and_returns_transform_format(): - """``transforms`` writes each binding's poses into the merged buffer at its offset. + def read(dst): + reads.append((start, dst.ptr)) + wp.copy(dst, wp.array(expected[start:end], dtype=wp.float32, device="cpu")) - The returned struct is ``SceneDataFormat.Transform`` with ``transforms`` set to - the merged ``wp.transformf`` array. - """ - import warp as _wp - - _wp.init() - - from isaaclab_ov.physics.ovphysx_manager import OvPhysxSceneDataBackend - - b = OvPhysxSceneDataBackend() - b._merged_transforms = _wp.zeros((3,), dtype=_wp.transformf, device="cpu") - - # Two bindings: first with 2 rows, second with 1 row. - buf_a = _wp.zeros((2, 7), dtype=_wp.float32, device="cpu") - buf_b = _wp.zeros((1, 7), dtype=_wp.float32, device="cpu") - - def fake_read_a(dst): - # Fill with row-distinct sentinel transforms (pos.x = row index, quat = identity). - import numpy as np - - host = np.array([[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0], [2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]], dtype=np.float32) - _wp.copy(dst, _wp.from_numpy(host, dtype=_wp.float32, device="cpu").reshape((2, 7))) - - def fake_read_b(dst): - import numpy as np - - host = np.array([[3.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]], dtype=np.float32) - _wp.copy(dst, _wp.from_numpy(host, dtype=_wp.float32, device="cpu").reshape((1, 7))) - - # ``pose_buf_transformf`` is the zero-copy transformf view over the float32 staging - # buffer; production code caches it at setup time. Tests mirror that shape here. - buf_a_tf = _wp.array(ptr=buf_a.ptr, shape=(2,), dtype=_wp.transformf, device="cpu", copy=False) - buf_b_tf = _wp.array(ptr=buf_b.ptr, shape=(1,), dtype=_wp.transformf, device="cpu", copy=False) - b._rigid_bindings = [ - { - "pattern": "/World/envs/env_*/Cube", - "pose": SimpleNamespace(read=fake_read_a, prim_paths=["/Cube0", "/Cube1"]), - "view": SimpleNamespace(read_into=lambda name, dst, _r=fake_read_a: _r(dst)), - "pose_buf": buf_a, - "pose_buf_transformf": buf_a_tf, - "row_offset": 0, - "row_count": 2, - }, - { - "pattern": "/World/envs/env_*/Pole", - "pose": SimpleNamespace(read=fake_read_b, prim_paths=["/Pole"]), - "view": SimpleNamespace(read_into=lambda name, dst, _r=fake_read_b: _r(dst)), - "pose_buf": buf_b, - "pose_buf_transformf": buf_b_tf, - "row_offset": 2, - "row_count": 1, - }, - ] - - out = b.transforms - assert out is b._scene_data - assert out.transforms is b._merged_transforms - - merged_host = out.transforms.numpy() # (3,) of transformf -> view as float32 (3, 7) for assertion - # Each transformf is 7 floats (pos.xyz + quat.xyzw). Verify row 0 / 1 / 2 contents. - flat = merged_host.view(" OVRTXRenderer: renderer._exported_usd_string = None renderer._initialized_scene = False renderer._use_ovstage = False + renderer._sdp = SimpleNamespace(backend=SimpleNamespace(transform_paths=[])) renderer._object_scales = None renderer._object_scales_by_path = {} return renderer @@ -308,17 +307,6 @@ def _record_xforms(value: np.ndarray) -> str: np.testing.assert_array_equal(xforms[0], expected) -def test_write_file_creates_parent_directory_and_writes_utf8(tmp_path: Path): - """_write_file creates nested directories and writes UTF-8 content.""" - output_dir = tmp_path / "nested" / "usd" - - _write_file(output_dir, "stage.usda", "#usda 1.0\n") - - output_path = output_dir / "stage.usda" - assert output_path.is_file() - assert output_path.read_text(encoding="utf-8") == "#usda 1.0\n" - - @pytest.mark.parametrize( "clone_plan", [ @@ -361,31 +349,31 @@ def test_prepare_stage_rejects_non_dense_environment_ids(monkeypatch: pytest.Mon _make_ovrtx_renderer_without_backend().prepare_stage(_make_multi_env_stage(2), 2) -def test_capture_object_scales_populates_source_and_destination_scale_array(): - """Projected source scales reach the body array without replacing a real destination scale.""" +@pytest.mark.parametrize("env_template", ["/World/envs/env_{}", "/World/Instances/World_{}"]) +def test_capture_object_scales_populates_source_and_destination_scale_array(env_template): + """Only declared prototype and shared scales reach the body array, independent of namespace.""" stage = Usd.Stage.CreateInMemory() - UsdGeom.Xform.Define(stage, "/World") - UsdGeom.Xform.Define(stage, "/World/envs") - UsdGeom.Xform.Define(stage, "/World/envs/env_0") - UsdGeom.Xform.Define(stage, "/World/envs/env_1") - UsdGeom.Xform.Define(stage, "/World/envs/env_2") - UsdGeom.Xform.Define(stage, "/World/envs/env_0/Object").AddScaleOp().Set(Gf.Vec3d(1.0, 1.0, 8.0)) - UsdGeom.Xform.Define(stage, "/World/envs/env_1/Object").AddScaleOp().Set(Gf.Vec3d(1.0, 1.0, 4.0)) + UsdGeom.Xform.Define(stage, f"{env_template.format(0)}/Object").AddScaleOp().Set(Gf.Vec3d(1, 1, 8)) + UsdGeom.Xform.Define(stage, f"{env_template.format(1)}/Object").AddScaleOp().Set(Gf.Vec3d(1, 1, 4)) + UsdGeom.Xform.Define(stage, "/World/Shared").AddScaleOp().Set(Gf.Vec3d(2, 3, 4)) + UsdGeom.Xform.Define(stage, "/World/envs/Unplanned").AddScaleOp().Set(Gf.Vec3d(5, 6, 7)) renderer = _make_ovrtx_renderer_without_backend() renderer._device = "cpu" plan = ClonePlan( - sources=("/World/envs/env_0",), - destinations=("/World/envs/env_{}",), - clone_mask=torch.ones((1, 3), dtype=torch.bool), - env_ids=torch.arange(3), + sources=(env_template.format(0), env_template.format(1)), + destinations=(env_template, env_template), + clone_mask=np.array([[True, False, True], [False, True, False]]), + env_ids=np.arange(3), + global_paths=("/World/Shared",), ) renderer._capture_object_scales(stage, plan) scales = renderer._create_object_scale_array( - ["/World/envs/env_0/Object", "/World/envs/env_1/Object", "/World/envs/env_2/Object"] + [f"{env_template.format(index)}/Object" for index in range(3)] + ["/World/Shared"] ) - np.testing.assert_allclose(scales.numpy(), np.array([[1.0, 1.0, 8.0], [1.0, 1.0, 4.0], [1.0, 1.0, 8.0]])) + np.testing.assert_allclose(scales.numpy(), [[1, 1, 8], [1, 1, 4], [1, 1, 8], [2, 3, 4]]) + assert "/World/envs/Unplanned" not in renderer._object_scales_by_path def test_prepare_stage_keeps_material_binding_inside_clone_source(monkeypatch: pytest.MonkeyPatch): @@ -433,15 +421,16 @@ def test_prepare_stage_writes_pre_ovrtx_stage_dump(tmp_path: Path, monkeypatch: stage = _make_multi_env_stage(2) renderer = _make_ovrtx_renderer_without_backend() - renderer.cfg.temp_usd_dir = str(tmp_path) + output_dir = tmp_path / "nested" / "usd" + renderer.cfg.temp_usd_dir = str(output_dir) expected_pre_export = stage.ExportToString() renderer.prepare_stage(stage, 2) - pre_stage_path = tmp_path / _PRE_OVRTX_STAGE_FILE + pre_stage_path = output_dir / _PRE_OVRTX_STAGE_FILE assert pre_stage_path.is_file() assert pre_stage_path.read_text(encoding="utf-8") == expected_pre_export - assert (tmp_path / _OVRTX_STAGE_FILE).exists() is False + assert (output_dir / _OVRTX_STAGE_FILE).exists() is False def test_prepare_stage_skips_temp_usd_write_when_temp_usd_dir_unset(monkeypatch: pytest.MonkeyPatch): @@ -575,8 +564,8 @@ def _write_array_attribute(prim_paths: list[str], attribute_name: str, tensors: ] -def test_prepare_stage_stores_clone_plan_and_exports(monkeypatch: pytest.MonkeyPatch): - """prepare_stage stores the clone plan and exports only its source-row content.""" +def test_prepare_stage_exports_only_clone_source_content(monkeypatch: pytest.MonkeyPatch): + """prepare_stage exports only its source-row content.""" num_envs = 4 published = ClonePlan( @@ -593,8 +582,6 @@ def test_prepare_stage_stores_clone_plan_and_exports(monkeypatch: pytest.MonkeyP renderer.prepare_stage(stage, 4) - assert renderer._clone_plan is published - # Only the env_0 source subtree keeps content. The rows clone the env roots themselves, so the # remaining roots are trimmed: OVRTX refuses to clone onto a prim that already exists. _assert_export_contains_env_roots_and_children(renderer._exported_usd_string, [0]) diff --git a/source/isaaclab_ov/test/test_ovrtx_deformable_bindings.py b/source/isaaclab_ov/test/test_ovrtx_deformable_bindings.py index fafa64992b2c..62b02d94e1e9 100644 --- a/source/isaaclab_ov/test/test_ovrtx_deformable_bindings.py +++ b/source/isaaclab_ov/test/test_ovrtx_deformable_bindings.py @@ -116,68 +116,8 @@ def _make_renderer_without_backend(device: str = "cpu") -> tuple[OVRTXRenderer, return renderer, renderer.backend.renderer -def test_points_array_binding_uses_write_not_map(): - """OVRTX array bindings accept ``List[DLTensor]`` via ``write()``, not mapped tensors.""" - binding = _FakePointsBinding("points") - with pytest.raises(RuntimeError, match="do not expose mapped point buffers"): - binding.map() - - -def test_setup_deformable_bindings_binds_surface_mesh_points(monkeypatch: pytest.MonkeyPatch): - """Surface deformable registry entries create OVRTX ``points`` array bindings.""" - renderer, backend = _make_renderer_without_backend() - entry = SimpleNamespace( - prim_path="/World/envs/env_[^/]+/Deformable", - vis_mesh_prim_path="/World/envs/env_[^/]+/Deformable/mesh", - deformable_type="surface", - particle_offsets=[7], - particles_per_body=3, - ) - - monkeypatch.setattr(NewtonManager, "_deformable_registry", [entry]) - - renderer._setup_deformable_bindings_legacy(num_envs=1) - - assert len(backend.calls) == 1 - assert backend.calls[0]["prim_paths"] == ["/World/envs/env_0/Deformable/mesh"] - assert backend.calls[0]["attribute_name"] == "points" - assert backend.calls[0]["dtype"] is np.float32 - assert backend.calls[0]["shape"] == (3,) - assert renderer._deformable_points_binding is backend.bindings["points"] - assert len(backend.writes) == 2 - assert backend.writes[0]["attribute_name"] == "omni:resetXformStack" - assert backend.writes[0]["prim_paths"] == ["/World/envs/env_0/Deformable/mesh"] - assert backend.writes[1]["attribute_name"] == "omni:xform" - assert backend.writes[1]["prim_paths"] == ["/World/envs/env_0/Deformable/mesh"] - assert len(renderer._deformable_particle_counts) == 1 - assert renderer._deformable_particle_counts[0] == 3 - assert renderer._deformable_particle_offsets == [7] - - -def test_setup_deformable_bindings_binds_volume_mesh_points(monkeypatch: pytest.MonkeyPatch): - """Volume deformable registry entries create OVRTX ``points`` bindings.""" - renderer, backend = _make_renderer_without_backend() - entry = SimpleNamespace( - prim_path="/World/envs/env_[^/]+/Deformable", - vis_mesh_prim_path="/World/envs/env_[^/]+/Deformable/mesh", - deformable_type="volume", - particle_offsets=[7], - particles_per_body=3, - ) - - monkeypatch.setattr(NewtonManager, "_deformable_registry", [entry]) - - renderer._setup_deformable_bindings_legacy(num_envs=1) - - assert len(backend.calls) == 1 - assert backend.calls[0]["prim_paths"] == ["/World/envs/env_0/Deformable/mesh"] - assert backend.calls[0]["attribute_name"] == "points" - assert renderer._deformable_points_binding is backend.bindings["points"] - assert renderer._deformable_particle_offsets == [7] - - def test_setup_deformable_bindings_binds_mixed_surface_and_volume_entries(monkeypatch: pytest.MonkeyPatch): - """Surface and volume deformable registry entries bind together with distinct offsets.""" + """Registry metadata binds every surface and volume instance without a USD stage.""" renderer, backend = _make_renderer_without_backend() surface_entry = SimpleNamespace( prim_path="/World/envs/env_[^/]+/DeformableSurface", @@ -194,99 +134,32 @@ def test_setup_deformable_bindings_binds_mixed_surface_and_volume_entries(monkey particles_per_body=3, ) + monkeypatch.setattr("isaaclab.sim.utils.stage.get_current_stage", lambda: None) monkeypatch.setattr(NewtonManager, "_deformable_registry", [surface_entry, volume_entry]) renderer._setup_deformable_bindings_legacy(num_envs=2) - assert backend.calls[0]["prim_paths"] == [ + paths = [ "/World/envs/env_0/DeformableSurface/mesh", "/World/envs/env_1/DeformableSurface/mesh", "/World/envs/env_0/DeformableVolume/mesh", "/World/envs/env_1/DeformableVolume/mesh", ] - assert renderer._deformable_particle_offsets == [0, 3, 6, 9] - assert renderer._deformable_particle_counts == [3, 3, 3, 3] - - -def test_setup_deformable_bindings_works_without_stage(monkeypatch: pytest.MonkeyPatch): - """Deformable bindings are created from registry metadata without a USD stage.""" - renderer, backend = _make_renderer_without_backend() - entry = SimpleNamespace( - prim_path="/World/envs/env_[^/]+/Deformable", - vis_mesh_prim_path="/World/envs/env_[^/]+/Deformable/mesh", - deformable_type="surface", - particle_offsets=[0], - particles_per_body=3, - ) - - monkeypatch.setattr("isaaclab.sim.utils.stage.get_current_stage", lambda: None) - monkeypatch.setattr(NewtonManager, "_deformable_registry", [entry]) - - renderer._setup_deformable_bindings_legacy(num_envs=1) - assert len(backend.calls) == 1 - assert backend.calls[0]["prim_paths"] == ["/World/envs/env_0/Deformable/mesh"] - assert renderer._deformable_points_binding is backend.bindings["points"] - - -def test_setup_deformable_bindings_binds_all_surface_mesh_instances(monkeypatch: pytest.MonkeyPatch): - """Surface deformable registry entries bind every cloned visual mesh instance.""" - renderer, backend = _make_renderer_without_backend() - entry = SimpleNamespace( - prim_path="/World/envs/env_[^/]+/Deformable", - vis_mesh_prim_path="/World/envs/env_[^/]+/Deformable/mesh", - deformable_type="surface", - particle_offsets=[0, 3, 6, 9], - particles_per_body=3, - ) - - monkeypatch.setattr(NewtonManager, "_deformable_registry", [entry]) - - renderer._setup_deformable_bindings_legacy(num_envs=4) - - expected_paths = [f"/World/envs/env_{i}/Deformable/mesh" for i in range(4)] - assert backend.calls[0]["prim_paths"] == expected_paths - assert renderer._deformable_particle_offsets == [0, 3, 6, 9] - assert renderer._deformable_particle_counts == [3, 3, 3, 3] - - -def test_update_deformable_points_writes_world_particle_positions(monkeypatch: pytest.MonkeyPatch): - """Newton ``particle_q`` slices are handed to OVRTX through :meth:`OVRTXRenderer.update_geometries`.""" - renderer, _backend = _make_renderer_without_backend() - renderer._deformable_points_binding = _FakePointsBinding("points") - renderer._deformable_particle_offsets = [1] - renderer._deformable_particle_counts = [3] - particle_q = wp.array( - [ - wp.vec3f(-1.0, -1.0, -1.0), - wp.vec3f(1.0, 2.0, 3.0), - wp.vec3f(4.0, 5.0, 6.0), - wp.vec3f(7.0, 8.0, 9.0), - ], - dtype=wp.vec3f, - device="cpu", - ) + assert backend.calls[0]["prim_paths"] == paths + assert backend.calls[0]["attribute_name"] == "points" + assert backend.calls[0]["dtype"] is np.float32 + assert backend.calls[0]["shape"] == (3,) + assert [write["attribute_name"] for write in backend.writes] == ["omni:resetXformStack", "omni:xform"] + assert all(write["prim_paths"] == paths for write in backend.writes) + particle_q = wp.array(np.arange(36, dtype=np.float32).reshape(12, 3), dtype=wp.vec3f, device="cpu") monkeypatch.setattr(NewtonManager, "get_state", classmethod(lambda cls: SimpleNamespace(particle_q=particle_q))) - - class _FakeStream: - cuda_stream = 42 - - renderer._warp_device = SimpleNamespace(stream=_FakeStream()) - + renderer._warp_device = SimpleNamespace(stream=SimpleNamespace(cuda_stream=42)) renderer.update_geometries() - - written = renderer._deformable_points_binding.written - assert written is not None - assert len(written) == 1 - assert written[0].ptr == particle_q[1:4].ptr - assert renderer._deformable_points_binding.write_kwargs is not None - assert renderer._deformable_points_binding.write_kwargs["data_access"] is DataAccess.ASYNC - assert renderer._deformable_points_binding.write_kwargs["cuda_stream"] == 42 - assert written[0].numpy().tolist() == [ - [1.0, 2.0, 3.0], - [4.0, 5.0, 6.0], - [7.0, 8.0, 9.0], - ] + written = backend.bindings["points"].written + assert len(written) == len(paths) + for points, offset in zip(written, surface_entry.particle_offsets + volume_entry.particle_offsets, strict=True): + np.testing.assert_array_equal(points.numpy(), particle_q.numpy()[offset : offset + 3]) def test_setup_deformable_bindings_rejects_offset_count_mismatch(monkeypatch: pytest.MonkeyPatch): @@ -334,33 +207,6 @@ def test_update_geometries_rejects_inconsistent_deformable_mapping(monkeypatch: renderer.update_geometries() -def test_setup_particle_points_bindings_binds_mpm_visual_prims(monkeypatch: pytest.MonkeyPatch): - """MPM particle visual prims create an OPTIMIZE ``points`` array binding.""" - renderer, backend = _make_renderer_without_backend() - particle_visual_prims = { - "/World/envs/env_0/Media/Particles": SimpleNamespace(offset=10, count=5), - "/World/envs/env_1/Media/Particles": SimpleNamespace(offset=15, count=5), - } - - monkeypatch.setattr(NewtonManager, "_particle_visual_prims", particle_visual_prims) - - renderer._setup_particle_bindings_legacy() - - assert len(backend.calls) == 1 - assert backend.calls[0]["prim_paths"] == [ - "/World/envs/env_0/Media/Particles", - "/World/envs/env_1/Media/Particles", - ] - assert backend.calls[0]["attribute_name"] == "points" - assert backend.calls[0]["flags"] is BindingFlag.OPTIMIZE - assert renderer._particle_points_binding is backend.bindings["points"] - assert renderer._particle_visual_offsets == [10, 15] - assert renderer._particle_visual_counts == [5, 5] - assert len(backend.writes) == 2 - assert backend.writes[0]["attribute_name"] == "omni:resetXformStack" - assert backend.writes[1]["attribute_name"] == "omni:xform" - - def test_setup_particle_points_bindings_binds_multiple_mpm_assets(monkeypatch: pytest.MonkeyPatch): """Multiple MPM assets bind as ``num_assets * num_envs`` points prims, like deformables.""" renderer, backend = _make_renderer_without_backend() @@ -375,6 +221,10 @@ def test_setup_particle_points_bindings_binds_multiple_mpm_assets(monkeypatch: p renderer._setup_particle_bindings_legacy() + assert len(backend.calls) == 1 + assert backend.calls[0]["attribute_name"] == "points" + assert backend.calls[0]["flags"] is BindingFlag.OPTIMIZE + assert [write["attribute_name"] for write in backend.writes] == ["omni:resetXformStack", "omni:xform"] # Binding order follows dict insertion order (no path sort). assert backend.calls[0]["prim_paths"] == [ "/World/envs/env_0/Media/Particles", @@ -382,55 +232,20 @@ def test_setup_particle_points_bindings_binds_multiple_mpm_assets(monkeypatch: p "/World/envs/env_0/Foam/Particles", "/World/envs/env_1/Foam/Particles", ] - assert renderer._particle_visual_offsets == [0, 5, 10, 13] - assert renderer._particle_visual_counts == [5, 5, 3, 3] - - -def test_update_particle_points_writes_world_particle_positions(monkeypatch: pytest.MonkeyPatch): - """The first MPM ``points`` update writes world-space positions through GPU ASYNC.""" - renderer, backend = _make_renderer_without_backend() - renderer._particle_points_binding = _FakePointsBinding("points") - renderer._particle_visual_offsets = [2] - renderer._particle_visual_counts = [2] - particle_q = wp.array( - [ - wp.vec3f(0.0, 0.0, 0.0), - wp.vec3f(1.0, 0.0, 0.0), - wp.vec3f(2.0, 3.0, 4.0), - wp.vec3f(5.0, 6.0, 7.0), - ], - dtype=wp.vec3f, - device="cpu", - ) + particle_q = wp.array(np.arange(48, dtype=np.float32).reshape(16, 3), dtype=wp.vec3f, device="cpu") monkeypatch.setattr(NewtonManager, "get_state", classmethod(lambda cls: SimpleNamespace(particle_q=particle_q))) - - class _FakeStream: - cuda_stream = 42 - - renderer._warp_device = SimpleNamespace(stream=_FakeStream()) - + renderer._warp_device = SimpleNamespace(stream=SimpleNamespace(cuda_stream=42)) renderer.update_geometries() - - assert len(backend.writes) == 0 - written = renderer._particle_points_binding.written - assert written is not None - assert len(written) == 1 - assert written[0].ptr == particle_q[2:4].ptr - assert written[0].numpy().tolist() == [ - [2.0, 3.0, 4.0], - [5.0, 6.0, 7.0], - ] - assert renderer._particle_points_binding.write_kwargs is not None - assert renderer._particle_points_binding.write_kwargs["data_access"] is DataAccess.ASYNC - assert renderer._particle_points_binding.write_kwargs["cuda_stream"] == 42 + for points, visual in zip(backend.bindings["points"].written, particle_visual_prims.values(), strict=True): + np.testing.assert_array_equal(points.numpy(), particle_q.numpy()[visual.offset : visual.offset + visual.count]) def test_update_geometries_writes_deformable_and_mpm_bindings(monkeypatch: pytest.MonkeyPatch): """Deformable and MPM points use GPU ASYNC writes from the first update.""" renderer, backend = _make_renderer_without_backend() renderer._deformable_points_binding = _FakePointsBinding("deformable_points") - renderer._deformable_particle_offsets = [0] - renderer._deformable_particle_counts = [2] + renderer._deformable_particle_offsets = [1] + renderer._deformable_particle_counts = [1] renderer._particle_points_binding = _FakePointsBinding("points") renderer._particle_visual_offsets = [2] renderer._particle_visual_counts = [2] @@ -456,7 +271,8 @@ class _FakeStream: deformable_written = renderer._deformable_points_binding.written assert deformable_written is not None assert len(deformable_written) == 1 - assert deformable_written[0].ptr == particle_q[0:2].ptr + assert deformable_written[0].ptr == particle_q[1:2].ptr + np.testing.assert_array_equal(deformable_written[0].numpy(), particle_q.numpy()[1:2]) assert renderer._deformable_points_binding.write_kwargs is not None assert renderer._deformable_points_binding.write_kwargs["data_access"] is DataAccess.ASYNC assert renderer._deformable_points_binding.write_kwargs["cuda_stream"] == 42 @@ -466,10 +282,10 @@ class _FakeStream: assert mpm_written is not None assert len(mpm_written) == 1 assert mpm_written[0].ptr == particle_q[2:4].ptr + np.testing.assert_array_equal(mpm_written[0].numpy(), particle_q.numpy()[2:4]) assert renderer._particle_points_binding.write_kwargs is not None assert renderer._particle_points_binding.write_kwargs["data_access"] is DataAccess.ASYNC assert renderer._particle_points_binding.write_kwargs["cuda_stream"] == 42 - assert len(backend.writes) == 0 def _install_cable_shapes(shapes: dict[str, list[int]], monkeypatch: pytest.MonkeyPatch) -> None: @@ -593,31 +409,72 @@ def _write(query, attribute, **kwargs): assert tensors[0].dtype.lanes == 3 -def test_update_transforms_writes_caller_owned_buffer(monkeypatch: pytest.MonkeyPatch): - """Object xforms fill a persistent GPU buffer and blocking ASYNC write, not map/unmap.""" - renderer, _ = _make_renderer_without_backend() - buffer = object() - renderer._object_xform_binding = _FakePointsBinding("omni:xform") - renderer._object_newton_indices = [0, 1] - renderer._object_scales = object() - renderer._object_transform_buffer = buffer - - monkeypatch.setattr(NewtonManager, "get_state", classmethod(lambda cls: SimpleNamespace(body_q=object()))) - launch_kwargs: dict = {} +@pytest.mark.parametrize("use_ovstage", [False, True]) +def test_update_transforms_consumes_sdp_matrices_once_per_publication(monkeypatch, use_ovstage): + """Both OVRTX paths bind published bodies and consume SDP's scaled, transposed matrices.""" + from isaaclab.scene_data import SceneDataFormat, SceneDataProvider - def _capture_launch(*args, **kwargs): - launch_kwargs.update(kwargs) + def reject_newton_access(*args, **kwargs): + raise AssertionError("Rigid transform transport must not read Newton state") - monkeypatch.setattr(ovrtx_renderer_module.wp, "launch", _capture_launch) + monkeypatch.setattr(NewtonManager, "get_model", reject_newton_access) + monkeypatch.setattr(NewtonManager, "get_state", reject_newton_access) + assert not hasattr(ovrtx_renderer_module, "sync_newton_transforms_kernel") + renderer, _ = _make_renderer_without_backend() + paths = ["/World/Shared", "/World/envs/env_1/Object"] + poses = np.array([[1, 2, 3, 0, 0, 0, 1], [4, 5, 6, 0, 0, 0, 1]], dtype=np.float32) + transforms = SceneDataFormat.Transform() + transforms.transforms = wp.array(poses, dtype=wp.transformf, device="cpu") + backend = SimpleNamespace(transforms=transforms, transforms_version=0, transform_count=2, transform_paths=paths) + backend.get_transforms = lambda _format: transforms + renderer._sdp = SceneDataProvider(backend) + renderer._transform_version = -1 + renderer._object_scales_by_path = {paths[0]: (2, 3, 4)} renderer._warp_device = SimpleNamespace(stream=SimpleNamespace(cuda_stream=99)) + renderer._use_ovstage = use_ovstage + renderer._current_ordinal = 5 + writes = [] + + if use_ovstage: + renderer.backend.paths = SimpleNamespace(create_path_list_from_strings=lambda actual: actual) + renderer.backend.stage = SimpleNamespace( + query_from_path_list=lambda actual: actual, + write_attribute=lambda query, attribute, **kwargs: ( + writes.append((query, attribute, kwargs)) or SimpleNamespace(wait=lambda: None) + ), + ) + monkeypatch.setattr(ovrtx_renderer_module, "xform_tensor_from_warp", lambda matrices: matrices) + renderer._setup_xform_bindings_ovstage() + assert renderer._object_xform_query == paths + writes.clear() + else: + renderer._setup_xform_bindings_legacy() + assert renderer.backend.renderer.calls[0]["prim_paths"] == paths + renderer._object_xform_binding.write = lambda matrices, **kwargs: writes.append((None, matrices, kwargs)) renderer.update_transforms() - - assert launch_kwargs["inputs"][0] is buffer - assert launch_kwargs["dim"] == 2 - assert renderer._object_xform_binding.written is buffer - assert renderer._object_xform_binding.write_kwargs["data_access"] is DataAccess.ASYNC - assert renderer._object_xform_binding.write_kwargs["cuda_stream"] == 99 + renderer.update_transforms() + assert len(writes) == 1 + matrices = writes[0][2]["tensors"] if use_ovstage else writes[0][1] + expected = np.tile(np.eye(4), (2, 1, 1)) + expected[0, :3, :3] = np.diag([2, 3, 4]) + expected[:, 3, :3] = poses[:, :3] + np.testing.assert_array_equal(matrices.numpy(), expected) + assert writes[0][2]["cuda_stream"] == 99 + if use_ovstage: + assert writes[0][2]["ordinal"] == 5 + else: + assert writes[0][2]["data_access"] is DataAccess.ASYNC + + poses[:, 0] += 10 + transforms.transforms.assign(poses) + backend.transforms_version += 1 + renderer.update_transforms() + assert len(writes) == 2 + updated = writes[1][2]["tensors"] if use_ovstage else writes[1][1] + assert updated is matrices + expected[:, 3, :3] = poses[:, :3] + np.testing.assert_array_equal(updated.numpy(), expected) def test_update_camera_writes_without_mapping(monkeypatch: pytest.MonkeyPatch): diff --git a/source/isaaclab_ov/test/test_ovrtx_renderer_contract.py b/source/isaaclab_ov/test/test_ovrtx_renderer_contract.py index 979bb13a8ab9..22339172a3a3 100644 --- a/source/isaaclab_ov/test/test_ovrtx_renderer_contract.py +++ b/source/isaaclab_ov/test/test_ovrtx_renderer_contract.py @@ -88,6 +88,7 @@ def _make_ovrtx_renderer_without_backend() -> OVRTXRenderer: @pytest.fixture(autouse=True) def _simulation_registry(monkeypatch): sim = types.SimpleNamespace(_backend_registry=[]) + sim.get_scene_data_provider = lambda: types.SimpleNamespace(backend=types.SimpleNamespace(transform_paths=[])) sim.get_or_create_backend = SimulationContext.get_or_create_backend.__get__(sim) sim.close_backend = SimulationContext.close_backend.__get__(sim) monkeypatch.setattr(SimulationContext, "_instance", sim) @@ -505,11 +506,6 @@ def fake_clone(src, *, device): assert clone_calls == [(source, "cuda:0")] -class _FakeArray: - def __init__(self, shape): - self.shape = shape - - def test_launch_extract_all_tiles_rejects_wider_output_channels(): """An output wider than the tiled input would read out of bounds, so it must raise before launching.""" renderer = _make_ovrtx_renderer_without_backend() @@ -517,55 +513,9 @@ def test_launch_extract_all_tiles_rejects_wider_output_channels(): render_data = _make_ovrtx_camera_render_data() with pytest.raises(ValueError, match="out of bounds"): - renderer._launch_extract_all_tiles(render_data, _FakeArray((8, 16, 3)), _FakeArray((2, 8, 16, 4))) - - -def test_launch_extract_all_tiles_launches_kernel_when_channels_are_compatible(monkeypatch): - """Equal or narrower output channel counts pass validation and reach the kernel launch.""" - renderer = _make_ovrtx_renderer_without_backend() - renderer._device = "cpu" - render_data = _make_ovrtx_camera_render_data() - render_data.num_cols = 2 - - launch_calls = [] - monkeypatch.setattr(wp, "launch", lambda **kwargs: launch_calls.append(kwargs)) - - tiled_buffer = _FakeArray((8, 16, 4)) - output_buffer = _FakeArray((2, 8, 16, 3)) - renderer._launch_extract_all_tiles(render_data, tiled_buffer, output_buffer) - - assert len(launch_calls) == 1 - assert launch_calls[0]["inputs"][:2] == [tiled_buffer, output_buffer] - - -def test_ovrtx_read_output_copies_no_pixel_data(): - """OVRTXRenderer.read_output copies no pixel data; with empty renderer_info it leaves info untouched.""" - renderer = _make_ovrtx_renderer_without_backend() - render_data = _make_ovrtx_camera_render_data() - camera_data = CameraData() - camera_data.info = {} - camera_data._output = {} - - result = renderer.read_output(render_data, camera_data) - assert result is None - assert render_data.warp_buffers == {} - assert camera_data.info == {} - assert camera_data.output == {} - - -def test_ovrtx_read_output_forwards_renderer_info(): - """OVRTXRenderer.read_output forwards render_data.renderer_info (e.g. semantic idToLabels) into info.""" - renderer = _make_ovrtx_renderer_without_backend() - render_data = _make_ovrtx_camera_render_data() - id_to_labels = {"2": {"class": "cartpole"}} - render_data.renderer_info = {"semantic_segmentation": {"idToLabels": id_to_labels}} - - camera_data = CameraData() - camera_data.info = {"semantic_segmentation": None} - camera_data._output = {} - - renderer.read_output(render_data, camera_data) - assert camera_data.info["semantic_segmentation"] == {"idToLabels": id_to_labels} + renderer._launch_extract_all_tiles( + render_data, types.SimpleNamespace(shape=(8, 16, 3)), types.SimpleNamespace(shape=(2, 8, 16, 4)) + ) def test_ovrtx_read_output_clears_stale_metadata_and_keeps_seeded_keys(): @@ -826,6 +776,7 @@ def test_ovrtx_cleanup_releases_only_the_given_render_data(cleanup_directly, use if cleanup_directly: render_data.cleanup() + renderer.cleanup(None) renderer.cleanup(render_data) renderer.cleanup(render_data) @@ -856,6 +807,34 @@ def test_ovrtx_cleanup_without_render_data_keeps_renderer_state(): assert renderer._render_product_paths == ["/RenderCamera_0/RenderProduct_camera"] assert renderer._initialized_scene is True +@pytest.mark.parametrize( + "camera_path", + [ + "/World/Camera", + "/World/envs/env_1/Camera", + "/World/envs/env_00/Camera", + "/World/envs/env_0", + "/World/envs/env_0/", + ], +) +def test_create_render_data_rejects_cameras_outside_source_environment(camera_path): + """Camera registration requires a source camera beneath env_0 before touching the backend.""" + from isaaclab.renderers.camera_render_spec import CameraRenderSpec + + renderer = _make_ovrtx_renderer_without_backend() + renderer.backend.renderer = MagicMock() + spec = CameraRenderSpec( + cfg=_make_camera_cfg(["depth"]), + device="cpu", + num_instances=2, + camera_prim_paths=(camera_path,), + view_count=2, + ) + + with pytest.raises(ValueError, match="/World/envs/env_0/"): + renderer.create_render_data(spec) + + assert not renderer.backend.renderer.mock_calls @pytest.mark.parametrize("use_ovstage", [False, True]) @@ -1043,7 +1022,6 @@ def close(self) -> None: renderer._particle_paths_list = "particle" renderer._cable_points_query = "cable" renderer._cable_paths_list = "cable" - renderer._object_newton_indices = object() renderer._deformable_particle_offsets = [0] renderer._deformable_particle_counts = [1] renderer._particle_visual_offsets = [0] @@ -1061,8 +1039,7 @@ def test_ovrtx_close_releases_legacy_renderer_state(): """Borrowers unbind their tensor bindings before the registry closes the native engine.""" events: list[str] = [] renderer = _make_legacy_renderer_with_backend(events) - render_data = renderer._camera_render_data[0] - + renderer.close() renderer.close() assert "destroy_renderer" not in events SimulationContext.instance().close_backend(renderer.backend) @@ -1075,28 +1052,12 @@ def test_ovrtx_close_releases_legacy_renderer_state(): "unbind:cable", "destroy_renderer", ] - assert renderer._camera_xform_binding is None - assert renderer._camera_render_data == [] - assert render_data.camera_xform_binding is None - assert render_data.renderer_info == {} - assert renderer._object_xform_binding is None - assert renderer._object_transform_buffer is None - assert renderer._deformable_points_binding is None - assert renderer._particle_points_binding is None - assert renderer._cable_points_binding is None - assert renderer._particle_workaround_applied is False - assert renderer.backend.renderer is None - assert renderer._render_product_paths == [] - assert renderer._output_id_color_buffers == {} - assert renderer._initialized_scene is False def test_ovrtx_close_releases_ovstage_renderer_state(): """Queries release before the native engine, which must detach before stage resources close.""" events: list[str] = [] renderer = _make_ovstage_renderer_with_backend(events) - render_data = renderer._camera_render_data[0] - renderer.close() assert "destroy_renderer" not in events SimulationContext.instance().close_backend(renderer.backend) @@ -1116,21 +1077,6 @@ def test_ovrtx_close_releases_ovstage_renderer_state(): "destroy_renderer", "exit_stack_close", ] - assert renderer._camera_xform_query is None - assert renderer._camera_render_data == [] - assert render_data.camera_xform_query is None - assert render_data.renderer_info == {} - assert renderer._particle_paths_list is None - assert renderer._cable_points_query is None - assert renderer._cable_paths_list is None - assert renderer._object_newton_indices is None - assert renderer.backend.renderer is None - assert renderer.backend.stage is None - assert renderer.backend.paths is None - assert renderer._render_product_paths == [] - assert renderer._output_id_color_buffers == {} - assert renderer._initialized_scene is False - assert renderer._current_ordinal == 0 events.clear() renderer.close() assert events == [] diff --git a/source/isaaclab_physx/changelog.d/sdp-transform-publication.rst b/source/isaaclab_physx/changelog.d/sdp-transform-publication.rst new file mode 100644 index 000000000000..9593f8b146cd --- /dev/null +++ b/source/isaaclab_physx/changelog.d/sdp-transform-publication.rst @@ -0,0 +1,8 @@ +Changed +^^^^^^^ + +* Published PhysX rigid transforms and their producer-owned version through SDP, and routed Isaac RTX + transform updates through one simulation-owned ``FabricBackend`` shared with Kit while preserving + native PhysX Fabric updates. Stage/device identified the resource; transforms remained binding state, + with SDP passed explicitly to updates. Fabric selection and hierarchy state moved out of core ``RenderContext``. + Kit app updates requested current SDP transforms without an additional physics ``forward()`` call. diff --git a/source/isaaclab_physx/isaaclab_physx/assets/articulation/articulation.py b/source/isaaclab_physx/isaaclab_physx/assets/articulation/articulation.py index e1950a8ab516..228fd14618dd 100644 --- a/source/isaaclab_physx/isaaclab_physx/assets/articulation/articulation.py +++ b/source/isaaclab_physx/isaaclab_physx/assets/articulation/articulation.py @@ -560,6 +560,7 @@ def write_root_link_pose_to_sim_index( self.data._reset_pose() # set into simulation self.root_view.set_root_transforms(self.data._root_link_pose_w.data.view(wp.float32), indices=sim_env_ids) + SimulationManager.invalidate_transforms(kinematics=True) def write_root_link_pose_to_sim_mask( self, @@ -657,6 +658,7 @@ def write_root_com_pose_to_sim_index( self.data._reset_pose(from_link=False) # set into simulation self.root_view.set_root_transforms(self.data._root_link_pose_w.data.view(wp.float32), indices=sim_env_ids) + SimulationManager.invalidate_transforms(kinematics=True) def write_root_com_pose_to_sim_mask( self, @@ -1057,6 +1059,7 @@ def write_joint_state_to_sim_index( self.data._reset_velocity() # set into simulation self.root_view.set_dof_positions(joint_pos_backend, indices=sim_env_ids) + SimulationManager.invalidate_transforms(kinematics=True) self.root_view.set_dof_velocities(joint_vel_backend, indices=sim_env_ids) def write_joint_state_to_sim_mask( @@ -1161,6 +1164,7 @@ def write_joint_position_to_sim_index( self.data._reset_velocity() # set into simulation self.root_view.set_dof_positions(joint_pos_backend, indices=sim_env_ids) + SimulationManager.invalidate_transforms(kinematics=True) def write_joint_position_to_sim_mask( self, diff --git a/source/isaaclab_physx/isaaclab_physx/assets/articulation/articulation_data.py b/source/isaaclab_physx/isaaclab_physx/assets/articulation/articulation_data.py index c8d776e756b4..57a9fb933294 100644 --- a/source/isaaclab_physx/isaaclab_physx/assets/articulation/articulation_data.py +++ b/source/isaaclab_physx/isaaclab_physx/assets/articulation/articulation_data.py @@ -153,6 +153,7 @@ def _ensure_fk_fresh(self) -> None: """ if self._fk_timestamp < self._sim_timestamp: self._physics_sim_view.update_articulations_kinematic() + SimulationManager._kinematics_dirty = False self._fk_timestamp = self._sim_timestamp def _reset_pose(self, from_link: bool = True) -> None: diff --git a/source/isaaclab_physx/isaaclab_physx/assets/rigid_object/rigid_object.py b/source/isaaclab_physx/isaaclab_physx/assets/rigid_object/rigid_object.py index c190b516cac4..3529f307c8fa 100644 --- a/source/isaaclab_physx/isaaclab_physx/assets/rigid_object/rigid_object.py +++ b/source/isaaclab_physx/isaaclab_physx/assets/rigid_object/rigid_object.py @@ -371,6 +371,7 @@ def write_root_link_pose_to_sim_index( self.data._reset_pose() # set into simulation self.root_view.set_transforms(self._get_root_link_pose_w_f32(), indices=sim_env_ids) + SimulationManager.invalidate_transforms() def write_root_link_pose_to_sim_mask( self, @@ -463,6 +464,7 @@ def write_root_com_pose_to_sim_index( self.data._reset_pose(from_link=False) # set into simulation self.root_view.set_transforms(self._get_root_link_pose_w_f32(), indices=sim_env_ids) + SimulationManager.invalidate_transforms() def write_root_com_pose_to_sim_mask( self, diff --git a/source/isaaclab_physx/isaaclab_physx/assets/rigid_object_collection/rigid_object_collection.py b/source/isaaclab_physx/isaaclab_physx/assets/rigid_object_collection/rigid_object_collection.py index 3f8afeb8aefa..05936dee34ea 100644 --- a/source/isaaclab_physx/isaaclab_physx/assets/rigid_object_collection/rigid_object_collection.py +++ b/source/isaaclab_physx/isaaclab_physx/assets/rigid_object_collection/rigid_object_collection.py @@ -475,6 +475,7 @@ def write_body_link_pose_to_sim_index( self.reshape_data_to_view_2d(self.data._body_link_pose_w.data, device=self.device).view(wp.float32), indices=view_ids, ) + SimulationManager.invalidate_transforms() def write_body_link_pose_to_sim_mask( self, @@ -585,6 +586,7 @@ def write_body_com_pose_to_sim_index( self.reshape_data_to_view_2d(self.data._body_link_pose_w.data, device=self.device).view(wp.float32), indices=view_ids, ) + SimulationManager.invalidate_transforms() def write_body_com_pose_to_sim_mask( self, diff --git a/source/isaaclab_physx/isaaclab_physx/physics/physx_manager.py b/source/isaaclab_physx/isaaclab_physx/physics/physx_manager.py index ea5ede85eda2..43b8188e30f7 100644 --- a/source/isaaclab_physx/isaaclab_physx/physics/physx_manager.py +++ b/source/isaaclab_physx/isaaclab_physx/physics/physx_manager.py @@ -30,6 +30,7 @@ import omni.physx import omni.timeline import omni.usd +import usdrt from pxr import Sdf, Usd, UsdPhysics, UsdUtils import isaaclab.sim as sim_utils @@ -187,7 +188,8 @@ class PhysxSceneDataBackend(SceneDataBackend): """Borrowed native resource; its lifetime belongs to the simulation registry.""" def __init__(self): - self._scene_data = SceneDataFormat.Transform() + self._transforms = SceneDataFormat.Transform() + self.transforms_version = 0 self._points_data = SceneDataFormat.Points() self.clear() @@ -197,7 +199,11 @@ def clear(self) -> None: self._rigid_body_view: omni.physics.tensors.RigidBodyView | None = None self._volume_deformable_view: omni.physics.tensors.DeformableBodyView | None = None self._surface_deformable_view: omni.physics.tensors.DeformableBodyView | None = None - self._scene_data.transforms = None + self._transforms.transforms = None + self.transforms_version += 1 + self._poses_version = self._fabric_version = -1 + self._fabric_transforms = SceneDataFormat.FabricMatrix44() + self._fabric_selection = None self._points_data.points = None self._geometry_paths: list[str] = [] self._geometry_counts: list[int] = [] @@ -358,12 +364,21 @@ def geometry_counts(self) -> list[int]: self._discover_deformable_geometry() return self._geometry_counts + @property + def native_transform_formats(self) -> tuple[Any, ...]: + """Native pose formats available without extracting or converting body state.""" + if PhysxManager._fabric is not None: + return SceneDataFormat.Transform, SceneDataFormat.FabricMatrix44 + return (SceneDataFormat.Transform,) + @property def transforms(self) -> SceneDataFormat.Transform: - """Return the current PhysX rigid body transforms as :class:`SceneDataFormat.Transform`.""" - if view := self.get_rigid_body_view(): - self._scene_data.transforms = view.get_transforms().view(wp.transformf) - return self._scene_data + """Publish native rigid-body poses [m, xyzw].""" + PhysxManager.pre_render() + if self._poses_version != self.transforms_version and (view := self.get_rigid_body_view()): + self._transforms.transforms = view.get_transforms().view(wp.transformf) + self._poses_version = self.transforms_version + return self._transforms @property def transform_count(self) -> int: @@ -379,6 +394,26 @@ def transform_paths(self) -> list[str]: return list(view.prim_paths) return [] + def get_transforms(self, output_format: Any) -> SceneDataFormat.Transform | SceneDataFormat.FabricMatrix44: + """Publish the requested native representation, refreshing only that representation.""" + if output_format is not SceneDataFormat.FabricMatrix44 or PhysxManager._fabric is None: + return self.transforms + PhysxManager.pre_render() + if self._fabric_version != self.transforms_version: + PhysxManager._fabric.force_update(0.0, 0.0) + if self._fabric_selection is None: + stage = usdrt.Usd.Stage.Attach(PhysxManager._stage_id) + self._fabric_selection = stage.SelectPrims( + require_applied_schemas=["PhysicsRigidBodyAPI"], + require_attrs=[(usdrt.Sdf.ValueTypeNames.Matrix4d, "omni:fabric:worldMatrix", usdrt.Usd.Access.Read)], + device=str(PhysicsManager._device), + ) + if self._fabric_selection.PrepareForReuse() or self._fabric_transforms.matrices is None: + self._fabric_transforms.matrices = wp.fabricarray(self._fabric_selection, "omni:fabric:worldMatrix") + self.transforms_version += 1 + self._fabric_version = self.transforms_version + return self._fabric_transforms + class PhysxManager(PhysicsManager): """Manages PhysX physics simulation lifecycle. @@ -395,6 +430,7 @@ class PhysxManager(PhysicsManager): _timeline: ClassVar[omni.timeline.ITimeline] = omni.timeline.get_timeline_interface() _event_bus: ClassVar[carb.eventdispatcher.IEventDispatcher] = carb.eventdispatcher.get_eventdispatcher() _scene_data_backend: ClassVar[PhysxSceneDataBackend | None] = None + _kinematics_dirty: ClassVar[bool] = False backend: ClassVar[PhysxBackend | None] = None """Borrowed native resource, available after physics warmup and released on stop.""" @@ -404,7 +440,6 @@ class PhysxManager(PhysicsManager): _stage_id: ClassVar[int] = -1 _subscriptions: ClassVar[dict[str, Any]] = {} _fabric: ClassVar[Any] = None - _update_fabric: ClassVar[Callable[[float, float], None] | None] = None _anim_recorder: ClassVar[AnimationRecorder | None] = None _callback_exception: ClassVar[Exception | None] = None @@ -447,6 +482,7 @@ def initialize(cls, sim_context: SimulationContext) -> None: cls._load_fabric() cls._anim_recorder = AnimationRecorder(sim_context) cls._scene_data_backend = PhysxSceneDataBackend() + cls._kinematics_dirty = False # force update cycle to apply dt sim = PhysicsManager._sim @@ -499,16 +535,32 @@ def reset(cls, soft: bool = False) -> None: if cls.backend is not None: cls.backend.simulation_view._backend.initialize_kinematic_bodies() + cls.invalidate_transforms(kinematics=True) cls.raise_callback_exception_if_any() @classmethod def forward(cls) -> None: """Update articulation kinematics and fabric for rendering.""" sim = PhysicsManager._sim - if cls._fabric is not None and cls._update_fabric is not None: - if cls.backend is not None and sim is not None and sim.is_playing(): - cls.backend.simulation_view.update_articulations_kinematic() - cls._update_fabric(0.0, 0.0) + if cls.backend is not None and sim is not None and sim.is_playing(): + cls.backend.simulation_view.update_articulations_kinematic() + cls._kinematics_dirty = False + cls.invalidate_transforms() + if cls._fabric is not None: + cls._scene_data_backend.get_transforms(SceneDataFormat.FabricMatrix44) + + @classmethod + def invalidate_transforms(cls, *, kinematics: bool = False) -> None: + """Invalidate both native pose representations after writes; defer FK when needed.""" + cls._kinematics_dirty |= kinematics + cls._scene_data_backend.transforms_version += 1 + + @classmethod + def pre_render(cls) -> None: + """Complete pending pose writes before SDP publishes articulation transforms.""" + if cls._kinematics_dirty and cls.backend is not None: + cls.backend.simulation_view.update_articulations_kinematic() + cls._kinematics_dirty = False @classmethod def get_scene_data_backend(cls) -> SceneDataBackend: @@ -535,6 +587,8 @@ def step(cls) -> None: physx_sim = omni.physx.get_physx_simulation_interface() physx_sim.simulate(sim.cfg.dt, 0.0) physx_sim.fetch_results() + cls._kinematics_dirty = False + cls.invalidate_transforms() device = PhysicsManager._device if "cuda" in device: torch.cuda.set_device(device) @@ -589,8 +643,9 @@ def _sync_fabric_after_resume(cls) -> None: cls._re_sync_fabric() if cls.backend is not None: cls.backend.simulation_view.update_articulations_kinematic() - if cls._update_fabric is not None: - cls._update_fabric(0.0, 0.0) + cls._kinematics_dirty = False + if cls._fabric is not None: + cls._fabric.force_update(0.0, 0.0) @classmethod def close(cls) -> None: @@ -610,11 +665,11 @@ def close(cls) -> None: cls._event_bus.dispatch_event(IsaacEvents.PRIM_DELETION.value, payload={"prim_path": "/"}) cls._fabric = None - cls._update_fabric = None cls._anim_recorder = None cls._warmup_needed = True cls._assets_loaded = True cls._callback_exception = None + cls._kinematics_dirty = False super().close() @@ -879,12 +934,10 @@ def _load_fabric(cls) -> None: from omni.physxfabric import get_physx_fabric_interface cls._fabric = get_physx_fabric_interface() - cls._update_fabric = getattr(cls._fabric, "force_update", cls._fabric.update) else: if ext_mgr.is_extension_enabled("omni.physx.fabric"): ext_mgr.set_extension_enabled_immediate("omni.physx.fabric", False) cls._fabric = None - cls._update_fabric = None # disable usd sync when fabric is enabled (via SettingsManager) for key in [ diff --git a/source/isaaclab_physx/isaaclab_physx/renderers/fabric.py b/source/isaaclab_physx/isaaclab_physx/renderers/fabric.py new file mode 100644 index 000000000000..5d8f1e6b8231 --- /dev/null +++ b/source/isaaclab_physx/isaaclab_physx/renderers/fabric.py @@ -0,0 +1,126 @@ +# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md). +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause + +"""Simulation-owned Fabric resource shared by Kit and Isaac RTX.""" + +from __future__ import annotations + +from dataclasses import field + +import warp as wp + +import usdrt +import usdrt.hierarchy +from pxr import Usd, UsdUtils + +from isaaclab.scene_data import SceneDataFormat, SceneDataProvider +from isaaclab.sim import BackendCfg +from isaaclab.utils import configclass + + +@wp.kernel(enable_backward=False) +def _capture_scales( + matrices: wp.fabricarray(dtype=wp.mat44d), + indices: wp.fabricarray(dtype=wp.int32), + scales: wp.array(dtype=wp.vec3f), +): + i = wp.tid() + matrix = wp.mat44f(matrices[i]) + scales[indices[i]] = wp.vec3f( + wp.length(wp.vec3f(matrix[0, 0], matrix[0, 1], matrix[0, 2])), + wp.length(wp.vec3f(matrix[1, 0], matrix[1, 1], matrix[1, 2])), + wp.length(wp.vec3f(matrix[2, 0], matrix[2, 1], matrix[2, 2])), + ) + + +class FabricBackend: + """Own one stage's native Fabric handles and shared transform bindings. + + Consumers supply the simulation's SDP explicitly. Its producer layout stays fixed for the + binding's lifetime; it is not part of the native stage/device identity. + """ + + def __init__(self, cfg: FabricBackendCfg): + self.device = cfg.device + self.stage = usdrt.Usd.Stage.Attach(UsdUtils.StageCache.Get().GetId(cfg.stage).ToLongInt()) + self.hierarchy = usdrt.hierarchy.IFabricHierarchy().get_fabric_hierarchy( + self.stage.GetFabricId(), self.stage.GetStageIdAsStageId() + ) + self.transforms: SceneDataFormat.FabricMatrix44 | None = None + self._selection = self._write_selection = None + self._mapping = self._scales = None + self._version = -1 + + def bind_transforms(self, provider: SceneDataProvider) -> None: + """Bind the initialized simulation's rigid destinations once; native Fabric needs no conversion binding.""" + if self.transforms is not None: + return + if SceneDataFormat.FabricMatrix44 in provider.backend.native_transform_formats: + self.transforms = SceneDataFormat.FabricMatrix44() + return + + stage = self.stage + stage.SynchronizeToFabric() + self.hierarchy.update_world_xforms() + for index, path in enumerate(provider.backend.transform_paths): + prim = stage.GetPrimAtPath(path) + if not prim or not prim.HasAPI("PhysicsRigidBodyAPI"): + continue + prim.CreateAttribute("isaaclab:transformIndex", usdrt.Sdf.ValueTypeNames.Int, custom=True).Set(index) + # Physics publishes absolute body poses; only visual descendants inherit them. + self.hierarchy.set_reset_xform_stack(prim.GetPath().fabricPath, True) + attrs = [ + (usdrt.Sdf.ValueTypeNames.Matrix4d, "omni:fabric:worldMatrix", usdrt.Usd.Access.Read), + (usdrt.Sdf.ValueTypeNames.Int, "isaaclab:transformIndex", usdrt.Usd.Access.Read), + (usdrt.Sdf.ValueTypeNames.Matrix4d, "omni:fabric:localMatrix", usdrt.Usd.Access.Read), + ] + self._selection = stage.SelectPrims(require_attrs=attrs, device=self.device) + self._write_selection = stage.SelectPrims( + require_attrs=[*attrs[:-1], (*attrs[-1][:2], usdrt.Usd.Access.ReadWrite)], device=self.device + ) + self._scales = wp.empty(provider.transform_count, dtype=wp.vec3f, device=self.device) + self.transforms = SceneDataFormat.FabricMatrix44() + + def update_transforms(self, provider: SceneDataProvider) -> None: + """Request SDP transforms and propagate converted body matrices to visual descendants.""" + self.bind_transforms(provider) + changed = self._selection is not None and self._selection.PrepareForReuse() + if self._selection is not None and (changed or self.transforms.matrices is None): + self._write_selection.PrepareForReuse() + self._mapping = wp.fabricarray(self._selection, "isaaclab:transformIndex") + if self.transforms.matrices is None: + wp.launch( + _capture_scales, + dim=len(self._mapping), + inputs=[wp.fabricarray(self._selection, "omni:fabric:worldMatrix"), self._mapping], + outputs=[self._scales], + device=self._scales.device, + ) + self.transforms = SceneDataFormat.FabricMatrix44() + self.transforms.matrices = wp.fabricarray(self._write_selection, "omni:fabric:localMatrix") + provider.get_transforms(self.transforms, self._mapping, scales=self._scales) + version = provider.backend.transforms_version + if self._selection is not None and (changed or self._version != version): + self._write_selection.PrepareForReuse() + device = self._scales.device + wp.synchronize_stream(device) + if not self.hierarchy.update_world_xforms_gpu(not changed and self._version != -1): + raise RuntimeError("Fabric GPU transform hierarchy update failed.") + wp.synchronize_device(device) + self._version = version + + def close(self) -> None: + """Release stage-bound selections and borrowed SDP buffers.""" + self.transforms = self._selection = self._write_selection = None + self._mapping = self._scales = self.hierarchy = self.stage = None + + +@configclass +class FabricBackendCfg(BackendCfg): + """Native Fabric identity; the stage is borrowed from the active simulation.""" + + class_type: type = FabricBackend + stage: Usd.Stage = field(kw_only=True, metadata={"copy": False}) + device: str = field(kw_only=True) diff --git a/source/isaaclab_physx/isaaclab_physx/renderers/isaac_rtx_renderer.py b/source/isaaclab_physx/isaaclab_physx/renderers/isaac_rtx_renderer.py index 1e4d8eee058f..011e0b55f093 100644 --- a/source/isaaclab_physx/isaaclab_physx/renderers/isaac_rtx_renderer.py +++ b/source/isaaclab_physx/isaaclab_physx/renderers/isaac_rtx_renderer.py @@ -23,6 +23,7 @@ from isaaclab.app.settings_manager import get_settings_manager from isaaclab.renderers import BaseRenderer, RenderBufferKind, RenderBufferSpec from isaaclab.renderers.camera_render_spec import CameraRenderSpec +from isaaclab.sim import SimulationContext from isaaclab.sim.utils import enable_extension from isaaclab.utils.version import get_isaac_sim_version from isaaclab.utils.warp.kernels import reshape_tiled_image @@ -195,6 +196,12 @@ def __init__(self, cfg: IsaacRtxRendererCfg): ensure_rtx_hydra_engine_attached() # ``/isaaclab/render/rtx_sensors`` is owned by ``Camera.__init__`` (must be set pre-``sim.reset()``). + def initialize(self) -> None: + """Bind shared Fabric destinations after scene creation.""" + sim = SimulationContext.instance() + self._fabric = sim.get_or_create_backend(sim.fabric_cfg) + self._fabric.bind_transforms(sim.get_scene_data_provider()) + @property def visual_material_writer(self): """Write material channels directly through Fabric.""" @@ -571,9 +578,8 @@ def set_outputs(self, render_data: IsaacRtxRenderData, output_data: dict[str, Pr ) def update_transforms(self) -> None: - """No-op for Isaac RTX - uses USD scene directly. - See :meth:`~isaaclab.renderers.base_renderer.BaseRenderer.update_transforms`.""" - pass + """Update shared Fabric transforms and propagate the visual hierarchy.""" + self._fabric.update_transforms(SimulationContext.instance().get_scene_data_provider()) def update_geometries(self) -> None: """No-op for Isaac RTX - uses USD scene directly. diff --git a/source/isaaclab_physx/isaaclab_physx/renderers/isaac_rtx_renderer_utils.py b/source/isaaclab_physx/isaaclab_physx/renderers/isaac_rtx_renderer_utils.py index f0d893dddb0d..67d487e4ae56 100644 --- a/source/isaaclab_physx/isaaclab_physx/renderers/isaac_rtx_renderer_utils.py +++ b/source/isaaclab_physx/isaaclab_physx/renderers/isaac_rtx_renderer_utils.py @@ -219,32 +219,21 @@ def ensure_isaac_rtx_render_update(force: bool = False) -> None: if sim is None: return - render_generation = getattr(sim, "render_generation", getattr(sim, "_render_generation", 0)) - key = (id(sim), sim._physics_step_count, render_generation) + key = (id(sim), sim.get_physics_step_count(), sim.render_generation) if _last_render_update_key == key: return # Already pumped this step (by another camera or a visualizer) - # If a visualizer already pumps the Kit app loop, mark as done and skip. - # However, on the very first call for a new SimulationContext, the visualizer - # has not had a chance to pump yet (sim.render() was never called), so we - # must perform the initial app.update() ourselves to populate annotator buffers. + # Prime annotators once; afterward the Kit visualizer owns its app updates. first_call_for_sim = _last_render_update_key[0] != id(sim) if not first_call_for_sim and any(viz.pumps_app_update() for viz in sim.visualizers): _last_render_update_key = key return - # Pump when continuous rendering is active (GUI/RTX sensors/visualizers/XR). ``is_rendering`` - # excludes headless offscreen rendering so the per-step loop does not pump between frames. - # Offscreen frames are produced on demand: the ``--video`` / ``rgb_array`` path calls this with - # ``force=True`` (see :func:`pump_kit_app_for_headless_video_render_if_needed`) to pump exactly - # when a frame is requested, without making every step pump. + # Headless offscreen capture requests a frame explicitly with force=True. if not force and not sim.is_rendering: return - # Sync physics results → Fabric so RTX sees updated positions. - # physics_manager.step() only runs simulate()/fetch_results() and does NOT - # call _update_fabric(), so without this the render would lag one frame behind. - sim.physics_manager.forward() + sim.get_or_create_backend(sim.fabric_cfg).update_transforms(sim.get_scene_data_provider()) import omni.kit.app diff --git a/source/isaaclab_physx/test/renderers/test_isaac_rtx_renderer_contract.py b/source/isaaclab_physx/test/renderers/test_isaac_rtx_renderer_contract.py index 9703728b5121..34c05e227800 100644 --- a/source/isaaclab_physx/test/renderers/test_isaac_rtx_renderer_contract.py +++ b/source/isaaclab_physx/test/renderers/test_isaac_rtx_renderer_contract.py @@ -19,10 +19,13 @@ from packaging import version from isaaclab.renderers import RenderBufferKind, RenderBufferSpec +from isaaclab.sim import SimulationContext from isaaclab.utils.renderers import ISAAC_RTX_SHOW_ALL_PARTITIONS_BY_DEFAULT_SETTING def _install_omni_stubs(monkeypatch): + sim = SimpleNamespace(stage=object(), device="cpu", get_scene_data_provider=MagicMock()) + monkeypatch.setattr(SimulationContext, "instance", lambda: sim) omni_module = sys.modules.get("omni", types.ModuleType("omni")) replicator_module = types.ModuleType("omni.replicator") replicator_core_module = types.ModuleType("omni.replicator.core") @@ -264,24 +267,6 @@ def _create_attribute(name, value_type): assert global_setting_calls == [] -def test_render_product_uuid_name_format_is_sdf_safe(): - """``rp_{uuid4().hex}`` matches the create_render_data naming contract and is SDF-safe.""" - import uuid - - from pxr import Sdf - - names = [f"rp_{uuid.uuid4().hex}" for _ in range(64)] - assert len(set(names)) == len(names) - for name in names: - assert name.startswith("rp_") - hex_part = name.removeprefix("rp_") - assert len(hex_part) == 32 - int(hex_part, 16) # raises if not hex - assert "-" not in name - assert Sdf.Path.IsValidIdentifier(name) - assert Sdf.Path.IsValidPathString(f"/Render/{name}") - - @pytest.mark.parametrize( ("has_gui", "expected_disable_color_render"), [ diff --git a/source/isaaclab_physx/test/renderers/test_isaac_rtx_renderer_utils.py b/source/isaaclab_physx/test/renderers/test_isaac_rtx_renderer_utils.py index a36a888a4f35..4c7878231184 100644 --- a/source/isaaclab_physx/test/renderers/test_isaac_rtx_renderer_utils.py +++ b/source/isaaclab_physx/test/renderers/test_isaac_rtx_renderer_utils.py @@ -3,11 +3,7 @@ # # SPDX-License-Identifier: BSD-3-Clause -"""Unit tests for RTX streaming wait helpers. - -Covers callback state updates, subscription behavior, and timeout-aware wait -logic in :mod:`isaaclab_physx.renderers.isaac_rtx_renderer_utils`. -""" +"""Unit tests for RTX streaming waits and render-update cadence.""" from __future__ import annotations @@ -32,7 +28,6 @@ # test-specific timeout overrides for _STREAMING_WAIT_TIMEOUT_S STREAMING_TIMEOUT_S = 0.1 -STREAMING_TIMEOUT_SHORT_S = 0.01 # simulated per-update sleep to advance wall-clock time inside the wait loop MOCK_UPDATE_SLEEP_S = 0.02 @@ -80,54 +75,29 @@ def mock_omni_kit_app(): yield mock_module -# --------------------------------------------------------------------------- -# _get_stage_streaming_busy -# --------------------------------------------------------------------------- - - -class TestGetStageStreamingBusy: - """Synchronous streaming status query delegates to UsdContext.""" - - def test_returns_true_when_busy(self, mock_omni_usd): - mock_ctx = MagicMock() - mock_ctx.get_stage_streaming_status.return_value = True - mock_omni_usd.get_context.return_value = mock_ctx - assert rtx_utils._get_stage_streaming_busy() is True - - def test_returns_false_when_idle(self, mock_omni_usd): - mock_ctx = MagicMock() - mock_ctx.get_stage_streaming_status.return_value = False - mock_omni_usd.get_context.return_value = mock_ctx - assert rtx_utils._get_stage_streaming_busy() is False - - def test_returns_false_when_no_context(self, mock_omni_usd): - mock_omni_usd.get_context.return_value = None - assert rtx_utils._get_stage_streaming_busy() is False - - # --------------------------------------------------------------------------- # _wait_for_streaming_complete # --------------------------------------------------------------------------- class TestWaitForStreamingComplete: - """Blocking wait pumps app.update() while busy and respects timeout. + """Blocking wait pumps app.update() while busy and respects timeout.""" - These tests patch ``_get_stage_streaming_busy`` at the module level so - they don't depend on ``omni.usd`` being importable. - """ - - def test_returns_immediately_when_not_busy(self, mock_omni_kit_app): - """Skips loop and issues only the final update when idle.""" + @pytest.mark.parametrize("has_context", [False, True]) + def test_returns_immediately_when_not_busy(self, mock_omni_usd, mock_omni_kit_app, has_context): + """Idle and absent stages need only the final update.""" mock_app = MagicMock() mock_omni_kit_app.get_app.return_value = mock_app + context = mock_omni_usd.get_context.return_value + context.get_stage_streaming_status.return_value = False + if not has_context: + mock_omni_usd.get_context.return_value = None - with patch.object(rtx_utils, "_get_stage_streaming_busy", return_value=False): - rtx_utils._wait_for_streaming_complete() + rtx_utils._wait_for_streaming_complete() mock_app.update.assert_called_once() - def test_pumps_updates_until_idle(self, mock_omni_kit_app): + def test_pumps_updates_until_idle(self, mock_omni_usd, mock_omni_kit_app): """Pumps updates until streaming reports idle.""" mock_app = MagicMock() mock_omni_kit_app.get_app.return_value = mock_app @@ -141,13 +111,13 @@ def _count_update(): loop_calls += 1 mock_app.update.side_effect = _count_update + mock_omni_usd.get_context.return_value.get_stage_streaming_status.side_effect = _streaming_status - with patch.object(rtx_utils, "_get_stage_streaming_busy", side_effect=_streaming_status): - rtx_utils._wait_for_streaming_complete() + rtx_utils._wait_for_streaming_complete() assert mock_app.update.call_count == MOCK_ITERATIONS_BEFORE_IDLE + 1 - def test_respects_timeout(self, monkeypatch, mock_omni_kit_app): + def test_respects_timeout(self, monkeypatch, mock_omni_kit_app, caplog): """Exits wait loop on timeout if busy never clears.""" monkeypatch.setattr(rtx_utils, "_STREAMING_WAIT_TIMEOUT_S", STREAMING_TIMEOUT_S) mock_app = MagicMock() @@ -158,48 +128,7 @@ def test_respects_timeout(self, monkeypatch, mock_omni_kit_app): rtx_utils._wait_for_streaming_complete() assert mock_app.update.call_count > 0 - - def test_timeout_logs_warning(self, monkeypatch, mock_omni_kit_app): - """Logs warning when timeout is reached while still busy.""" - monkeypatch.setattr(rtx_utils, "_STREAMING_WAIT_TIMEOUT_S", STREAMING_TIMEOUT_SHORT_S) - mock_app = MagicMock() - mock_omni_kit_app.get_app.return_value = mock_app - mock_logger = MagicMock() - - with ( - patch.object(rtx_utils, "_get_stage_streaming_busy", return_value=True), - patch.object(rtx_utils, "logger", mock_logger), - ): - rtx_utils._wait_for_streaming_complete() - - mock_logger.warning.assert_called_once() - assert "RTX streaming did not complete within" in mock_logger.warning.call_args[0][0] - - def test_logs_info_on_non_trivial_completion(self, mock_omni_kit_app): - """Logs completion info when streaming finishes after delay.""" - mock_app = MagicMock() - mock_omni_kit_app.get_app.return_value = mock_app - mock_logger = MagicMock() - call_count = 0 - - def _streaming_status(): - return call_count < 1 - - def _become_idle_after_delay(): - nonlocal call_count - time.sleep(MOCK_UPDATE_SLEEP_S) - call_count += 1 - - mock_app.update.side_effect = _become_idle_after_delay - - with ( - patch.object(rtx_utils, "_get_stage_streaming_busy", side_effect=_streaming_status), - patch.object(rtx_utils, "logger", mock_logger), - ): - rtx_utils._wait_for_streaming_complete() - - mock_logger.info.assert_called_once() - assert "RTX streaming completed in" in mock_logger.info.call_args[0][0] + assert "RTX streaming did not complete within" in caplog.text # --------------------------------------------------------------------------- @@ -219,6 +148,7 @@ def mock_sim(self): """A minimal mock of :class:`SimulationContext`.""" sim = MagicMock() sim._physics_step_count = 0 + sim.get_physics_step_count.side_effect = lambda: sim._physics_step_count sim._render_generation = 0 sim.render_generation = 0 sim.is_rendering = True @@ -239,41 +169,19 @@ def mock_sim_context(self, monkeypatch): monkeypatch.setattr(rtx_utils, "sim_utils", types.SimpleNamespace(SimulationContext=sim_context)) return sim_context - def test_first_call_with_visualizer_still_pumps( - self, mock_sim, mock_sim_context, pumping_visualizer, mock_omni_kit_app - ): - """Regression: first call for a new sim must pump even with a visualizer. - - Without the fix (commit 2e8ace7), a visualizer returning - ``pumps_app_update() == True`` caused the function to skip - ``app.update()`` on the very first call. The visualizer had not - pumped yet (``sim.render()`` was never called), so annotator - buffers were never populated and cameras hung waiting for data. - """ - mock_sim.visualizers = [pumping_visualizer] - mock_app = MagicMock() - mock_omni_kit_app.get_app.return_value = mock_app - mock_sim_context.instance.return_value = mock_sim - - with ( - patch.object(rtx_utils, "_get_stage_streaming_busy", return_value=False), - ): - rtx_utils.ensure_isaac_rtx_render_update() - - mock_app.update.assert_called_once() - - def test_second_call_with_visualizer_skips_pump( + def test_visualizer_pumps_only_after_initial_render_update( self, mock_sim, mock_sim_context, pumping_visualizer, mock_omni_kit_app ): - """After the first call, a visualizer that pumps causes the skip.""" + """Publish the first frame before yielding app updates to an active visualizer.""" mock_sim.visualizers = [pumping_visualizer] mock_app = MagicMock() mock_omni_kit_app.get_app.return_value = mock_app mock_sim_context.instance.return_value = mock_sim + provider = mock_sim.get_scene_data_provider.return_value + update_transforms = mock_sim.get_or_create_backend.return_value.update_transforms + mock_app.update.side_effect = lambda: update_transforms.assert_called_once_with(provider) - with ( - patch.object(rtx_utils, "_get_stage_streaming_busy", return_value=False), - ): + with patch.object(rtx_utils, "_get_stage_streaming_busy", return_value=False): rtx_utils.ensure_isaac_rtx_render_update() mock_app.update.assert_called_once() mock_app.update.reset_mock() @@ -282,6 +190,9 @@ def test_second_call_with_visualizer_skips_pump( rtx_utils.ensure_isaac_rtx_render_update() mock_app.update.assert_not_called() + mock_sim.get_or_create_backend.assert_called_once_with(mock_sim.fabric_cfg) + update_transforms.assert_called_once_with(provider) + mock_sim.physics_manager.forward.assert_not_called() def test_no_sim_is_noop(self, mock_sim_context, mock_omni_kit_app): """No-op when SimulationContext.instance() returns None.""" @@ -310,13 +221,17 @@ def test_dedup_same_step(self, mock_sim, mock_sim_context, mock_omni_kit_app): mock_app.update.assert_not_called() - def test_not_rendering_skips(self, mock_sim, mock_sim_context, mock_omni_kit_app): - """No ``app.update()`` when rendering is disabled.""" + @pytest.mark.parametrize("force", [False, True]) + def test_not_rendering_pumps_only_when_forced(self, mock_sim, mock_sim_context, mock_omni_kit_app, force): + """Offscreen capture publishes through SDP only when a frame is requested.""" mock_sim.is_rendering = False mock_app = MagicMock() mock_omni_kit_app.get_app.return_value = mock_app mock_sim_context.instance.return_value = mock_sim - rtx_utils.ensure_isaac_rtx_render_update() + with patch.object(rtx_utils, "_get_stage_streaming_busy", return_value=False): + rtx_utils.ensure_isaac_rtx_render_update(force=force) - mock_app.update.assert_not_called() + assert mock_app.update.call_count == int(force) + assert mock_sim.get_or_create_backend.return_value.update_transforms.call_count == int(force) + mock_sim.physics_manager.forward.assert_not_called() diff --git a/source/isaaclab_physx/test/sim/test_physx_scene_data_backend.py b/source/isaaclab_physx/test/sim/test_physx_scene_data_backend.py index 467f94461fad..6f0874866d47 100644 --- a/source/isaaclab_physx/test/sim/test_physx_scene_data_backend.py +++ b/source/isaaclab_physx/test/sim/test_physx_scene_data_backend.py @@ -4,13 +4,107 @@ # SPDX-License-Identifier: BSD-3-Clause from types import SimpleNamespace +from unittest.mock import Mock +import numpy as np import pytest +import warp as wp pytest.importorskip("pxr") pytest.importorskip("omni.physics.tensors") +@pytest.mark.parametrize("operation", ["step", "forward"]) +def test_pose_publication_refreshes_after_physics_but_reuses_clean_reads(monkeypatch, operation): + """SDP borrows native poses once per dirty generation and completes pending joint writes.""" + from isaaclab_physx.physics import physx_manager + + from isaaclab.physics import PhysicsManager + from isaaclab.scene_data import SceneDataFormat, SceneDataProvider + + manager = physx_manager.PhysxManager + fabric = Mock() + monkeypatch.setattr(manager, "_fabric", fabric) + backend = physx_manager.PhysxSceneDataBackend() + transforms = wp.zeros(1, dtype=wp.transformf, device="cpu") + view = Mock(count=1, get_transforms=Mock(return_value=transforms)) + backend._rigid_body_view = view + monkeypatch.setattr(backend, "get_rigid_body_view", Mock(wraps=backend.get_rigid_body_view)) + sim_view = Mock() + monkeypatch.setattr(manager, "backend", SimpleNamespace(simulation_view=sim_view)) + monkeypatch.setattr(manager, "_scene_data_backend", backend) + monkeypatch.setattr(manager, "_kinematics_dirty", False) + monkeypatch.setattr(manager, "_anim_recorder", None) + monkeypatch.setattr( + PhysicsManager, "_sim", SimpleNamespace(stage=object(), cfg=SimpleNamespace(dt=0.01), is_playing=lambda: True) + ) + monkeypatch.setattr(PhysicsManager, "_device", "cpu") + monkeypatch.setattr(physx_manager.omni.physx, "get_physx_simulation_interface", Mock(return_value=Mock())) + provider = SceneDataProvider(backend) + fabric_matrices = wp.zeros(1, dtype=wp.mat44d, device="cpu") + backend._fabric_selection = SimpleNamespace( + PrepareForReuse=Mock(return_value=False), + __fabric_arrays_interface__={ + "version": 1, + "device": "cpu", + "attribs": { + "omni:fabric:worldMatrix": { + "type": (True, "f8", 16, 0, "matrix"), + "access": 1, + "pointers": [fabric_matrices.ptr], + "counts": [1], + } + }, + }, + ) + assert provider.get_transforms(SceneDataFormat.FabricMatrix44()) + version = backend.transforms_version + assert provider.get_transforms(SceneDataFormat.FabricMatrix44()) + fabric.force_update.assert_called_once_with(0.0, 0.0) + backend.get_rigid_body_view.assert_not_called() + view.get_transforms.assert_not_called() + assert backend.transforms_version == version + native = SceneDataFormat.Transform() + assert provider.get_transforms(native) + assert native.transforms.ptr == transforms.ptr + output = SceneDataFormat.Matrix44() + assert provider.get_transforms(output) + matrices = output.matrices + view.get_transforms.assert_called_once_with() + + transforms.fill_(wp.transformf(wp.vec3f(1, 2, 3), wp.quat_identity())) + getattr(manager, operation)() + assert backend.transforms_version > version + version = backend.transforms_version + manager.pre_render() + manager.pre_render() + assert sim_view.update_articulations_kinematic.call_count == int(operation == "forward") + assert provider.get_transforms(output) + assert output.matrices is matrices + np.testing.assert_array_equal(matrices.numpy()[0, :3, 3], [1, 2, 3]) + assert view.get_transforms.call_count == 2 + provider.get_transforms(SceneDataFormat.FabricMatrix44()) + provider.get_transforms(SceneDataFormat.FabricMatrix44()) + assert fabric.force_update.call_count == 2 + + transforms.fill_(wp.transformf(wp.vec3f(4, 5, 6), wp.quat_identity())) + manager.invalidate_transforms(kinematics=True) + assert backend.transforms_version > version + version = backend.transforms_version + provider.get_transforms(SceneDataFormat.FabricMatrix44()) + provider.get_transforms(SceneDataFormat.FabricMatrix44()) + assert sim_view.update_articulations_kinematic.call_count == 1 + int(operation == "forward") + assert fabric.force_update.call_count == 3 + assert backend.transforms_version == version + provider.get_transforms(native) + assert provider.get_transforms(output) + np.testing.assert_array_equal(output.matrices.numpy()[0, :3, 3], [4, 5, 6]) + assert view.get_transforms.call_count == 3 + assert backend.transforms_version == version + backend.clear() + assert backend.transforms_version > version + + @pytest.mark.parametrize("joint_has_rigid_body_api", [False, True]) def test_rigid_body_view_uses_exact_path_for_joint_name_collision(monkeypatch, joint_has_rigid_body_api): """Joint names must keep same-named rigid bodies out of wildcard views.""" diff --git a/source/isaaclab_physx/test/sim/test_views_xform_prim_fabric.py b/source/isaaclab_physx/test/sim/test_views_xform_prim_fabric.py index e719dea8462f..2eb6583faac8 100644 --- a/source/isaaclab_physx/test/sim/test_views_xform_prim_fabric.py +++ b/source/isaaclab_physx/test/sim/test_views_xform_prim_fabric.py @@ -26,11 +26,13 @@ import warp as wp # noqa: E402 from frame_view_contract_utils import * # noqa: F401, F403, E402 from frame_view_contract_utils import CHILD_OFFSET, ViewBundle # noqa: E402, F401 +from isaaclab_physx.physics import PhysxCfg # noqa: E402 from isaaclab_physx.sim.views import FabricFrameView as FrameView # noqa: E402 -from pxr import Gf, UsdGeom # noqa: E402 +from pxr import Gf, UsdGeom, UsdPhysics # noqa: E402 import isaaclab.sim as sim_utils # noqa: E402 +from isaaclab.scene_data import SceneDataFormat, SceneDataProvider # noqa: E402 pytestmark = pytest.mark.isaacsim_ci PARENT_POS = (0.0, 0.0, 1.0) @@ -135,6 +137,50 @@ def factory(num_envs: int, device: str) -> ViewBundle: # ------------------------------------------------------------------ +@pytest.mark.parametrize("device", [device for device in test_devices() if device.startswith("cuda")]) +def test_sdp_native_gpu_fabric_binding_preserves_live_physx_pose(device, request): + """Native GPU binding preserves live poses and publishes same-step writes without forward().""" + _skip_if_unavailable(device) + prim = UsdGeom.Cube.Define(sim_utils.get_current_stage(), "/World/Cube").GetPrim() + UsdPhysics.RigidBodyAPI.Apply(prim) + UsdPhysics.CollisionAPI.Apply(prim) + sim = sim_utils.SimulationContext( + sim_utils.SimulationCfg(physics=PhysxCfg(), device=device, gravity=(0, 0, 0), use_fabric=True) + ) + sim.set_setting("/physics/fabricUpdateTransformations", True) + sim.reset() + frame_view = FrameView("/World/Cube", device=device) + request.addfinalizer(frame_view.close) + frame_view.get_world_poses() # Initialize its authored pose before the native pose write. + view = sim.physics_manager.get_physics_sim_view().create_rigid_body_view("/World/Cube") + view.set_transforms( + wp.array([[1, 2, 3, 0, 0, 0, 1]], dtype=wp.float32, device=device), + indices=wp.array([0], dtype=wp.int32, device=device), + ) + sim.step(render=False) + sim.forward() + before = tuple(value.torch.clone() for value in frame_view.get_world_poses()) + torch.testing.assert_close(before[0], torch.tensor([[1, 2, 3]], dtype=torch.float32, device=device)) + provider = SceneDataProvider(sim.get_scene_data_provider().backend) + output = SceneDataFormat.FabricMatrix44() + assert provider.get_transforms(output) + assert output.matrices.shape == (1,) + for value, expected in zip(frame_view.get_world_poses(), before, strict=True): + torch.testing.assert_close(value.torch, expected, rtol=0, atol=0) + + step_count = sim.get_physics_step_count() + view.set_transforms( + wp.array([[-2, 0.5, 4, 0, 0, 0, 1]], dtype=wp.float32, device=device), + indices=wp.array([0], dtype=wp.int32, device=device), + ) + sim.physics_manager.invalidate_transforms() + provider.get_transforms(output) + torch.testing.assert_close( + frame_view.get_world_poses()[0].torch, torch.tensor([[-2, 0.5, 4]], dtype=torch.float32, device=device) + ) + assert sim.get_physics_step_count() == step_count + + @pytest.mark.parametrize("device", test_devices()) def test_float_scale_initializes_fabric(device): """A legal float3 scale initializes Fabric without changing the FP32 view contract.""" @@ -317,23 +363,6 @@ def test_writer_scope_exception_recovers_state(device, view_factory): assert torch.allclose(follow_up_t, torch.tensor([[10.0, 11.0, 12.0]] * 2, device=device), atol=1e-5) -@pytest.mark.parametrize("device", ["cuda:0"]) -def test_prepare_for_reuse_detects_topology_change(device, view_factory): - """Each persistent ``PrimSelection`` exposes ``PrepareForReuse`` and returns a - bool. When the underlying Fabric topology is unchanged it returns False. - """ - bundle = view_factory(1, device) - view = bundle.view - view.get_world_poses() # trigger Fabric init - - assert view._fabric_sel.sel_ro is not None, "RO selection not initialized" - assert view._fabric_sel.sel_rw is not None, "RW selection not initialized" - for selection in (view._fabric_sel.sel_ro, view._fabric_sel.sel_rw): - result = selection.PrepareForReuse() - assert isinstance(result, bool), f"PrepareForReuse should return bool, got {type(result)}" - assert not result, "PrepareForReuse should return False when no topology change" - - @pytest.mark.parametrize("device", test_devices()) def test_selections_match_only_the_view_prims(device, view_factory): """Selections contain only the managed child prims and their unique parents. diff --git a/source/isaaclab_visualizers/changelog.d/sdp-transform-publication.rst b/source/isaaclab_visualizers/changelog.d/sdp-transform-publication.rst new file mode 100644 index 000000000000..45b28bb33218 --- /dev/null +++ b/source/isaaclab_visualizers/changelog.d/sdp-transform-publication.rst @@ -0,0 +1,6 @@ +Changed +^^^^^^^ + +* Routed Kit viewport transform updates through SDP, sharing a registry-owned Fabric binding with camera + renderers and preserving native PhysX Fabric updates. No visualizer configuration changes were required. + Headless viewport transforms and asset tracking refreshed only when a frame was requested. diff --git a/source/isaaclab_visualizers/isaaclab_visualizers/kit/kit_visualizer.py b/source/isaaclab_visualizers/isaaclab_visualizers/kit/kit_visualizer.py index fc9c4e6019bb..2b2602d14590 100644 --- a/source/isaaclab_visualizers/isaaclab_visualizers/kit/kit_visualizer.py +++ b/source/isaaclab_visualizers/isaaclab_visualizers/kit/kit_visualizer.py @@ -199,6 +199,9 @@ def initialize(self, scene_data_provider: SceneDataProvider) -> None: ) self._setup_streaming_view(num_envs) + sim = SimulationContext.instance() + self._fabric = sim.get_or_create_backend(sim.fabric_cfg) + self._fabric.bind_transforms(scene_data_provider) self._is_initialized = True self._setup_initial_camera_view() @@ -213,13 +216,13 @@ def step(self, dt: float) -> None: self._app_pumped_this_step = False self._sim_time += dt self._step_counter += 1 - # Update dynamic asset tracking before the frame renders. - if self.cfg.origin_type == "asset": - self._update_asset_tracking_camera() # Headless mode: skip the app update and camera panel refresh; rendering is # triggered on demand by render_rgb_array() / render_tiled_rgb_array(). if self._runtime_headless: return + self._fabric.update_transforms(self._scene_data_provider) + if self.cfg.origin_type == "asset": + self._update_asset_tracking_camera() _externally_paused = self.is_training_paused() if not _externally_paused: try: @@ -289,6 +292,9 @@ def render_rgb_array(self) -> np.ndarray: import omni.kit.app import omni.replicator.core as rep + self._fabric.update_transforms(self._scene_data_provider) + if self._runtime_headless and self.cfg.origin_type == "asset": + self._update_asset_tracking_camera() camera_path = self._controlled_camera_path or "/OmniverseKit_Persp" w, h = self.cfg.window_width, self.cfg.window_height @@ -418,10 +424,6 @@ def add_live_plots( if isinstance(source, DirectScalarLivePlots): self.kit_manager_visualizers[source.manager_name] = DirectScalarLiveVisualizer(source) - def requires_forward_before_step(self) -> bool: - """OV viewport relies on refreshed kinematic state before render.""" - return True - def pumps_app_update(self) -> bool: """KitVisualizer calls app.update() in step(), so render() should not do it again.""" return True @@ -1238,7 +1240,7 @@ def _setup_initial_camera_view(self) -> None: def _update_asset_tracking_camera(self) -> None: """Update the viewport camera to track an asset root or body. - Called every :meth:`step` when :attr:`KitVisualizerCfg.origin_type` is ``"asset"``. + Called before viewport frames when :attr:`KitVisualizerCfg.origin_type` is ``"asset"``. Parses :attr:`~KitVisualizerCfg.origin_track_path`: ``"asset_name"`` tracks the root, ``"asset_name/body_name"`` tracks a specific body. """ diff --git a/source/isaaclab_visualizers/test/test_kit_visualizer_scene_partitioning.py b/source/isaaclab_visualizers/test/test_kit_visualizer_scene_partitioning.py index 92c7186dfb1b..51a5f40a9106 100644 --- a/source/isaaclab_visualizers/test/test_kit_visualizer_scene_partitioning.py +++ b/source/isaaclab_visualizers/test/test_kit_visualizer_scene_partitioning.py @@ -19,6 +19,28 @@ from isaaclab.utils.renderers import ISAAC_RTX_SHOW_ALL_PARTITIONS_BY_DEFAULT_SETTING +@pytest.mark.parametrize("headless", [False, True]) +def test_viewport_pose_publication_is_deferred_for_headless_capture(monkeypatch, headless): + visualizer = KitVisualizer(KitVisualizerCfg(headless=headless, origin_type="asset")) + visualizer._is_initialized = True + visualizer._fabric = MagicMock() + visualizer._scene_data_provider = MagicMock() + monkeypatch.setattr(visualizer, "is_training_paused", lambda: True) + tracking = MagicMock() + monkeypatch.setattr(visualizer, "_update_asset_tracking_camera", tracking) + monkeypatch.setattr(visualizer, "_update_camera_image_panel", MagicMock()) + monkeypatch.setattr(visualizer, "_refresh_partial_viz_point_instancers_if_needed", MagicMock()) + + visualizer.step(0.1) + + assert tracking.call_count == int(not headless) + request = visualizer._fabric.update_transforms + if headless: + request.assert_not_called() + else: + request.assert_called_once_with(visualizer._scene_data_provider) + + @pytest.mark.parametrize("generated", [False, True]) def test_streaming_renderer_registers_before_visualizer_initialization(monkeypatch, generated): sim = MagicMock() diff --git a/source/isaaclab_visualizers/test/visualizer_golden_utils.py b/source/isaaclab_visualizers/test/visualizer_golden_utils.py index fdfb330ab905..0cf9a35b1277 100644 --- a/source/isaaclab_visualizers/test/visualizer_golden_utils.py +++ b/source/isaaclab_visualizers/test/visualizer_golden_utils.py @@ -388,7 +388,7 @@ def _capture_frame(env, viz_type: str, capture_mode: str, backend: str, actions: return _viz_utils._capture_visualizer_tiled_camera_rgb(_get_active_visualizer(env, viz_type)) if viz_type == "kit": return _viz_utils._capture_kit_viewport_with_pose_reapply( - env, _get_active_visualizer(env, "kit"), physics_backend=backend, prior_physics_steps=buffer_steps + env, _get_active_visualizer(env, "kit"), physics_backend=backend ) newton_viz = _get_active_visualizer(env, "newton") viewer = getattr(newton_viz, "_viewer", None) @@ -463,7 +463,6 @@ def _capture_frame(env, viz_type: str, capture_mode: str, backend: str, actions: _get_active_visualizer(env, "kit"), resolution=_viz_utils._SHADOW_HAND_KIT_INTEGRATION_RENDER_RESOLUTION, physics_backend=backend, - prior_physics_steps=0, ) newton_viz = _get_active_visualizer(env, "newton") viewer = getattr(newton_viz, "_viewer", None) @@ -539,7 +538,6 @@ def _capture_frame(env, viz_type: str, capture_mode: str, backend: str, actions: _get_active_visualizer(env, "kit"), resolution=_viz_utils._ANYMAL_D_KIT_INTEGRATION_RENDER_RESOLUTION, physics_backend=backend, - prior_physics_steps=_viz_utils._START_BUFFER_STEPS, ) newton_viz = _get_active_visualizer(env, "newton") viewer = getattr(newton_viz, "_viewer", None) @@ -602,13 +600,7 @@ def _capture_frame(env, viz_type: str, capture_mode: str, actions: torch.Tensor) if capture_mode == "tiled": return _viz_utils._capture_visualizer_tiled_camera_rgb(_get_active_visualizer(env, viz_type)) if viz_type == "kit": - # Do NOT call env.sim.render() here: the VBD cloth solver never sets - # NewtonManager._newton_fabric_ready, so env.sim.render() blocks in - # the Fabric sync path indefinitely on some GPU/driver combinations - # (observed 48+ min hang on RTX PRO 4500 Blackwell). Instead use - # app_updates_only=True which drives RTX TAA via lightweight app.update() - # ticks without triggering Newton Fabric sync. The 12%/SSIM-0.85 - # thresholds are loose enough to accept the resulting frame quality. + # Warm up RTX TAA without advancing the cloth simulation. return _viz_utils._capture_kit_viewport_with_pose_reapply( env, _get_active_visualizer(env, "kit"), diff --git a/source/isaaclab_visualizers/test/visualizer_integration_utils.py b/source/isaaclab_visualizers/test/visualizer_integration_utils.py index a7e95474820a..dc99ff79b925 100644 --- a/source/isaaclab_visualizers/test/visualizer_integration_utils.py +++ b/source/isaaclab_visualizers/test/visualizer_integration_utils.py @@ -142,13 +142,7 @@ """Hard cap on render frames pumped during convergence-based warmup.""" _FRANKA_CLOTH_KIT_VIEWPORT_WARMUP_FRAMES = 20 -"""Franka cloth kit-viewport warmup uses lightweight ``app.update()`` ticks -(not ``env.sim.render()``). Each ``env.sim.render()`` call for the VBD cloth -scene blocks in the Newton Fabric sync path (the VBD cloth solver never sets -``NewtonManager._newton_fabric_ready``), causing hangs on some GPU/driver -combinations. 20 ``app.update()`` ticks drive RTX TAA accumulation without -triggering the Fabric sync, producing an acceptable frame within the -loose 12% / SSIM-0.85 thresholds.""" +"""Bounded RTX TAA warmup for the Franka cloth viewport capture.""" _WARMUP_STABLE_DIFF_PCT = 0.5 """Fraction of pixels (%) with inter-frame L2 > 1.0 below which two consecutive frames are @@ -1060,78 +1054,29 @@ def _reapply_kit_camera_pose(env, kit_visualizer: KitVisualizer) -> None: _update_active_simulation_app() -def _force_newton_transforms_resync() -> None: - """Force-mark Newton body transforms and particles dirty and re-sync to USD Fabric. - - Needed when the Fabric SelectPrims check fails on a prior pre_render() call (GPU - attribute propagation delay), leaving dirty flags cleared without writing positions. - """ - with contextlib.suppress(Exception): - from isaaclab_newton.physics import NewtonManager # noqa: PLC0415 - - if NewtonManager._usdrt_stage is not None and NewtonManager.backend is not None: - NewtonManager._transforms_dirty = True - NewtonManager.sync_transforms_to_fabric() - NewtonManager._particles_dirty = True - NewtonManager.sync_particles_to_usd() - - -def _drain_until_newton_fabric_ready(max_updates: int = 200, updates_per_iter: int = 2) -> None: - """Pump Kit updates until Newton has written body positions to Fabric. - - Polls ``NewtonManager._newton_fabric_ready`` (set after the first successful - SelectPrims call) with real-time sleeps so the GPU can process pending Fabric work. - Returns immediately if already ready (common case after a normal physics warmup). - - The tiled-camera path uses ``max_updates=600`` safely (tiled cameras are not rendered - until ``camera_sensor.update()``); the viewport path keeps a lower ceiling to limit - contaminated TAA frames accumulating during the drain. - """ - with contextlib.suppress(Exception): - from isaaclab_newton.physics import NewtonManager # noqa: PLC0415 - - for _ in range(max(0, int(max_updates))): - if NewtonManager._newton_fabric_ready: - return - with contextlib.suppress(Exception): - import torch # noqa: PLC0415 - - if torch.cuda.is_available(): - torch.cuda.synchronize() - _force_newton_transforms_resync() - _drain_kit_app_updates(updates_per_iter) - - def _capture_kit_viewport_with_pose_reapply( env, kit_visualizer: KitVisualizer, resolution: tuple[int, int] | None = None, physics_backend: str = "", - prior_physics_steps: int = 0, max_warmup_frames: int | None = None, app_updates_only: bool = False, ) -> np.ndarray: """Set the configured eye/lookat, warm RTX, then capture. - Re-applies the camera between the two ``app.update()`` calls in the warmup loop so - that Newton stage init (which resets the viewport camera) does not affect the final - frame. When ``prior_physics_steps > 0``, also re-syncs Newton body transforms - between the two calls so the correct pose is rendered. + Re-applies the camera after rendering so Newton viewport initialization does not + change the captured viewpoint. The normal render path refreshes body transforms. Args: env: The simulation environment. kit_visualizer: The active :class:`KitVisualizer` instance. resolution: Optional ``(width, height)`` override for the render product. - physics_backend: ``"newton"`` to enable per-render camera reapply and body- - transform re-sync. - prior_physics_steps: When > 0, injects a Newton body-transform re-sync - between the two ``app.update()`` calls. + physics_backend: ``"newton"`` to enable per-render camera reapply. max_warmup_frames: When set, overrides the default convergence cap. Use a small value when per-frame render cost is very high and test thresholds are loose enough that convergence is not required (e.g. franka cloth RTX). - app_updates_only: When True, uses lightweight ``app.update()`` ticks instead - of ``env.sim.render()``. Required for VBD cloth scenes where - ``env.sim.render()`` blocks in the Newton Fabric sync path. + app_updates_only: When True, warms RTX with ``app.update()`` ticks instead + of ``env.sim.render()``. """ kit_visualizer.set_camera_view(kit_visualizer.cfg.eye, kit_visualizer.cfg.lookat) camera_path = getattr(kit_visualizer, "_controlled_camera_path", None) @@ -1139,14 +1084,11 @@ def _capture_kit_viewport_with_pose_reapply( annotator, render_product = _build_rgb_annotator_for_camera(camera_path, resolution=resolution) try: if physics_backend == "newton": - _drain_until_newton_fabric_ready() prev: np.ndarray | None = None for i in range(_WARMUP_MAX_FRAMES): kit_visualizer.set_camera_view(kit_visualizer.cfg.eye, kit_visualizer.cfg.lookat) env.sim.render() kit_visualizer.set_camera_view(kit_visualizer.cfg.eye, kit_visualizer.cfg.lookat) - if prior_physics_steps > 0: - _force_newton_transforms_resync() _update_active_simulation_app() with contextlib.suppress(Exception): annotator.get_data() @@ -1187,9 +1129,8 @@ def _warm_kit_rtx_render_product( satisfy :func:`_frames_converged` or :data:`_WARMUP_MAX_FRAMES` is reached. When ``max_frames_override`` is set, it replaces both caps above — useful when the per-frame render cost is high and loose thresholds make convergence unnecessary. - When ``app_updates_only`` is True, replaces ``env.sim.render()`` with lightweight - ``app.update()`` calls. Use this for VBD cloth scenes where ``env.sim.render()`` - blocks in the Newton Fabric sync path (VBD cloth particles never set the ready flag). + When ``app_updates_only`` is True, warms RTX with ``app.update()`` calls without + advancing the visualizers. """ if max_frames_override is not None: max_frames = max_frames_override @@ -1369,23 +1310,7 @@ def _capture_visualizer_tiled_camera_rgb( if force_recompute and getattr(visualizer, "_camera_is_owned", False): visualizer._update_owned_camera_poses() if isinstance(visualizer, KitVisualizer): - # Probe with a short drain to detect backend: on Newton, _newton_fabric_ready is set - # after the first iteration; on PhysX it is never set so we skip the full drain and - # let _pump_tiled_until_stable handle convergence instead. - _drain_until_newton_fabric_ready(max_updates=20, updates_per_iter=4) - try: - from isaaclab_newton.physics import NewtonManager # noqa: PLC0415 - - if NewtonManager._newton_fabric_ready: - if not paused: - _drain_until_newton_fabric_ready(max_updates=600, updates_per_iter=4) - _update_active_simulation_app() - if not paused: - _force_newton_transforms_resync() - else: - _update_active_simulation_app() - except Exception: - _update_active_simulation_app() + _update_active_simulation_app() return _pump_tiled_until_stable(camera_sensor, camera_indices) rgb_batch = camera_rgb_batch(camera_sensor, camera_indices) frame = compose_rgb_grid_tensor(rgb_batch).detach().cpu().numpy() @@ -1727,10 +1652,7 @@ def _make_anymal_d_env(visualizer_kind: str | tuple[str, ...], backend_kind: str _FRANKA_CLOTH_WARMUP_STEPS = 1 """Steps after reset before capturing the franka cloth scene. -One step lets Newton propagate articulation FK so all robot arm links are visible at the -correct positions. At 0 steps, Newton has not yet synced body positions to USD Fabric, -leaving the arm links at the origin and invisible in the Kit viewport. One step also lets -the cloth begin falling under gravity while remaining in a nearly-deterministic pose — the +One step lets the cloth begin falling under gravity while remaining in a nearly-deterministic pose — the VBD solver's non-deterministic parallel reductions accumulate over many steps, so capturing at 1 step keeps inter-run pixel variance much lower than at 20 steps. This mirrors the approach used in the kitless rendering tests in From b92d1d9665ebf3a947918d9fc1dc7d85db785dd9 Mon Sep 17 00:00:00 2001 From: Piotr Barejko Date: Thu, 24 Sep 2026 08:18:00 -0700 Subject: [PATCH 2/8] Batch camera rendering and fix moving-camera and cloned-camera updates (#7977) (cherry picked from commit 1e520e9d5cb44f54b70190f704f9d3dd634fa60f) --- docs/source/api/lab/isaaclab.sensors.rst | 23 ++ docs/source/concepts/renderers.rst | 30 ++ docs/source/concepts/sensors/index.rst | 6 + scripts/benchmarks/nsys_trace.json | 2 + .../changelog.d/render-batch.minor.rst | 10 + .../isaaclab/isaaclab/benchmark/stepping.py | 10 +- .../isaaclab/renderers/base_renderer.py | 16 +- .../isaaclab/renderers/render_context.py | 34 +- .../isaaclab/scene/interactive_scene.py | 13 +- .../isaaclab/sensors/camera/camera.py | 49 ++- .../isaaclab/isaaclab/sensors/sensor_base.py | 81 +++- .../isaaclab/test/benchmark/test_stepping.py | 28 +- .../test_simulation_render_context.py | 381 +++++++++++++++++- .../changelog.d/ovrtx-render-batch.minor.rst | 5 + .../isaaclab_ov/renderers/ovrtx_renderer.py | 75 ++-- .../test/test_ovrtx_renderer_contract.py | 77 +++- .../test/test_ovrtx_visual_material_sync.py | 15 +- .../changelog.d/isaac-rtx-render-batch.rst | 5 + .../renderers/isaac_rtx_renderer.py | 29 +- .../test_isaac_rtx_renderer_contract.py | 63 ++- 20 files changed, 867 insertions(+), 85 deletions(-) create mode 100644 source/isaaclab/changelog.d/render-batch.minor.rst create mode 100644 source/isaaclab_ov/changelog.d/ovrtx-render-batch.minor.rst create mode 100644 source/isaaclab_physx/changelog.d/isaac-rtx-render-batch.rst diff --git a/docs/source/api/lab/isaaclab.sensors.rst b/docs/source/api/lab/isaaclab.sensors.rst index f95c0cbbf365..2935e99bd0a8 100644 --- a/docs/source/api/lab/isaaclab.sensors.rst +++ b/docs/source/api/lab/isaaclab.sensors.rst @@ -46,6 +46,29 @@ Sensor Base ----------- +.. rubric:: Extending batch updates + +Sensors opt into eager batch updates by overriding +:attr:`SensorBase.supports_batch_update`, which defaults to ``False``. +Eager scene updates advance sensors in order, then batch the remaining buffer refreshes for +sensors that opted in. Lazy data access remains per sensor. + +For standalone use, :meth:`SensorBase.update_batch` performs the complete eager update for a +sequence of batch-capable sensors. It calls each sensor's ``update()`` once in the supplied order, +then refreshes pending buffers together. Each call advances sensor clocks by ``dt``, so use it +in place of separate ``sensor.update(dt)`` calls. An unsupported sensor raises ``ValueError`` +before any input is updated; update unsupported sensors individually. + +An implementation can override the static ``_update_buffers_batch_impl(sensors)`` hook to perform +shared work. Sensors inheriting the same hook are grouped together, including instances of +different subclasses. The hook must respect each sensor's outdated-environment mask and fill its +buffers; ``SensorBase`` handles the update timestamps after the hook succeeds. The default hook +calls each sensor's individual buffer implementation. + +Camera subclasses that override the individual buffer-update hooks retain individual updates by +default. They must explicitly opt into batching and ensure their batch implementation preserves +the custom behavior. + .. autoclass:: SensorBase :members: diff --git a/docs/source/concepts/renderers.rst b/docs/source/concepts/renderers.rst index 1568f259e34e..f4f55dd45934 100644 --- a/docs/source/concepts/renderers.rst +++ b/docs/source/concepts/renderers.rst @@ -428,6 +428,36 @@ For the RTX renderer (requires Isaac Sim): For RTX renderer settings, see :doc:`/source/how-to/configure_rendering`. +.. _renderer-camera-batching: + +Batching camera renders +----------------------- + +:meth:`~isaaclab.renderers.BaseRenderer.render_batch` accepts a sequence of render-data objects +owned by the same renderer. Prepare the camera poses, intrinsics, and shared scene state before +rendering, then read each camera's output. An empty sequence performs no rendering. + +.. code-block:: python + + renderer.render_batch([first_render_data, second_render_data]) + renderer.read_output(first_render_data, first_camera_data) + renderer.read_output(second_render_data, second_camera_data) + +:meth:`~isaaclab.renderers.BaseRenderer.render` continues to accept a single render-data object. +The default ``render_batch()`` implementation calls ``render()`` for each entry, so existing +custom renderers and single-camera callers need no changes. OVRTX overrides ``render_batch()`` +to submit the requested camera products in one native renderer step. + +With eager sensor updates (``scene.cfg.lazy_sensor_update=False``), ``scene.update()`` advances +sensor clocks in scene order and collects batch-capable sensors. After the loop, the camera +batch implementation prepares the remaining due captures for submission. +:meth:`~isaaclab.renderers.RenderContext.render_into_cameras` groups these requests by renderer +instance, renders each group, and reads its outputs. The context does not retain a pending-camera +queue. Each camera retains its own update period and reset state. + +With lazy sensor updates, reading a camera's ``data`` refreshes only that camera's sensor buffers +and capture timestamps. It does not refresh peer camera sensors sharing the renderer. + Core concepts ------------- diff --git a/docs/source/concepts/sensors/index.rst b/docs/source/concepts/sensors/index.rst index fe48c7b69810..5a1bc3dd81e1 100644 --- a/docs/source/concepts/sensors/index.rst +++ b/docs/source/concepts/sensors/index.rst @@ -22,6 +22,12 @@ sensor derives from :class:`~isaaclab.sensors.SensorBase` and follows the same l debug visualization is requested. * :meth:`~isaaclab.sensors.SensorBase.reset` clears per-environment timestamps and internal state. +With eager sensor updates (``scene.cfg.lazy_sensor_update=False``), ``scene.update(dt)`` refreshes +sensor data before returning. It advances sensor clocks in scene order, updating ordinary sensors +individually and collecting sensors that support batch updates. After the loop, it refreshes the +collected sensors' pending buffers together. Each sensor retains its own update period and reset +state. Camera sensors use this mechanism to :ref:`batch captures by renderer `. + Sensor data is exposed through :class:`~isaaclab.utils.warp.ProxyArray` buffers, including camera outputs. Use the ``torch`` property for a cached zero-copy Torch view or ``warp`` for the underlying Warp array. diff --git a/scripts/benchmarks/nsys_trace.json b/scripts/benchmarks/nsys_trace.json index e3a5b8a16675..fa8523455cb8 100644 --- a/scripts/benchmarks/nsys_trace.json +++ b/scripts/benchmarks/nsys_trace.json @@ -89,6 +89,7 @@ "color": "0xFFC107", "module": "isaaclab.renderers.render_context", "functions": [ + {"function": "RenderContext.render_into_cameras", "color": "0xFFC107"}, {"function": "RenderContext.render_into_camera", "color": "0xFFC107"}, {"function": "RenderContext.update_scene_state", "color": "0xFFD54F"}, {"function": "RenderContext.ensure_initialize", "color": "0xFFE082"}, @@ -266,6 +267,7 @@ "OVRTXRenderer.update_geometries", "OVRTXRenderer.update_camera", "OVRTXRenderer.render", + "OVRTXRenderer.render_batch", "OVRTXRenderer.read_output", {"module": "isaaclab_ov.renderers.ovrtx_usd", "function": "create_scene_partition_attributes"}, {"module": "isaaclab_ov.renderers.ovrtx_usd", "function": "export_stage_to_string"} diff --git a/source/isaaclab/changelog.d/render-batch.minor.rst b/source/isaaclab/changelog.d/render-batch.minor.rst new file mode 100644 index 000000000000..51df5f5c29b4 --- /dev/null +++ b/source/isaaclab/changelog.d/render-batch.minor.rst @@ -0,0 +1,10 @@ +Added +^^^^^ + +* Added ``BaseRenderer.render_batch()`` with a default loop over the existing single-camera + ``render()`` interface, allowing renderers to optimize multiple camera captures. +* Added ``SensorBase.supports_batch_update`` to opt sensors into eager scene batching and + ``SensorBase.update_batch()`` for standalone updates of batch-capable sensors. +* Added ``RenderContext.render_into_cameras()`` to group prepared captures by renderer and + batch due cameras during eager scene updates. Preserved per-camera lazy reads, update + periods, and reset state. diff --git a/source/isaaclab/isaaclab/benchmark/stepping.py b/source/isaaclab/isaaclab/benchmark/stepping.py index 34f3ead2dc52..d72a22439e01 100644 --- a/source/isaaclab/isaaclab/benchmark/stepping.py +++ b/source/isaaclab/isaaclab/benchmark/stepping.py @@ -65,24 +65,24 @@ def profile_renderers( originals = [] try: for _, renderer in render_context._renderer_entries: - render = renderer.render - original = vars(renderer).get("render", missing) + render = renderer.render_batch + original = vars(renderer).get("render_batch", missing) @wraps(render) def timed_render(render_data: Any, _render=render) -> None: with wp.ScopedTimer(RENDER_PROFILE_SCOPE, dict=scope_timings, print=False, synchronize=True): return _render(render_data) - renderer.render = timed_render + renderer.render_batch = timed_render originals.append((renderer, original)) yield timings finally: for renderer, original in reversed(originals): if original is missing: - del renderer.render + del renderer.render_batch else: - renderer.render = original + renderer.render_batch = original @contextmanager diff --git a/source/isaaclab/isaaclab/renderers/base_renderer.py b/source/isaaclab/isaaclab/renderers/base_renderer.py index 2f81f5538763..ba79dc6d81e8 100644 --- a/source/isaaclab/isaaclab/renderers/base_renderer.py +++ b/source/isaaclab/isaaclab/renderers/base_renderer.py @@ -15,7 +15,7 @@ from .output_contract import RenderBufferKind, RenderBufferSpec if TYPE_CHECKING: - from collections.abc import Callable + from collections.abc import Callable, Sequence import torch import warp as wp @@ -183,6 +183,20 @@ def render(self, render_data: Any) -> None: """ pass + def render_batch(self, render_data: Sequence[Any]) -> None: + """Render a collection of cameras into their bound output buffers. + + All camera poses and shared scene state must be prepared before calling this method. + An empty sequence is a no-op. Each object must belong to this renderer and appear once. + The default implementation calls :meth:`render` for each camera; subclasses may override + this method to submit all cameras together. + + Args: + render_data: Renderer-specific objects from :meth:`create_render_data`. + """ + for data in render_data: + self.render(data) + @abstractmethod def read_output(self, render_data: Any, camera_data: CameraData) -> None: """Read rendered outputs from the renderer into the camera data container. diff --git a/source/isaaclab/isaaclab/renderers/render_context.py b/source/isaaclab/isaaclab/renderers/render_context.py index bba6f7a76bf7..9d3e7ac55310 100644 --- a/source/isaaclab/isaaclab/renderers/render_context.py +++ b/source/isaaclab/isaaclab/renderers/render_context.py @@ -9,6 +9,7 @@ import logging import warnings +from collections.abc import Sequence from typing import TYPE_CHECKING, Any import torch @@ -334,10 +335,37 @@ def render_into_camera( camera_data: CameraData, physics_step_count: int, ) -> None: - """Sync scene state, render, and read outputs into ``camera_data``.""" + """Sync scene state and capture one camera through :meth:`render_into_cameras`.""" + self.render_into_cameras([(renderer, render_data, camera_data)], physics_step_count) + + def render_into_cameras( + self, + requests: Sequence[tuple[BaseRenderer, Any, CameraData]], + physics_step_count: int, + ) -> None: + """Render prepared cameras in batches grouped by renderer instance. + + Camera poses must be updated before this call. Requests are used only for this + submission; the context does not retain cameras or manage sensor timing. + + Args: + requests: Tuples of renderer, renderer-specific render data, and output camera data. + An empty sequence performs no work. + physics_step_count: Current physics step for shared scene synchronization. + """ + if not requests: + return + self.update_scene_state(physics_step_count) - renderer.render(render_data) - renderer.read_output(render_data, camera_data) + + groups: dict[int, tuple[BaseRenderer, list[tuple[Any, CameraData]]]] = {} + for renderer, render_data, camera_data in requests: + groups.setdefault(id(renderer), (renderer, []))[1].append((render_data, camera_data)) + + for renderer, cameras in groups.values(): + renderer.render_batch([render_data for render_data, _ in cameras]) + for render_data, camera_data in cameras: + renderer.read_output(render_data, camera_data) def reset_stage_prepare_flag(self) -> None: """Allow :meth:`ensure_prepare_stage` to run ``prepare_stage`` again (e.g. a new USD stage).""" diff --git a/source/isaaclab/isaaclab/scene/interactive_scene.py b/source/isaaclab/isaaclab/scene/interactive_scene.py index ca5d61ccd5ae..0ad7548cda61 100644 --- a/source/isaaclab/isaaclab/scene/interactive_scene.py +++ b/source/isaaclab/isaaclab/scene/interactive_scene.py @@ -516,8 +516,19 @@ def update(self, dt: float) -> None: for surface_gripper in self._surface_grippers.values(): surface_gripper.update(dt) # -- sensors + force_recompute = not self.cfg.lazy_sensor_update + batched_sensors: list[SensorBase] = [] for sensor in self._sensors.values(): - sensor.update(dt, force_recompute=not self.cfg.lazy_sensor_update) + if force_recompute and sensor.supports_batch_update: + # Defer buffer refresh until all sensors have advanced. + sensor.update(dt, force_recompute=False) + batched_sensors.append(sensor) + else: + sensor.update(dt, force_recompute=force_recompute) + + # Process batched sensors only during eager updates. + if batched_sensors: + SensorBase._process_batch(batched_sensors) """ Operations: Scene State. diff --git a/source/isaaclab/isaaclab/sensors/camera/camera.py b/source/isaaclab/isaaclab/sensors/camera/camera.py index 68b24a8cf77c..5324542ca62c 100644 --- a/source/isaaclab/isaaclab/sensors/camera/camera.py +++ b/source/isaaclab/isaaclab/sensors/camera/camera.py @@ -8,7 +8,7 @@ import logging import sys from collections.abc import Sequence -from typing import TYPE_CHECKING, Any, Literal +from typing import TYPE_CHECKING, Any, Literal, cast import numpy as np import torch @@ -329,6 +329,18 @@ def __str__(self) -> str: def num_instances(self) -> int: return self._view.count + @property + def supports_batch_update(self) -> bool: + """Whether captures can use the shared camera batch implementation. + + Custom scalar capture hooks retain their individual update path. Subclasses may opt + in explicitly when they also provide a compatible ``_update_buffers_batch_impl``. + """ + return ( + type(self)._update_buffers_impl is Camera._update_buffers_impl + and type(self)._update_outdated_buffers is SensorBase._update_outdated_buffers + ) + @property def data(self) -> CameraData: # update sensors if needed @@ -708,15 +720,18 @@ def _initialize_impl(self): # Create internal buffers (includes intrinsic matrix and pose init) self._create_buffers() - def _update_buffers_impl(self, env_mask: wp.array): - if not self._env_mask_has_any(env_mask): - return - # Increment frame count + def _prepare_camera(self, env_mask: wp.array) -> None: + """Advance capture frames and refresh requested poses before rendering.""" if self.cfg.update_latest_camera_pose: self._update_poses(env_mask=env_mask, frame_op=1) else: self._update_camera_state(env_mask=env_mask, frame_op=1) + def _update_buffers_impl(self, env_mask: wp.array): + if not self._env_mask_has_any(env_mask): + return + self._prepare_camera(env_mask) + sim_ctx = sim_utils.SimulationContext.instance() renderer = self._renderer assert renderer is not None @@ -731,6 +746,30 @@ def _update_buffers_impl(self, env_mask: wp.array): renderer.render(self._render_data) renderer.read_output(self._render_data, self._data) + @staticmethod + def _update_buffers_batch_impl(sensors: Sequence[SensorBase]) -> None: + """Prepare due cameras and render them together through their shared context.""" + cameras = cast(Sequence[Camera], sensors) + sim_ctx = sim_utils.SimulationContext.instance() + if sim_ctx is None: + for camera in cameras: + camera._update_buffers_impl(camera._is_outdated) + return + + ready = [] + for camera in cameras: + if not camera._env_mask_has_any(camera._is_outdated): + continue + camera._prepare_camera(camera._is_outdated) + ready.append(camera) + if not ready: + return + + sim_ctx.render_context.render_into_cameras( + [(camera._renderer, camera._render_data, camera._data) for camera in ready], + sim_ctx.get_physics_step_count(), + ) + """ Private Helpers """ diff --git a/source/isaaclab/isaaclab/sensors/sensor_base.py b/source/isaaclab/isaaclab/sensors/sensor_base.py index 274b7089309a..c405e78a2840 100644 --- a/source/isaaclab/isaaclab/sensors/sensor_base.py +++ b/source/isaaclab/isaaclab/sensors/sensor_base.py @@ -16,7 +16,7 @@ import sys import weakref from abc import ABC, abstractmethod -from collections.abc import Sequence +from collections.abc import Callable, Sequence from typing import TYPE_CHECKING, Any import warp as wp @@ -39,10 +39,9 @@ class SensorBase(ABC): """The base class for implementing a sensor. - The implementation is based on lazy evaluation. The sensor data is only updated when the user - tries accessing the data through the :attr:`data` property or sets ``force_compute=True`` in - the :meth:`update` method. This is done to avoid unnecessary computation when the sensor data - is not used. + The implementation is based on lazy evaluation. Sensor buffers are refreshed through the + :attr:`data` property, by setting ``force_recompute=True`` in :meth:`update`, or by calling + :meth:`update_batch`. This avoids unnecessary computation when sensor data is not used. The sensor is updated at the specified update period. If the update period is zero, then the sensor is updated at every simulation step. @@ -106,6 +105,15 @@ def device(self) -> str: """Memory device for computation.""" return self._device + @property + def supports_batch_update(self) -> bool: + """Whether eager buffer refreshes can share work through :meth:`update_batch`. + + Defaults to False. Sensors may opt in and implement ``_update_buffers_batch_impl`` + to share work with sensors that use the same batch implementation. + """ + return False + @property @abstractmethod def data(self) -> Any: @@ -211,6 +219,34 @@ def update(self, dt: float, force_recompute: bool = False): if force_recompute or self._is_visualizing: self._update_outdated_buffers(force_recompute=force_recompute) + @staticmethod + def update_batch(sensors: Sequence[SensorBase], dt: float) -> None: + """Advance batch-capable sensors and eagerly refresh their data in compatible groups. + + All sensors must report :attr:`supports_batch_update` as True. Calls each sensor's + :meth:`update` once in input order with ``force_recompute=False``, then processes + pending buffers through their shared ``_update_buffers_batch_impl`` static methods. + Use this method instead of calling :meth:`update` separately for the same time step. + + Uninitialized sensors and sensors already refreshed during the update loop are excluded + from batch processing. Each batch is marked updated only after its implementation + returns successfully. + + Args: + sensors: Batch-capable sensors to update, each appearing once. An empty sequence + performs no work. + dt: Time elapsed since the previous sensor update [s]. + + Raises: + ValueError: If any sensor does not support batch updates. No sensors are advanced + in this case. + """ + if any(not sensor.supports_batch_update for sensor in sensors): + raise ValueError("Batch updates require sensors with supports_batch_update=True.") + for sensor in sensors: + sensor.update(dt, force_recompute=False) + SensorBase._process_batch(sensors) + """ Implementation specific. """ @@ -276,6 +312,21 @@ def _update_buffers_impl(self, env_mask: wp.array): """ raise NotImplementedError + @staticmethod + def _update_buffers_batch_impl(sensors: Sequence[SensorBase]) -> None: + """Fill buffers for initialized sensors that share this batch implementation. + + Each sensor's ``_is_outdated`` mask selects its due environments and may be empty. + Implementations must fill the requested buffers before returning and leave timestamp + and generation bookkeeping to the caller. The default implementation + calls each sensor's ``_update_buffers_impl`` individually. + + Args: + sensors: Sensors whose buffers need checking, all using this static method. + """ + for sensor in sensors: + sensor._update_buffers_impl(sensor._is_outdated) + def _set_debug_vis_impl(self, debug_vis: bool): """Set debug visualization into visualization objects. @@ -398,6 +449,10 @@ def _update_outdated_buffers(self, force_recompute: bool = False) -> None: if not force_recompute and self._data_generation == self._data_generation_last_update: return self._update_buffers_impl(self._is_outdated) + self._mark_buffers_updated() + + def _mark_buffers_updated(self) -> None: + """Commit capture timestamps after the sensor's output buffers have been filled.""" # update timestamps and clear outdated flags wp.launch( update_outdated_envs_kernel, @@ -407,6 +462,22 @@ def _update_outdated_buffers(self, force_recompute: bool = False) -> None: ) self._data_generation_last_update = self._data_generation + @staticmethod + def _process_batch(sensors: Sequence[SensorBase]) -> None: + """Refresh pending buffers for batch-capable sensors whose timing has already advanced.""" + # A later sensor's update may already have refreshed an earlier sensor's data. + groups: dict[Callable[[Sequence[SensorBase]], None], list[SensorBase]] = {} + for sensor in sensors: + if not sensor.is_initialized or sensor._data_generation == sensor._data_generation_last_update: + continue + batch_impl = type(sensor)._update_buffers_batch_impl + groups.setdefault(batch_impl, []).append(sensor) + + for batch_impl, group in groups.items(): + batch_impl(group) + for sensor in group: + sensor._mark_buffers_updated() + def _resolve_indices_and_mask( self, env_ids: Sequence[int] | None = None, env_mask: wp.array | None = None ) -> wp.array: diff --git a/source/isaaclab/test/benchmark/test_stepping.py b/source/isaaclab/test/benchmark/test_stepping.py index 1bcacd3bac89..364d361140c8 100644 --- a/source/isaaclab/test/benchmark/test_stepping.py +++ b/source/isaaclab/test/benchmark/test_stepping.py @@ -97,11 +97,11 @@ def test_profile_renderers_wraps_each_renderer(monkeypatch, capsys): second_render = Mock(side_effect=ValueError("render failed")) class Renderer: - def render(self, render_data): + def render_batch(self, render_data): second_render(render_data) - first, second = SimpleNamespace(render=Mock()), Renderer() - originals = [first.render, second.render] + first, second = SimpleNamespace(render_batch=Mock()), Renderer() + originals = [first.render_batch, second.render_batch] context = SimpleNamespace(_renderer_entries=[(None, first), (None, second)]) synchronize = Mock() monkeypatch.setattr(wp, "synchronize", synchronize) @@ -110,28 +110,28 @@ def render(self, render_data): with pytest.raises(ValueError, match="render failed"): with profile_renderers(context, timings=timings) as collected: assert collected is timings - first.render("first") - second.render("second") + first.render_batch(["first"]) + second.render_batch(["second"]) - originals[0].assert_called_once_with("first") - second_render.assert_called_once_with("second") + originals[0].assert_called_once_with(["first"]) + second_render.assert_called_once_with(["second"]) assert synchronize.call_count == 4 assert len(timings) == 2 assert all(scope == RENDER_PROFILE_SCOPE and elapsed >= 0.0 for scope, elapsed in timings) assert RENDER_PROFILE_SCOPE not in capsys.readouterr().out - assert [first.render, second.render] == originals - assert "render" not in vars(second) + assert [first.render_batch, second.render_batch] == originals + assert "render_batch" not in vars(second) second_render.side_effect = None - first.render("unprofiled") + first.render_batch(["unprofiled"]) for enabled in (True, False): with profile_renderers(context, active=enabled) as later_timings: - first.render("first") - second.render("second") + first.render_batch(["first"]) + second.render_batch(["second"]) assert len(later_timings) == 2 * int(enabled) assert len(timings) == 2 - assert [first.render, second.render] == originals - assert "render" not in vars(second) + assert [first.render_batch, second.render_batch] == originals + assert "render_batch" not in vars(second) assert synchronize.call_count == 8 diff --git a/source/isaaclab/test/renderers/test_simulation_render_context.py b/source/isaaclab/test/renderers/test_simulation_render_context.py index 3ada5ae132d9..678a2cb557ed 100644 --- a/source/isaaclab/test/renderers/test_simulation_render_context.py +++ b/source/isaaclab/test/renderers/test_simulation_render_context.py @@ -7,18 +7,24 @@ from __future__ import annotations +import contextlib from types import SimpleNamespace from unittest.mock import Mock, call +import numpy as np import pytest import torch +import warp as wp from isaaclab.benchmark.stepping import RENDER_PROFILE_SCOPE, profile_renderers from isaaclab.renderers.base_renderer import BaseRenderer from isaaclab.renderers.render_context import RenderContext from isaaclab.renderers.renderer_cfg import RendererCfg +from isaaclab.scene import InteractiveScene +from isaaclab.sensors import Camera, SensorBase from isaaclab.sensors.camera.camera_data import CameraData from isaaclab.sim import BackendCfg, SimulationContext +from isaaclab.utils.warp import ProxyArray pytest.importorskip("isaaclab_physx") pytest.importorskip("isaaclab_newton") @@ -187,10 +193,10 @@ def test_render_into_camera_call_order_and_profile_output(sim, capsys, profile): assert renderer.mock_calls == [ call.update_transforms(), call.update_geometries(), - call.render(data), + call.render_batch([data]), call.read_output(data, camera), call.update_transforms(), - call.render(data), + call.render_batch([data]), call.read_output(data, camera), ] assert len(timings) == (2 if profile else 0) @@ -198,6 +204,41 @@ def test_render_into_camera_call_order_and_profile_output(sim, capsys, profile): assert RENDER_PROFILE_SCOPE not in capsys.readouterr().out +def test_default_render_batch_preserves_single_camera_render_contract(): + """Existing render implementations handle batches in order, including empty batches.""" + renderer = SimpleNamespace(render=Mock()) + render_data = (object(), object()) + + BaseRenderer.render_batch(renderer, ()) + renderer.render.assert_not_called() + BaseRenderer.render_batch(renderer, render_data) + assert renderer.render.call_args_list == [call(data) for data in render_data] + + +def test_render_into_cameras_groups_renderers_and_reads_each_output(sim): + first = sim.get_or_create_backend(RendererCfg(class_type=_renderer)) + second = sim.get_or_create_backend(RendererCfg(class_type=_renderer, renderer_type="second")) + data = [object() for _ in range(3)] + cameras = [CameraData() for _ in data] + + sim.render_context.render_into_cameras([], physics_step_count=1) + first.render_batch.assert_not_called() + second.render_batch.assert_not_called() + sim.render_context.render_into_cameras( + [(first, data[0], cameras[0]), (second, data[1], cameras[1]), (first, data[2], cameras[2])], + physics_step_count=1, + ) + + first.render_batch.assert_called_once_with([data[0], data[2]]) + second.render_batch.assert_called_once_with([data[1]]) + assert first.read_output.call_args_list == [call(data[0], cameras[0]), call(data[2], cameras[2])] + second.read_output.assert_called_once_with(data[1], cameras[1]) + for renderer in (first, second): + renderer.render.assert_not_called() + renderer.update_transforms.assert_called_once_with() + renderer.update_geometries.assert_called_once_with() + + def test_legacy_render_profile_scope_warns_and_preserves_import(): """The old scope import remains available during its deprecation period.""" with pytest.warns(DeprecationWarning, match="isaaclab.benchmark.stepping.RENDER_PROFILE_SCOPE"): @@ -236,3 +277,339 @@ def test_context_close_only_releases_writers_and_resets_bookkeeping(sim, fail_wr context.update_scene_state(1) assert renderer.initialize.call_count == renderer.prepare_stage.call_count == 2 assert renderer.update_transforms.call_count == renderer.update_geometries.call_count == 2 + + +class _CpuCamera(Camera): + """Exercise capture timing with CPU buffers and an in-memory pose source.""" + + def __init__(self, renderer, name, update_period=0.0): + self.cfg = SimpleNamespace(update_period=update_period, update_latest_camera_pose=True) + self._device = "cpu" + self._num_envs = 2 + self._is_initialized = True + self._is_visualizing = False + self._renderer = renderer + self._render_data = SimpleNamespace(name=name, pose=None) + self._data = CameraData() + self._data.create_buffers(2, "cpu") + self._data.info = {} + self._frame = ProxyArray(wp.zeros(2, dtype=wp.int64, device="cpu")) + self._ALL_INDICES = wp.array([0, 1], dtype=wp.int32, device="cpu") + self._ALL_ENV_MASK = wp.ones(2, dtype=wp.bool, device="cpu") + self._is_outdated = wp.ones(2, dtype=wp.bool, device="cpu") + self._timestamp = wp.zeros(2, device="cpu") + self._timestamp_last_update = wp.zeros(2, device="cpu") + self._data_generation = 0 + self._data_generation_last_update = -1 + self.pose = 0.0 + self._view = SimpleNamespace(count=2, xform_world_space_writer=self._pose_writer) + self.update(0.0) + + def __del__(self): + pass + + @contextlib.contextmanager + def _pose_writer(self): + def set_poses(positions, orientations, indices): + self.pose = float(positions.numpy()[0, 0]) + + yield SimpleNamespace(set_poses=set_poses) + + def _update_poses(self, env_ids=None, env_mask=None, frame_op=0): + self._render_data.pose = self.pose + self._update_camera_state(env_ids=env_ids, env_mask=env_mask, frame_op=frame_op) + + +class _CpuSensor(SensorBase): + """Exercise generic sensor updates without a rendering backend.""" + + def __init__(self, name, batches=None): + self.cfg = SimpleNamespace(update_period=0.0) + self._device = "cpu" + self._num_envs = 2 + self._is_initialized = True + self._is_visualizing = False + self._is_outdated = wp.ones(2, dtype=wp.bool, device="cpu") + self._timestamp = wp.zeros(2, device="cpu") + self._timestamp_last_update = wp.zeros(2, device="cpu") + self._data_generation = 0 + self._data_generation_last_update = -1 + self._data = np.zeros(2, dtype=int) + self.name = name + self.batches = batches + self.captures = [] + + def __del__(self): + pass + + @property + def data(self): + self._update_outdated_buffers() + return self._data + + def _initialize_impl(self): + pass + + def _update_buffers_impl(self, env_mask): + mask = env_mask.numpy() + self.captures.append(mask) + self._data[mask] += 1 + + +class _BatchSensor(_CpuSensor): + @property + def supports_batch_update(self): + return True + + @staticmethod + def _update_buffers_batch_impl(sensors): + sensors[0].batches.append([sensor.name for sensor in sensors]) + for sensor in sensors: + sensor._update_buffers_impl(sensor._is_outdated) + + +@pytest.fixture +def camera_batch_context(sim, monkeypatch): + ctx = sim.render_context + sim._physics_step_count = 1 + monkeypatch.setattr(SimulationContext, "_instance", sim) + renderer = sim.get_or_create_backend(NewtonWarpRendererCfg(class_type=_renderer)) + batches = [] + renderer.render.side_effect = lambda rd: batches.append([(rd.name, rd.pose)]) + renderer.render_batch = Mock(side_effect=lambda requests: batches.append([(rd.name, rd.pose) for rd in requests])) + renderer.read_output.side_effect = lambda rd, data: data.info.update(pose=rd.pose) + return ctx, renderer, batches + + +def _camera_scene(cameras, lazy=False, **sensors): + return SimpleNamespace( + sim=SimulationContext.instance(), + cfg=SimpleNamespace(lazy_sensor_update=lazy), + _sensors={**{camera._render_data.name: camera for camera in cameras}, **sensors}, + **{ + name: {} + for name in ( + "_articulations", + "_cable_objects", + "_deformable_objects", + "_rigid_objects", + "_rigid_object_collections", + "_surface_grippers", + ) + }, + ) + + +def test_camera_eager_updates_render_shared_batch_with_current_poses(camera_batch_context): + """Eager scene updates batch initialized cameras and still update other sensor types.""" + _, renderer, batches = camera_batch_context + cameras = [_CpuCamera(renderer, "wide"), _CpuCamera(renderer, "tele")] + cameras[0].pose, cameras[1].pose = 1.0, 2.0 + uninitialized = _CpuCamera(renderer, "uninitialized") + uninitialized._is_initialized = False + other_sensor = Mock(supports_batch_update=False) + scene = _camera_scene([*cameras, uninitialized], other=other_sensor) + + InteractiveScene.update(scene, 0.01) + assert batches == [[("wide", 1.0), ("tele", 2.0)]] + renderer.render.assert_not_called() + other_sensor.update.assert_called_once_with(0.01, force_recompute=True) + assert cameras[0].data.info["pose"] == 1.0 + assert cameras[1].data.info["pose"] == 2.0 + assert batches == [[("wide", 1.0), ("tele", 2.0)]] + for camera in cameras: + np.testing.assert_array_equal(camera.frame.warp.numpy(), [1, 1]) + np.testing.assert_allclose(camera._timestamp_last_update.numpy(), [0.01, 0.01]) + np.testing.assert_array_equal(uninitialized.frame.warp.numpy(), [0, 0]) + + +def test_camera_lazy_reads_leave_peer_cameras_outdated(camera_batch_context): + """Reading a lazy camera captures only that camera, even when a peer shares its renderer.""" + _, renderer, batches = camera_batch_context + cameras = [_CpuCamera(renderer, "wide"), _CpuCamera(renderer, "tele")] + cameras[0].pose, cameras[1].pose = 1.0, 2.0 + other_sensor = Mock(supports_batch_update=False) + scene = _camera_scene(cameras, lazy=True, other=other_sensor) + + InteractiveScene.update(scene, 0.01) + assert not batches + other_sensor.update.assert_called_once_with(0.01, force_recompute=False) + assert cameras[0].data.info["pose"] == 1.0 + assert batches == [[("wide", 1.0)]] + np.testing.assert_array_equal(cameras[1].frame.warp.numpy(), [0, 0]) + np.testing.assert_array_equal(cameras[1]._timestamp_last_update.numpy(), [0.0, 0.0]) + + assert cameras[1].data.info["pose"] == 2.0 + for camera in cameras: + assert camera.data.info["pose"] == camera.pose + np.testing.assert_array_equal(camera.frame.warp.numpy(), [1, 1]) + np.testing.assert_allclose(camera._timestamp_last_update.numpy(), [0.01, 0.01]) + assert batches == [[("wide", 1.0)], [("tele", 2.0)]] + renderer.render.assert_not_called() + + +def test_eager_scene_batches_multiple_sensor_families(camera_batch_context): + """Sensor families batch together unless an earlier update already refreshed their data.""" + _, renderer, camera_batches = camera_batch_context + sensor_batches = [] + + class InheritedSensor(_BatchSensor): + pass + + class OtherSensor(_BatchSensor): + @staticmethod + def _update_buffers_batch_impl(sensors): + _BatchSensor._update_buffers_batch_impl(sensors) + + class ObserverSensor(_CpuSensor): + def _update_buffers_impl(self, env_mask): + np.testing.assert_array_equal(observed.data, [1, 1]) + super()._update_buffers_impl(env_mask) + + observed = _BatchSensor("observed", sensor_batches) + observer = ObserverSensor("observer") + first = _BatchSensor("first", sensor_batches) + second = InheritedSensor("second", sensor_batches) + other = OtherSensor("other", sensor_batches) + cameras = [_CpuCamera(renderer, "wide"), _CpuCamera(renderer, "tele")] + scene = _camera_scene(cameras, observed=observed, observer=observer, first=first, other=other, second=second) + + InteractiveScene.update(scene, 0.25) + + assert camera_batches == [[("wide", 0.0), ("tele", 0.0)]] + assert sensor_batches == [["first", "second"], ["other"]] + for sensor in (observed, observer, first, second, other): + np.testing.assert_array_equal(sensor.data, [1, 1]) + np.testing.assert_allclose(sensor._timestamp.numpy(), [0.25, 0.25]) + np.testing.assert_allclose(sensor._timestamp_last_update.numpy(), [0.25, 0.25]) + assert len(sensor.captures) == 1 + + +def test_sensor_batch_advances_time_once_and_preserves_update_hooks(): + """Batch updates advance sensors in order and preserve visualization refreshes.""" + updates = [] + + class DefaultBatchSensor(_CpuSensor): + @property + def supports_batch_update(self): + return True + + def update(self, dt, force_recompute=False): + updates.append((self.name, dt, force_recompute)) + super().update(dt, force_recompute=force_recompute) + + batched = DefaultBatchSensor("batched") + visualized = DefaultBatchSensor("visualized") + visualized._is_visualizing = True + uninitialized = DefaultBatchSensor("uninitialized") + uninitialized._is_initialized = False + sensors = [batched, visualized, uninitialized] + + SensorBase.update_batch([], 0.25) + SensorBase.update_batch(sensors, 0.25) + + assert updates == [ + ("batched", 0.25, False), + ("visualized", 0.25, False), + ("uninitialized", 0.25, False), + ] + for sensor in (batched, visualized): + np.testing.assert_array_equal(sensor.data, [1, 1]) + np.testing.assert_allclose(sensor._timestamp.numpy(), [0.25, 0.25]) + np.testing.assert_allclose(sensor._timestamp_last_update.numpy(), [0.25, 0.25]) + assert len(sensor.captures) == 1 + assert not uninitialized.captures + np.testing.assert_array_equal(uninitialized._timestamp_last_update.numpy(), [0.0, 0.0]) + + +def test_sensor_batch_rejects_unsupported_input_before_updating(): + """Invalid batches leave all sensors untouched, including preceding supported sensors.""" + batches = [] + sensors = [_BatchSensor("supported", batches), _CpuSensor("unsupported")] + + with pytest.raises(ValueError, match="supports_batch_update"): + SensorBase.update_batch(sensors, 0.25) + + assert not batches + for sensor in sensors: + assert not sensor.captures + np.testing.assert_array_equal(sensor._timestamp.numpy(), [0.0, 0.0]) + + +@pytest.mark.parametrize("lazy", [False, True]) +@pytest.mark.parametrize("hook", ["_update_buffers_impl", "_update_outdated_buffers"]) +def test_camera_updates_preserve_custom_buffer_hooks(camera_batch_context, monkeypatch, lazy, hook): + """Custom camera capture hooks run in both eager and lazy scenes.""" + _, renderer, _ = camera_batch_context + custom_updates = Mock() + + class CustomCamera(_CpuCamera): + pass + + def custom_update(self, *args, **kwargs): + custom_updates() + return getattr(Camera, hook)(self, *args, **kwargs) + + monkeypatch.setattr(CustomCamera, hook, custom_update) + custom = CustomCamera(renderer, "custom") + peer = _CpuCamera(renderer, "peer") + InteractiveScene.update(_camera_scene([custom, peer], lazy=lazy), 0.01) + assert custom_updates.call_count == (0 if lazy else 1) + + assert custom.data.info["pose"] == 0.0 + renderer.render.assert_not_called() + if lazy: + custom_updates.assert_called_once_with() + renderer.render_batch.assert_called_once_with([custom._render_data]) + else: + assert renderer.render_batch.call_args_list == [call([custom._render_data]), call([peer._render_data])] + + +def test_camera_batch_respects_period_partial_reset_and_updated_pose(camera_batch_context): + """A fresh peer stays cached while due or reset cameras capture their current poses.""" + _, renderer, batches = camera_batch_context + fast = _CpuCamera(renderer, "fast") + slow = _CpuCamera(renderer, "slow", update_period=1.0) + scene = _camera_scene([fast, slow]) + + InteractiveScene.update(scene, 0.1) + InteractiveScene.update(scene, 0.2) + assert batches == [[("fast", 0.0), ("slow", 0.0)], [("fast", 0.0)]] + np.testing.assert_array_equal(slow.frame.warp.numpy(), [1, 1]) + np.testing.assert_allclose(slow._timestamp_last_update.numpy(), [0.1, 0.1]) + + slow.pose = 3.0 + slow.reset(env_mask=wp.array([False, True], dtype=wp.bool, device="cpu")) + InteractiveScene.update(scene, 0.1) + assert batches[-1] == [("fast", 0.0), ("slow", 3.0)] + assert slow.data.info["pose"] == 3.0 + np.testing.assert_array_equal(slow.frame.warp.numpy(), [1, 1]) + np.testing.assert_allclose(slow._timestamp.numpy(), [0.4, 0.1]) + np.testing.assert_allclose(slow._timestamp_last_update.numpy(), [0.1, 0.1]) + + InteractiveScene.update(scene, 0.9) + np.testing.assert_array_equal(slow.frame.warp.numpy(), [2, 1]) + np.testing.assert_allclose(slow._timestamp_last_update.numpy(), [1.3, 0.1]) + renderer.render.assert_not_called() + + +@pytest.mark.parametrize("failure", ["render_batch", "read_output"]) +def test_failed_eager_capture_can_retry_one_camera(camera_batch_context, failure): + """Failed captures leave timestamps outdated, and lazy retries do not capture peer cameras.""" + _, renderer, batches = camera_batch_context + cameras = [_CpuCamera(renderer, "wide"), _CpuCamera(renderer, "tele")] + scene = _camera_scene(cameras) + original = getattr(renderer, failure) + + setattr(renderer, failure, Mock(side_effect=RuntimeError("capture failed"))) + with pytest.raises(RuntimeError, match="capture failed"): + InteractiveScene.update(scene, 0.1) + for camera in cameras: + np.testing.assert_array_equal(camera._timestamp_last_update.numpy(), [0.0, 0.0]) + setattr(renderer, failure, original) + batches.clear() + + assert cameras[0].data.info["pose"] == 0.0 + assert batches == [[("wide", 0.0)]] + np.testing.assert_allclose(cameras[0]._timestamp_last_update.numpy(), [0.1, 0.1]) + np.testing.assert_array_equal(cameras[1]._timestamp_last_update.numpy(), [0.0, 0.0]) diff --git a/source/isaaclab_ov/changelog.d/ovrtx-render-batch.minor.rst b/source/isaaclab_ov/changelog.d/ovrtx-render-batch.minor.rst new file mode 100644 index 000000000000..b7bbdd0e30e7 --- /dev/null +++ b/source/isaaclab_ov/changelog.d/ovrtx-render-batch.minor.rst @@ -0,0 +1,5 @@ +Added +^^^^^ + +* Added an OVRTX ``render_batch()`` implementation that submitted all requested camera + products in one native renderer step while preserving the single-camera ``render()`` interface. diff --git a/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py b/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py index 729bac191a22..9bd241ecb3d2 100644 --- a/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py +++ b/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py @@ -106,7 +106,7 @@ if TYPE_CHECKING: from isaaclab_ppisp import PpispPipeline - from ovrtx import AttributeBinding + from ovrtx import AttributeBinding, RenderProductSetOutputs from isaaclab.renderers.base_renderer import VisualMaterialBatch from isaaclab.sensors.camera.camera_data import CameraData @@ -1606,8 +1606,8 @@ def _process_render_frame(self, render_data: OVRTXCameraRenderData, frame, outpu with self._map_render_var_to_dlpack(motion_var) as tiled_motion_vectors_data: self._launch_extract_all_tiles(render_data, tiled_motion_vectors_data, output_buffers["motion_vectors"]) - def _render_legacy(self, render_data: OVRTXCameraRenderData) -> None: - """Render the scene into the provided RenderData.""" + def _render_legacy(self, render_data: Sequence[OVRTXCameraRenderData]) -> None: + """Render the requested camera products in one native submission.""" if not self._initialized_scene: raise RuntimeError("Scene not initialized. Call initialize() first.") if self.backend.renderer is None or len(self._render_product_paths) == 0: @@ -1617,7 +1617,7 @@ def _render_legacy(self, render_data: OVRTXCameraRenderData) -> None: if material_writer is not None: material_writer.publish() products = self.backend.renderer.step( - render_products={render_data.render_product_path}, + render_products={data.render_product_path for data in render_data}, delta_time=1.0 / 60.0, ) finally: @@ -1625,21 +1625,28 @@ def _render_legacy(self, render_data: OVRTXCameraRenderData) -> None: drain_errors = contextlib.nullcontext() if sys.exc_info()[0] is None else contextlib.suppress(Exception) with drain_errors: material_writer.drain() - product_path = render_data.render_product_path - if product_path in products and len(products[product_path].frames) > 0: + self._process_render_products(render_data, products) + + def _process_render_products( + self, render_data: Sequence[OVRTXCameraRenderData], products: RenderProductSetOutputs + ) -> None: + """Populate camera outputs only after every requested product returned a frame.""" + for data in render_data: + if data.render_product_path not in products or not products[data.render_product_path].frames: + raise RuntimeError(f"OVRTX returned no frame for render product {data.render_product_path!r}.") + for data in render_data: self._process_render_frame( - render_data, - products[product_path].frames[0], - render_data.warp_buffers, + data, + products[data.render_product_path].frames[0], + data.warp_buffers, ) - # Post-render PPISP: HDR scene-linear → LDR RGBA. Source/destination - # buffers are the same warp buffer map used by extraction. - if render_data.ppisp_pipeline is not None: - render_data.ppisp_pipeline.apply( - render_data.warp_buffers[str(RenderBufferKind.RGB_HDR)], - render_data.warp_buffers[str(RenderBufferKind.RGBA)], - ) + # Post-render PPISP uses each camera's own HDR source and RGBA destination. + if data.ppisp_pipeline is not None: + data.ppisp_pipeline.apply( + data.warp_buffers[str(RenderBufferKind.RGB_HDR)], + data.warp_buffers[str(RenderBufferKind.RGBA)], + ) def _close_legacy(self) -> None: """Release the renderer's tensor bindings. See :meth:`close`.""" @@ -1801,7 +1808,21 @@ def update_camera_intrinsics(self, render_data: OVRTXCameraRenderData, intrinsic operation.wait() def render(self, render_data: OVRTXCameraRenderData) -> None: - """Render the scene into the provided RenderData.""" + """Render one camera product into its bound output buffers.""" + self.render_batch((render_data,)) + + def render_batch(self, render_data: Sequence[OVRTXCameraRenderData]) -> None: + """Render all requested camera products in one native submission. + + Args: + render_data: Cameras whose poses and output buffers have been prepared. An empty + sequence performs no work. + + Raises: + RuntimeError: If the scene is uninitialized or a requested product returns no frame. + """ + if not render_data: + return if self._use_ovstage: self._render_ovstage(render_data) else: @@ -2424,7 +2445,7 @@ def _update_camera_ovstage( cuda_stream=self._warp_device.stream.cuda_stream, ).wait() - def _render_ovstage(self, render_data: OVRTXCameraRenderData) -> None: + def _render_ovstage(self, render_data: Sequence[OVRTXCameraRenderData]) -> None: if not self._initialized_scene: raise RuntimeError("Scene not initialized. Call initialize() first.") if self.backend.renderer is None or len(self._render_product_paths) == 0: @@ -2442,26 +2463,12 @@ def _render_ovstage(self, render_data: OVRTXCameraRenderData) -> None: with drain_errors: material_writer.drain() products = self.backend.renderer.step( - render_products={render_data.render_product_path}, + render_products={data.render_product_path for data in render_data}, delta_time=1.0 / 60.0, ordinal=self._current_ordinal, ) self._current_ordinal += 1 - product_path = render_data.render_product_path - if product_path in products and len(products[product_path].frames) > 0: - self._process_render_frame( - render_data, - products[product_path].frames[0], - render_data.warp_buffers, - ) - - # Post-render PPISP: HDR scene-linear → LDR RGBA. Source/destination - # buffers are the same warp buffer map used by extraction. - if render_data.ppisp_pipeline is not None: - render_data.ppisp_pipeline.apply( - render_data.warp_buffers[str(RenderBufferKind.RGB_HDR)], - render_data.warp_buffers[str(RenderBufferKind.RGBA)], - ) + self._process_render_products(render_data, products) def _close_ovstage(self) -> None: """Release the renderer's stage queries and path lists. See :meth:`close`.""" diff --git a/source/isaaclab_ov/test/test_ovrtx_renderer_contract.py b/source/isaaclab_ov/test/test_ovrtx_renderer_contract.py index 22339172a3a3..47888b2c7cb2 100644 --- a/source/isaaclab_ov/test/test_ovrtx_renderer_contract.py +++ b/source/isaaclab_ov/test/test_ovrtx_renderer_contract.py @@ -164,6 +164,79 @@ def test_ovrtx_supported_output_types_key_set(): assert specs[RenderBufferKind.MOTION_VECTORS] == RenderBufferSpec(2, wp.float32) +@pytest.mark.parametrize("use_ovstage", [False, True]) +@pytest.mark.parametrize("missing_output", [None, "product", "frame"]) +@pytest.mark.parametrize("batch", [False, True]) +def test_ovrtx_render_submits_requested_products_and_routes_outputs(monkeypatch, use_ovstage, missing_output, batch): + """A submission fills each camera from its product and rejects incomplete results.""" + renderer = _make_ovrtx_renderer_without_backend() + renderer._use_ovstage = use_ovstage + renderer._initialized_scene = True + renderer._visual_material_writer_ref = None + renderer._current_ordinal = 7 + cameras = [_make_ovrtx_camera_render_data() for _ in range(2 if batch else 1)] + products = {} + processed = [] + postprocessed = [] + submissions = [] + published_ordinals = [] + for index, camera in enumerate(cameras): + camera.render_product_path = f"/Render/Camera{index}" + camera.warp_buffers = {str(RenderBufferKind.RGB_HDR): object(), str(RenderBufferKind.RGBA): object()} + camera.ppisp_pipeline = types.SimpleNamespace(apply=lambda *buffers: postprocessed.append(buffers)) + products[camera.render_product_path] = types.SimpleNamespace(frames=[object()]) + renderer._render_product_paths = [*products, "/Render/UnrequestedCamera"] + if missing_output == "product": + del products[cameras[-1].render_product_path] + elif missing_output == "frame": + products[cameras[-1].render_product_path].frames.clear() + + def step(**kwargs): + submissions.append(kwargs) + return products + + def advance_write_floor(*, ordinal): + published_ordinals.append(ordinal) + return types.SimpleNamespace(wait=lambda: None) + + renderer.backend.renderer = types.SimpleNamespace(step=step) + renderer.backend.stage = types.SimpleNamespace(advance_write_floor=advance_write_floor) + monkeypatch.setattr(renderer, "_process_render_frame", lambda *args: processed.append(args)) + + render = renderer.render_batch if batch else renderer.render + request = cameras if batch else cameras[0] + if missing_output is None: + render(request) + assert processed == [ + (camera, products[camera.render_product_path].frames[0], camera.warp_buffers) for camera in cameras + ] + assert postprocessed == [ + (camera.warp_buffers[str(RenderBufferKind.RGB_HDR)], camera.warp_buffers[str(RenderBufferKind.RGBA)]) + for camera in cameras + ] + else: + with pytest.raises(RuntimeError, match=cameras[-1].render_product_path): + render(request) + assert not processed + assert not postprocessed + + assert len(submissions) == 1 + assert submissions[0]["render_products"] == {camera.render_product_path for camera in cameras} + if use_ovstage: + assert submissions[0]["ordinal"] == 7 + assert published_ordinals == [7] + assert renderer._current_ordinal == 8 + else: + assert "ordinal" not in submissions[0] + assert not published_ordinals + + +def test_ovrtx_render_batch_empty_sequence_does_not_require_initialized_backend(): + """An empty render request has no backend work or initialization precondition.""" + renderer = OVRTXRenderer.__new__(OVRTXRenderer) + renderer.render_batch([]) + + @pytest.mark.integration @pytest.mark.rendering @pytest.mark.parametrize("use_ovstage", [False, True]) @@ -228,7 +301,6 @@ def camera_scope_exists(rd): return any(path.startswith(scope) for path in renderer.backend.renderer.query_prims()) def check_depth(rd, data, expected, label): - renderer.render(rd) depth = data.output["distance_to_image_plane"].torch for env_id in range(rd.num_envs): prefix = tmp_path / f"{label}_env{env_id}" @@ -272,6 +344,7 @@ def check_depth(rd, data, expected, label): renderer.set_outputs(rd, data.output) cameras.append((rd, data)) # Register the next camera after rendering has already started. + renderer.render(rd) check_depth(rd, data, 5.0 - index - 0.5, f"initial_cam{index}") if index == 1: normals = data.output["normals"].torch[:, height // 2, width // 2, :3] @@ -286,11 +359,13 @@ def check_depth(rd, data, expected, label): ) orientations = ProxyArray(wp.from_torch(quats, dtype=wp.quatf)) renderer.update_camera(cameras[1][0], positions, orientations, cameras[1][1].intrinsic_matrices) + renderer.render_batch([rd for rd, _ in cameras]) check_depth(*cameras[0], 4.5, "after_move_cam0") check_depth(*cameras[1], 5.5, "after_move_cam1") assert all(camera_scope_exists(rd) for rd, _ in cameras) renderer.cleanup(cameras[0][0]) assert not camera_scope_exists(cameras[0][0]) + renderer.render(cameras[1][0]) check_depth(*cameras[1], 5.5, "after_cleanup_cam1") renderer.cleanup(cameras[1][0]) renderer.cleanup(cameras[1][0]) diff --git a/source/isaaclab_ov/test/test_ovrtx_visual_material_sync.py b/source/isaaclab_ov/test/test_ovrtx_visual_material_sync.py index a7b502350e6d..80dc51074e5a 100644 --- a/source/isaaclab_ov/test/test_ovrtx_visual_material_sync.py +++ b/source/isaaclab_ov/test/test_ovrtx_visual_material_sync.py @@ -51,7 +51,7 @@ def bind_attribute(self, prim_paths, attribute_name, **kwargs): def step(self, **kwargs): self.events.append("step") - return {} + return {path: SimpleNamespace(frames=[object()]) for path in kwargs["render_products"]} class _NativeBinding: @@ -205,7 +205,8 @@ def test_ovstage_compiles_queries_and_publishes_selected_channel_zero_copy(): ) def test_render_publishes_and_drains_material_writes_at_backend_boundary(use_ovstage, expected_events): renderer, events = _renderer(use_ovstage=use_ovstage) - renderer._render_product_paths = ["/RenderCamera_0/Product"] + renderer._render_product_paths = ["/Render/Product0", "/Render/Product1"] + renderer._process_render_frame = lambda *args: None class Writer: def publish(self): @@ -216,8 +217,12 @@ def drain(self): writer = Writer() renderer._visual_material_writer_ref = lambda: writer - render = renderer._render_ovstage if use_ovstage else renderer._render_legacy - render(SimpleNamespace(render_product_path="/RenderCamera_0/Product", ppisp_pipeline=None)) + renderer.render_batch( + [ + SimpleNamespace(render_product_path=path, ppisp_pipeline=None, warp_buffers={}) + for path in renderer._render_product_paths + ] + ) assert events == expected_events @@ -249,7 +254,7 @@ def advance_write_floor(**_kwargs): renderer.backend.stage.advance_write_floor = advance_write_floor with pytest.raises(ValueError, match=failure): - renderer._render_ovstage(SimpleNamespace(render_product_path="/RenderCamera_0/Product", ppisp_pipeline=None)) + renderer.render(SimpleNamespace(render_product_path="/RenderCamera_0/Product", ppisp_pipeline=None)) assert events == expected_events diff --git a/source/isaaclab_physx/changelog.d/isaac-rtx-render-batch.rst b/source/isaaclab_physx/changelog.d/isaac-rtx-render-batch.rst new file mode 100644 index 000000000000..2c604eaa95e1 --- /dev/null +++ b/source/isaaclab_physx/changelog.d/isaac-rtx-render-batch.rst @@ -0,0 +1,5 @@ +Fixed +^^^^^ + +* Avoided repeated Isaac RTX render-update checks by checking once per camera batch before + extracting each camera's annotator outputs. diff --git a/source/isaaclab_physx/isaaclab_physx/renderers/isaac_rtx_renderer.py b/source/isaaclab_physx/isaaclab_physx/renderers/isaac_rtx_renderer.py index 011e0b55f093..de60212345bb 100644 --- a/source/isaaclab_physx/isaaclab_physx/renderers/isaac_rtx_renderer.py +++ b/source/isaaclab_physx/isaaclab_physx/renderers/isaac_rtx_renderer.py @@ -11,6 +11,7 @@ import logging import math import uuid +from collections.abc import Sequence from dataclasses import dataclass, field from typing import TYPE_CHECKING, Any, NoReturn @@ -612,12 +613,19 @@ def update_camera_intrinsics(self, render_data: IsaacRtxRenderData, intrinsics: device=parameters.device, ) - def render(self, render_data: IsaacRtxRenderData): - """Extract data from annotators and write to output buffers. - See :meth:`~isaaclab.renderers.base_renderer.BaseRenderer.render`.""" - spec = render_data.spec - output_data = render_data.output_data - if output_data is None or spec is None: + def render(self, render_data: IsaacRtxRenderData) -> None: + """Render one camera product into its bound output buffers.""" + self.render_batch((render_data,)) + + def render_batch(self, render_data: Sequence[IsaacRtxRenderData]) -> None: + """Ensure a shared RTX update once, then extract each camera's annotator outputs. + + Args: + render_data: Cameras whose poses and output buffers have been prepared. Entries + without a spec or output buffers are skipped. An empty sequence performs no work. + """ + cameras = [data for data in render_data if data.output_data is not None and data.spec is not None] + if not cameras: return # Ensure the RTX renderer has been pumped so annotator buffers are fresh. @@ -625,6 +633,15 @@ def render(self, render_data: IsaacRtxRenderData): # for the current physics step, or if a visualizer already pumped it. ensure_isaac_rtx_render_update() + for data in cameras: + self._read_annotator_output(data) + + def _read_annotator_output(self, render_data: IsaacRtxRenderData) -> None: + """Extract one camera's annotator data into its bound output buffers.""" + spec = render_data.spec + output_data = render_data.output_data + assert output_data is not None and spec is not None + view_count = spec.view_count cfg = spec.cfg device = spec.device diff --git a/source/isaaclab_physx/test/renderers/test_isaac_rtx_renderer_contract.py b/source/isaaclab_physx/test/renderers/test_isaac_rtx_renderer_contract.py index 34c05e227800..5f05e23f8420 100644 --- a/source/isaaclab_physx/test/renderers/test_isaac_rtx_renderer_contract.py +++ b/source/isaaclab_physx/test/renderers/test_isaac_rtx_renderer_contract.py @@ -425,7 +425,8 @@ def test_deterministic_flag_gates_rtx_determinism_settings(monkeypatch, stored, @pytest.mark.parametrize("data_type", ["rgba", "normals"]) -def test_render_treats_empty_annotator_frame_as_not_ready(monkeypatch, data_type): +@pytest.mark.parametrize("batched", [False, True], ids=["single", "batch"]) +def test_render_treats_empty_annotator_frame_as_not_ready(monkeypatch, data_type, batched): """An empty warm-up frame should clear its output without slicing or launching a reshape.""" _install_omni_stubs(monkeypatch) import isaaclab_physx.renderers.isaac_rtx_renderer as rtx_renderer @@ -450,15 +451,71 @@ def test_render_treats_empty_annotator_frame_as_not_ready(monkeypatch, data_type renderer.cfg = IsaacRtxRendererCfg() with ( - patch.object(rtx_renderer, "ensure_isaac_rtx_render_update"), + patch.object(rtx_renderer, "ensure_isaac_rtx_render_update") as update, patch.object(rtx_renderer.wp, "launch") as launch, ): - renderer.render(render_data) + if batched: + renderer.render_batch([render_data]) + else: + renderer.render(render_data) + update.assert_called_once_with() output_buffer.zero_.assert_called_once_with() launch.assert_not_called() +@pytest.mark.parametrize( + "states", + [ + pytest.param((), id="empty"), + pytest.param(("no_spec", "no_output"), id="uninitialized"), + pytest.param(("ready", "ready"), id="ready"), + pytest.param(("no_spec", "ready", "no_output", "ready"), id="mixed"), + ], +) +def test_render_batch_updates_once_before_extracting_ready_cameras(monkeypatch, states): + """Ready cameras share one RTX update and receive their own annotator pixels.""" + _install_omni_stubs(monkeypatch) + import isaaclab_physx.renderers.isaac_rtx_renderer as rtx_renderer + from isaaclab_physx.renderers.isaac_rtx_renderer_cfg import IsaacRtxRendererCfg + + renderer = rtx_renderer.IsaacRtxRenderer.__new__(rtx_renderer.IsaacRtxRenderer) + renderer.cfg = IsaacRtxRendererCfg() + operations = MagicMock() + render_data_list = [] + expected_calls = [call.update()] if "ready" in states else [] + expected_outputs = [] + for index, state in enumerate(states): + annotator = getattr(operations, f"camera_{index}") + frame = np.full((1, 1, 4), index + 1, dtype=np.uint8) + annotator.get_data.return_value = frame + output_buffer = wp.zeros((1, 1, 1, 4), dtype=wp.uint8, device="cpu") + render_data_list.append( + SimpleNamespace( + annotators={"rgba": annotator}, + output_data=None if state == "no_output" else {"rgba": SimpleNamespace(warp=output_buffer)}, + spec=( + None + if state == "no_spec" + else SimpleNamespace(view_count=1, device="cpu", cfg=SimpleNamespace(width=1, height=1)) + ), + renderer_info={}, + ppisp_pipeline=None, + _hdr_scratch_wp=None, + ) + ) + if state == "ready": + expected_calls.append(getattr(call, f"camera_{index}").get_data()) + expected_outputs.append((output_buffer, frame[np.newaxis])) + + with patch.object(rtx_renderer, "ensure_isaac_rtx_render_update", operations.update): + renderer.render_batch(render_data_list) + + assert operations.mock_calls == expected_calls + for output_buffer, expected in expected_outputs: + np.testing.assert_array_equal(output_buffer.numpy(), expected) + + def test_isaac_rtx_read_output_clears_stale_metadata_and_keeps_seeded_keys(monkeypatch): """read_output replaces (not merges): a dropped annotator info resets its info entry, seeded keys persist.""" _install_omni_stubs(monkeypatch) From f480362385b6a4db6b7dbf8f58f5e6790240d75b Mon Sep 17 00:00:00 2001 From: Mustafa Haiderbhai Date: Thu, 24 Sep 2026 09:14:35 -0700 Subject: [PATCH 3/8] Format merged OVRTX contract tests --- source/isaaclab_ov/test/test_ovrtx_renderer_contract.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/source/isaaclab_ov/test/test_ovrtx_renderer_contract.py b/source/isaaclab_ov/test/test_ovrtx_renderer_contract.py index 47888b2c7cb2..e6656b185220 100644 --- a/source/isaaclab_ov/test/test_ovrtx_renderer_contract.py +++ b/source/isaaclab_ov/test/test_ovrtx_renderer_contract.py @@ -882,6 +882,8 @@ def test_ovrtx_cleanup_without_render_data_keeps_renderer_state(): assert renderer._render_product_paths == ["/RenderCamera_0/RenderProduct_camera"] assert renderer._initialized_scene is True + + @pytest.mark.parametrize( "camera_path", [ From 9852926673fd0afbfb08ae8cf5de0f63c3feebbe Mon Sep 17 00:00:00 2001 From: Mustafa Haiderbhai Date: Thu, 24 Sep 2026 11:29:52 -0700 Subject: [PATCH 4/8] Backport camera source path cleanup (#7983) (cherry picked from commit ef821d927986f4d91c85399a1e882d968293a158) --- .../camera-render-spec-paths.major.rst | 6 ++ .../isaaclab/renderers/camera_render_spec.py | 9 ++- .../isaaclab/sensors/camera/camera.py | 5 -- .../changelog.d/ovrtx-camera-source-path.rst | 7 ++ .../isaaclab_ov/renderers/ovrtx_renderer.py | 71 ++++++++++--------- .../isaaclab_ov/renderers/ovrtx_usd.py | 2 +- .../isaaclab_ov/test/test_ovrtx_clone_plan.py | 3 +- .../test/test_ovrtx_deformable_bindings.py | 2 +- .../test/test_ovrtx_renderer_contract.py | 4 -- source/isaaclab_ov/test/test_ovrtx_usd.py | 1 - 10 files changed, 56 insertions(+), 54 deletions(-) create mode 100644 source/isaaclab/changelog.d/camera-render-spec-paths.major.rst create mode 100644 source/isaaclab_ov/changelog.d/ovrtx-camera-source-path.rst diff --git a/source/isaaclab/changelog.d/camera-render-spec-paths.major.rst b/source/isaaclab/changelog.d/camera-render-spec-paths.major.rst new file mode 100644 index 000000000000..dd40abb0ae92 --- /dev/null +++ b/source/isaaclab/changelog.d/camera-render-spec-paths.major.rst @@ -0,0 +1,6 @@ +Removed +^^^^^^^ + +* **Breaking:** Removed ``CameraRenderSpec.camera_path_relative_to_env_0``. Remove this argument + from render-spec constructors and use the absolute paths in ``camera_prim_paths`` instead. + OVRTX derived cloned camera paths from the authored source camera internally. diff --git a/source/isaaclab/isaaclab/renderers/camera_render_spec.py b/source/isaaclab/isaaclab/renderers/camera_render_spec.py index 9797c97baa61..f30cc2c31e7f 100644 --- a/source/isaaclab/isaaclab/renderers/camera_render_spec.py +++ b/source/isaaclab/isaaclab/renderers/camera_render_spec.py @@ -23,10 +23,10 @@ class CameraRenderSpec: cfg: Camera configuration (data types, resolution, filters, etc.). device: Torch device string (e.g. ``"cuda:0"``) used by GPU annotators and Warp. num_instances: Number of tiled camera instances (environments). - camera_prim_paths: Absolute USD paths for each environment's camera prim. - view_count: Number of camera prims (must match ``len(camera_prim_paths)``). - camera_path_relative_to_env_0: Camera prim path with ``/World/envs/env_0/`` prefix - stripped; required by OVRTX. Empty string if the first camera is not under env 0. + camera_prim_paths: Absolute paths of the authored camera prims. When the renderer + clones environments internally, this may contain only the source camera path; + the renderer resolves its logical per-environment paths during registration. + view_count: Number of logical camera instances in the sensor view. """ cfg: CameraCfg @@ -34,4 +34,3 @@ class CameraRenderSpec: num_instances: int camera_prim_paths: tuple[str, ...] view_count: int - camera_path_relative_to_env_0: str diff --git a/source/isaaclab/isaaclab/sensors/camera/camera.py b/source/isaaclab/isaaclab/sensors/camera/camera.py index 5324542ca62c..bf7eb40bdcf3 100644 --- a/source/isaaclab/isaaclab/sensors/camera/camera.py +++ b/source/isaaclab/isaaclab/sensors/camera/camera.py @@ -664,10 +664,6 @@ def _initialize_impl(self): # any renderer-side per-camera setup) and ``create_render_data`` consume # it, and the prims are already authored at this point. cam_paths = tuple(str(p.GetPath()) for p in sim_utils.find_matching_prims(self.cfg.prim_path, self.stage)) - env_0_prefix = "/World/envs/env_0/" - rel_under_env0 = ( - cam_paths[0].removeprefix(env_0_prefix) if cam_paths and cam_paths[0].startswith(env_0_prefix) else "" - ) device_str = self._device if isinstance(self._device, str) else str(self._device) render_spec = CameraRenderSpec( cfg=self.cfg, @@ -675,7 +671,6 @@ def _initialize_impl(self): num_instances=self._num_envs, camera_prim_paths=cam_paths, view_count=self._num_envs, - camera_path_relative_to_env_0=rel_under_env0, ) # Delegate per-camera USD setup to the renderer — must run **before** diff --git a/source/isaaclab_ov/changelog.d/ovrtx-camera-source-path.rst b/source/isaaclab_ov/changelog.d/ovrtx-camera-source-path.rst new file mode 100644 index 000000000000..fe9f940d34b6 --- /dev/null +++ b/source/isaaclab_ov/changelog.d/ovrtx-camera-source-path.rst @@ -0,0 +1,7 @@ +Changed +^^^^^^^ + +* Derived OVRTX cloned camera paths from the authored absolute source path in + ``CameraRenderSpec.camera_prim_paths``. Remove the ``camera_path_relative_to_env_0`` argument + when constructing render specs; OVRTX validated the source path under + ``/World/envs/env_0/`` and resolved the per-environment paths internally. diff --git a/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py b/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py index 9bd241ecb3d2..398b06002bc4 100644 --- a/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py +++ b/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py @@ -211,6 +211,30 @@ def _gpu_side_render_var_sync_enabled() -> bool: return value == "1" +def _get_cloned_camera_paths(camera_prim_path: str, num_instances: int) -> list[str]: + """Return paths for the source camera in env_0 and its clones in every other environment. + + Cloned cameras may be absent from the authored USD. OVRTX still needs one path per + environment; these can be synthesized because :meth:`OVRTXRenderer.prepare_stage` + requires environment ids ordered from zero. + + Args: + camera_prim_path: Absolute path of the source camera under ``/World/envs/env_0/``. + num_instances: Number of environments the camera is replicated into. + + Returns: + One absolute camera prim path per environment, in environment id order. + + Raises: + ValueError: If the source camera does not live under ``/World/envs/env_0/``. + """ + env_0_prefix = "/World/envs/env_0/" + camera_rel_path = camera_prim_path.removeprefix(env_0_prefix) + if not camera_prim_path.startswith(env_0_prefix) or not camera_rel_path: + raise ValueError(f"OVRTX cameras must be under {env_0_prefix}, got {camera_prim_path!r}.") + return [f"/World/envs/env_{i}/{camera_rel_path}" for i in range(num_instances)] + + def _write_file(output_dir: Path, file_name: str, content: str) -> None: """Write ``content`` to ``output_dir / file_name``. @@ -229,31 +253,6 @@ def _write_file(output_dir: Path, file_name: str, content: str) -> None: logger.info("Wrote USD file: %s", output_path) -def _env_camera_prim_paths(camera_path_relative_to_env_0: str | None, num_instances: int) -> list[str]: - """Per-env camera prim paths derived from the env 0 prototype. - - ``CameraRenderSpec.camera_prim_paths`` names only the camera prims authored on the USD stage, - which is one prototype per spawn variant whenever USD replication does not run. That is the - kitless case: the clone plan routes ``UsdReplicateContext`` only under Kit, so OvPhysx, Newton - and OVRTX each replicate the prototype themselves. OVRTX still needs one path per environment, - which is safe to synthesize because :meth:`OVRTXRenderer.prepare_stage` requires env ids - ordered from zero. - - Args: - camera_path_relative_to_env_0: Camera prim path with the ``/World/envs/env_0/`` prefix stripped. - num_instances: Number of environments the camera is replicated into. - - Returns: - One absolute camera prim path per environment, in env id order. - - Raises: - ValueError: If the camera prototype does not live under ``/World/envs/env_0/``. - """ - if not camera_path_relative_to_env_0: - raise ValueError("OVRTX cameras must be under /World/envs/env_0/.") - return [f"/World/envs/env_{i}/{camera_path_relative_to_env_0}" for i in range(num_instances)] - - def _write_combined_stage(output_dir: Path, scene_usd: str, render_product_usd: str) -> None: """Write the scene and render product prims in one debug layer, preserving scene metadata.""" from pxr import Sdf @@ -406,7 +405,7 @@ def __init__(self, cfg: OVRTXRendererCfg): self._cable_points: wp.array | None = None self._initialized_scene = False self._exported_usd_string: str | None = None - self._camera_rel_path: str | None = None + self._camera_prim_path: str | None = None self._output_id_color_buffers: dict[str, wp.array] = {} self._clone_plan: ClonePlan | None = None self._visual_material_writer_ref: weakref.ReferenceType[OVRTXVisualMaterialWriter] | None = None @@ -582,7 +581,7 @@ def _initialize_camera_render_data_from_spec_legacy( first_cam_path = spec.camera_prim_paths[0] if not first_cam_path.startswith(env_0_prefix): raise RuntimeError(f"Expected camera prim under '{env_0_prefix}', got '{first_cam_path}'") - self._camera_rel_path = spec.camera_path_relative_to_env_0 + self._camera_prim_path = first_cam_path logger.info("Injecting camera definitions...") @@ -610,7 +609,7 @@ def _initialize_camera_render_data_from_spec_legacy( render_data.resources.callback(self.backend.renderer.remove_usd, reference) logger.info("OVRTX loaded USD from string successfully") - camera_paths = _env_camera_prim_paths(self._camera_rel_path, num_envs) + camera_paths = _get_cloned_camera_paths(self._camera_prim_path, num_envs) if num_envs > 1: self._clone_sources_in_ovrtx() self._update_scene_partitions_after_clone(num_envs) @@ -691,7 +690,7 @@ def _update_scene_partitions_after_clone(self, num_envs: int): logger.info("Writing scene partitions for %d environments...", num_envs) partition_tokens = [f"env_{i}" for i in range(num_envs)] env_prim_paths = [f"/World/envs/env_{i}" for i in range(num_envs)] - camera_prim_paths = _env_camera_prim_paths(self._camera_rel_path, num_envs) + camera_prim_paths = _get_cloned_camera_paths(self._camera_prim_path, num_envs) self.backend.renderer.write_attribute( env_prim_paths, @@ -923,6 +922,7 @@ def create_render_data(self, spec: CameraRenderSpec) -> OVRTXCameraRenderData: Performs OVRTX initialization (stage export, USD load, bindings) on first call, matching the interface of Isaac RTX and Newton Warp which need no separate initialize(). """ + camera_paths = _get_cloned_camera_paths(spec.camera_prim_paths[0], spec.num_instances) # Normalize aliases such as "cuda" before comparing cameras sharing this renderer. warp_device = wp.get_device(spec.device) if self._initialized_scene and str(warp_device) != self._device: @@ -947,10 +947,9 @@ def create_render_data(self, spec: CameraRenderSpec) -> OVRTXCameraRenderData: else: self._register_camera(spec, render_data) if not self._use_ovstage: - intrinsic_prim_paths = _env_camera_prim_paths(spec.camera_path_relative_to_env_0, spec.num_instances) for name in _CAMERA_INTRINSIC_ATTRIBUTES: binding = self.backend.renderer.bind_attribute( - prim_paths=intrinsic_prim_paths, + prim_paths=camera_paths, attribute_name=name, dtype="float32", prim_mode=PrimMode.EXISTING_ONLY, @@ -967,7 +966,9 @@ def create_render_data(self, spec: CameraRenderSpec) -> OVRTXCameraRenderData: def _register_camera(self, spec: CameraRenderSpec, render_data: OVRTXCameraRenderData) -> None: """Add another tiled product and camera binding without reloading the shared scene.""" - camera_paths = _env_camera_prim_paths(spec.camera_path_relative_to_env_0, spec.num_instances) + camera_paths = _get_cloned_camera_paths(spec.camera_prim_paths[0], spec.num_instances) + if not camera_paths: + raise ValueError("OVRTX cameras must be under /World/envs/env_0/.") scope = render_data.render_scope_name product_path = render_data.render_product_path usd = build_render_product_as_string( @@ -1911,7 +1912,7 @@ def _initialize_camera_render_data_from_spec_ovstage( first_cam_path = spec.camera_prim_paths[0] if not first_cam_path.startswith(env_0_prefix): raise RuntimeError(f"Expected camera prim under '{env_0_prefix}', got '{first_cam_path}'") - self._camera_rel_path = spec.camera_path_relative_to_env_0 + self._camera_prim_path = first_cam_path logger.info("Injecting camera definitions...") @@ -1954,7 +1955,7 @@ def _initialize_camera_render_data_from_spec_ovstage( self._initialized_scene = True - camera_paths = _env_camera_prim_paths(self._camera_rel_path, num_envs) + camera_paths = _get_cloned_camera_paths(self._camera_prim_path, num_envs) # Re-author the RenderProduct's camera relationship after clone. ``stage.clone`` recreates the per-env # cameras, so the RenderProduct must be pointed at the freshly-interned camera path ids to discover every @@ -2055,7 +2056,7 @@ def _update_scene_partitions_after_clone_ovstage(self, num_envs: int): """Update scene partition attributes on cloned environments and cameras (ovstage path).""" logger.info("Writing scene partitions for %d environments...", num_envs) env_prim_paths = [f"/World/envs/env_{i}" for i in range(num_envs)] - camera_prim_paths = _env_camera_prim_paths(self._camera_rel_path, num_envs) + camera_prim_paths = _get_cloned_camera_paths(self._camera_prim_path, num_envs) # TOKEN_ID semantic tells ovstage the uint64 values are interned string tokens, not raw integers; # the renderer resolves them back to the original "env_N" strings for scene-partition lookup. token_ids = np.array([self.backend.paths.intern_token(f"env_{i}") for i in range(num_envs)], dtype=np.uint64) diff --git a/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_usd.py b/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_usd.py index b641dc047551..c5261f638ae7 100644 --- a/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_usd.py +++ b/source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_usd.py @@ -208,7 +208,7 @@ def build_render_scope_usd( if spec.cfg.isp_cfg is not None and "rgb_hdr" not in data_types: data_types.append("rgb_hdr") tiled_width, tiled_height = _tiled_resolution(spec.num_instances, spec.cfg.width, spec.cfg.height) - camera_path = f"/World/envs/env_0/{spec.camera_path_relative_to_env_0}" + camera_path = spec.camera_prim_paths[0] render_var_configs = get_render_var_configs(data_types, render_data.render_scope_name) minimal_mode = next( (_RTX_MINIMAL_MODES[data_type] for data_type in data_types if data_type in _RTX_MINIMAL_MODES), None diff --git a/source/isaaclab_ov/test/test_ovrtx_clone_plan.py b/source/isaaclab_ov/test/test_ovrtx_clone_plan.py index 948373975704..ab1ac8d061d3 100644 --- a/source/isaaclab_ov/test/test_ovrtx_clone_plan.py +++ b/source/isaaclab_ov/test/test_ovrtx_clone_plan.py @@ -112,7 +112,7 @@ def _make_ovrtx_renderer_without_backend() -> OVRTXRenderer: renderer._device = "cuda:0" # __init__'s default, replaced by create_render_data(spec) # create_render_data resolves this from the spec; tests that bypass it get the default. renderer._warp_device = SimpleNamespace(ordinal=0) - renderer._camera_rel_path = "Camera" + renderer._camera_prim_path = "/World/envs/env_0/Camera" renderer._render_product_paths = [] renderer._camera_render_data = [] renderer._next_camera_id = 0 @@ -146,7 +146,6 @@ def _make_camera_render_spec(num_envs: int = 1, device: str = "cpu") -> CameraRe num_instances=num_envs, camera_prim_paths=camera_paths, view_count=num_envs, - camera_path_relative_to_env_0="Camera", ) diff --git a/source/isaaclab_ov/test/test_ovrtx_deformable_bindings.py b/source/isaaclab_ov/test/test_ovrtx_deformable_bindings.py index 62b02d94e1e9..941767ebfae9 100644 --- a/source/isaaclab_ov/test/test_ovrtx_deformable_bindings.py +++ b/source/isaaclab_ov/test/test_ovrtx_deformable_bindings.py @@ -97,7 +97,7 @@ def _make_renderer_without_backend(device: str = "cpu") -> tuple[OVRTXRenderer, renderer.cfg = OVRTXRendererCfg() renderer.backend = SimpleNamespace() renderer._device = device - renderer._camera_rel_path = "Camera" + renderer._camera_prim_path = "/World/envs/env_0/Camera" renderer._clone_plan = None renderer.backend.renderer = _FakeOVRTXBackend() renderer._deformable_points_binding = None diff --git a/source/isaaclab_ov/test/test_ovrtx_renderer_contract.py b/source/isaaclab_ov/test/test_ovrtx_renderer_contract.py index e6656b185220..f00bd50a3a40 100644 --- a/source/isaaclab_ov/test/test_ovrtx_renderer_contract.py +++ b/source/isaaclab_ov/test/test_ovrtx_renderer_contract.py @@ -330,7 +330,6 @@ def check_depth(rd, data, expected, label): num_instances=2, camera_prim_paths=tuple(f"/World/envs/env_{i}/cam{index}" for i in range(2)), view_count=2, - camera_path_relative_to_env_0=f"cam{index}", ) rd = renderer.create_render_data(spec) data = CameraData.allocate( @@ -531,7 +530,6 @@ def fake_map(self, render_var): device="cpu", num_instances=2, camera_prim_paths=[f"/World/envs/env_{i}/cam{camera_id}" for i in range(2)], - camera_path_relative_to_env_0=f"cam{camera_id}", ) ) stage = stages[render_data.render_product_path] @@ -940,7 +938,6 @@ def test_intrinsic_updates_target_the_given_camera(monkeypatch, use_ovstage): device="cpu", num_instances=2, camera_prim_paths=camera_paths, - camera_path_relative_to_env_0=camera_paths[0].rsplit("/", 1)[1], ) ) for camera_paths in paths @@ -998,7 +995,6 @@ def test_registered_camera_expands_env_0_prototype_to_every_env(monkeypatch, use device="cpu", num_instances=3, camera_prim_paths=(f"/World/envs/env_0/{relative_path}",), - camera_path_relative_to_env_0=relative_path, ) ) expected_paths = [f"/World/envs/env_{i}/{relative_path}" for i in range(3)] diff --git a/source/isaaclab_ov/test/test_ovrtx_usd.py b/source/isaaclab_ov/test/test_ovrtx_usd.py index dbad2eb43664..bd04c7b0f50e 100644 --- a/source/isaaclab_ov/test/test_ovrtx_usd.py +++ b/source/isaaclab_ov/test/test_ovrtx_usd.py @@ -67,7 +67,6 @@ def camera_spec(): num_instances=4, camera_prim_paths=tuple(f"/World/envs/env_{i}/Robot/head_cam" for i in range(4)), view_count=4, - camera_path_relative_to_env_0="Robot/head_cam", ) From 99ac675e96e6c5de561a6276691421bea3e03d2c Mon Sep 17 00:00:00 2001 From: Mustafa Haiderbhai Date: Thu, 24 Sep 2026 11:32:10 -0700 Subject: [PATCH 5/8] Backport parallel Newton CI tests (#8006) (cherry picked from commit 94aa09f8bd73c750026060315234aee226ada7ee) --- .github/actions/run-package-tests/action.yml | 7 ++ .github/actions/run-tests/action.yml | 9 ++- .github/actions/run-tests/run_tests.sh | 6 ++ .github/workflows/build.yaml | 3 + .github/workflows/tools-tests.yml | 3 +- conftest.py | 9 +++ pyproject.toml | 2 + tools/conftest.py | 82 ++++++++++++++++---- tools/hang_dump.py | 4 +- tools/test_crash_journal.py | 36 +++++++++ uv.lock | 25 ++++++ 11 files changed, 166 insertions(+), 20 deletions(-) diff --git a/.github/actions/run-package-tests/action.yml b/.github/actions/run-package-tests/action.yml index 2f32aa5f031c..f6134bedbfe4 100644 --- a/.github/actions/run-package-tests/action.yml +++ b/.github/actions/run-package-tests/action.yml @@ -47,6 +47,12 @@ inputs: spawned by tools/conftest.py (combined with device-split selectors). default: '' required: false + pytest-workers: + description: >- + Number of pytest-xdist worker processes each per-file pytest run spawned by + tools/conftest.py splits its tests across. Empty or 1 runs every file serially. + default: '' + required: false shard-index: description: 'Zero-based shard index' default: '' @@ -333,6 +339,7 @@ runs: filter-pattern: ${{ inputs.filter-pattern }} exclude-pattern: ${{ inputs.exclude-pattern }} test-k-expr: ${{ inputs.test-k-expr }} + pytest-workers: ${{ inputs.pytest-workers }} shard-index: ${{ inputs.shard-index }} shard-count: ${{ inputs.shard-count }} curobo-only: ${{ inputs.curobo-only }} diff --git a/.github/actions/run-tests/action.yml b/.github/actions/run-tests/action.yml index 510c9e4cc033..572c81cfb038 100644 --- a/.github/actions/run-tests/action.yml +++ b/.github/actions/run-tests/action.yml @@ -49,6 +49,12 @@ inputs: can deselect parametrized cases (e.g. "not ovphysx"). default: '' required: false + pytest-workers: + description: >- + Number of pytest-xdist worker processes each per-file pytest run spawned by + tools/conftest.py splits its tests across. Empty or 1 runs every file serially. + default: '' + required: false curobo-only: description: 'Run only cuRobo and SkillGen tests (requires the cuRobo Docker image)' default: 'false' @@ -155,13 +161,14 @@ runs: TEST_NODE_IDS_KEY: ${{ inputs.test-node-ids-key }} TEST_PATH: ${{ inputs.test-path }} TEST_K_EXPR_INPUT: ${{ inputs.test-k-expr }} + PYTEST_WORKERS_INPUT: ${{ inputs.pytest-workers }} CI_MARKER_INPUT: ${{ inputs.ci-marker }} VOLUME_MOUNT_SOURCE: ${{ inputs.volume-mount-source }} WARP_CACHE_HOST_DIR: ${{ inputs.warp-cache-host-dir }} WHEELHOUSE_HOST_DIR: ${{ inputs.wheelhouse-host-dir }} WHEELHOUSE_PACKAGES: ${{ inputs.wheelhouse-packages }} run: | - bash .github/actions/run-tests/run_tests.sh "$TEST_PATH" "$RESULT_FILE" "$CONTAINER_NAME" "$IMAGE_TAG" "$REPORTS_DIR" "$PYTEST_OPTIONS" "$FILTER_PATTERN" "$EXCLUDE_PATTERN" "$CUROBO_ONLY" "$INCLUDE_FILES" "$QUARANTINED_ONLY" "$SHARD_INDEX" "$SHARD_COUNT" "$VOLUME_MOUNT_SOURCE" "$EXTRA_PIP_PACKAGES" "$TEST_NODE_IDS_FILE" "$TEST_NODE_IDS_KEY" "$WHEELHOUSE_HOST_DIR" "$WHEELHOUSE_PACKAGES" "$TEST_K_EXPR_INPUT" "$CI_MARKER_INPUT" "$STANDALONE_SCRIPT_SCOPE" "$STANDALONE_SCRIPT_VISUALIZER" "$STANDALONE_SCRIPT_RUNTIME_GROUP" "$WARP_CACHE_HOST_DIR" "$EXTRA_UV_PACKAGES" "$OVRTX_SHADER_CACHE_HOST_DIR" + bash .github/actions/run-tests/run_tests.sh "$TEST_PATH" "$RESULT_FILE" "$CONTAINER_NAME" "$IMAGE_TAG" "$REPORTS_DIR" "$PYTEST_OPTIONS" "$FILTER_PATTERN" "$EXCLUDE_PATTERN" "$CUROBO_ONLY" "$INCLUDE_FILES" "$QUARANTINED_ONLY" "$SHARD_INDEX" "$SHARD_COUNT" "$VOLUME_MOUNT_SOURCE" "$EXTRA_PIP_PACKAGES" "$TEST_NODE_IDS_FILE" "$TEST_NODE_IDS_KEY" "$WHEELHOUSE_HOST_DIR" "$WHEELHOUSE_PACKAGES" "$TEST_K_EXPR_INPUT" "$CI_MARKER_INPUT" "$STANDALONE_SCRIPT_SCOPE" "$STANDALONE_SCRIPT_VISUALIZER" "$STANDALONE_SCRIPT_RUNTIME_GROUP" "$WARP_CACHE_HOST_DIR" "$EXTRA_UV_PACKAGES" "$OVRTX_SHADER_CACHE_HOST_DIR" "$PYTEST_WORKERS_INPUT" - name: Kill container on cancellation if: cancelled() shell: bash diff --git a/.github/actions/run-tests/run_tests.sh b/.github/actions/run-tests/run_tests.sh index 655f3a3919f3..fc1fac63f037 100755 --- a/.github/actions/run-tests/run_tests.sh +++ b/.github/actions/run-tests/run_tests.sh @@ -38,6 +38,7 @@ run_tests() { local warp_cache_host_dir="${25}" local extra_uv_packages="${26}" local ovrtx_shader_cache_host_dir="${27}" + local pytest_workers="${28}" local logs_pid="" local wait_pid="" local docker_wait_file="/tmp/.docker_exit_${container_name}" @@ -194,6 +195,11 @@ run_tests() { echo "Setting per-file pytest -k expression: $test_k_expr" fi + if [ -n "$pytest_workers" ]; then + docker_env_args+=(-e "TEST_PYTEST_WORKERS=$pytest_workers") + echo "Setting TEST_PYTEST_WORKERS=$pytest_workers" + fi + if [ -n "$ci_marker" ]; then docker_env_args+=(-e "CI_MARKER=$ci_marker") echo "Setting CI_MARKER=$ci_marker" diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 3ddad1bb5f45..e4c7ba26d5aa 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -631,6 +631,9 @@ jobs: isaacsim-base-image: ${{ needs.config.outputs.isaacsim_image_name }} isaacsim-version: ${{ needs.config.outputs.isaacsim_image_tag }} filter-pattern: "isaaclab_newton" + # Split each test file across worker processes. Four fit the g7.4xlarge runner: + # each worker holds its own simulation, ~1 GB of GPU and ~3.5 GB of host memory. + pytest-workers: "4" warp-cache: restore container-name: isaac-lab-newton-test diff --git a/.github/workflows/tools-tests.yml b/.github/workflows/tools-tests.yml index 0903ebea5727..950565da45ca 100644 --- a/.github/workflows/tools-tests.yml +++ b/.github/workflows/tools-tests.yml @@ -52,8 +52,9 @@ jobs: # flaky, so they run without an Isaac Sim install or a full project sync. flaky drives the # rerun in test_crash_during_a_flaky_retry_is_blamed_on_the_retried_test; without it # installed that test skips itself rather than failing, so keep it in this list. + # pytest-xdist likewise drives test_an_xdist_run_journals_each_event_once. - name: Install test dependencies - run: bash "$GITHUB_WORKSPACE/.github/actions/_lib/with-python-package-retries.sh" python3 -m pip install pytest junitparser flaky pyyaml + run: bash "$GITHUB_WORKSPACE/.github/actions/_lib/with-python-package-retries.sh" python3 -m pip install pytest pytest-xdist junitparser flaky pyyaml - name: Run tools tests env: diff --git a/conftest.py b/conftest.py index d5bf93c8ca11..05b3721a43be 100644 --- a/conftest.py +++ b/conftest.py @@ -51,10 +51,19 @@ def _journal_write(record: dict) -> None: The per-record flush is the whole point: it puts the data in the OS page cache before the next test starts, so a process killed by a signal cannot take down verdicts it had already reported. Journaling failures are swallowed — losing debug context must never fail a run. + + Under ``pytest-xdist`` the controller receives every worker's start, report and finish, so only + it journals those; each worker journaling too would record every event twice and leave a + worker crash that xdist recovered from looking like an in-flight test. The controller never + collects, so the ``collected`` record comes from the first worker instead (every worker + collects the same items). """ path = os.environ.get(JOURNAL_ENV_VAR) if not path: return + worker = os.environ.get("PYTEST_XDIST_WORKER") + if worker and (record["event"] != "collected" or worker != "gw0"): + return try: with open(path, "a", encoding="utf-8") as handle: handle.write(json.dumps(record, separators=(",", ":")) + "\n") diff --git a/pyproject.toml b/pyproject.toml index 493b18cd78eb..8ecc273c0e98 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -126,6 +126,8 @@ importers = [ test = [ "pytest", "pytest-mock", + # Splits a test file across worker processes; see TEST_PYTEST_WORKERS in tools/conftest.py. + "pytest-xdist", "junitparser", "flaky", # numba subclasses coverage.types.Tracer at import; >=7.6.1 restores that shim diff --git a/tools/conftest.py b/tools/conftest.py index cd0c2ac9237d..ba93a2c7347c 100644 --- a/tools/conftest.py +++ b/tools/conftest.py @@ -49,12 +49,21 @@ def pytest_ignore_collect(collection_path, config): AppLauncher prints ``[ISAACLAB] AppLauncher initialization complete`` to ``sys.__stderr__`` (never suppressed) when Kit finishes initializing, and pytest -prints ``collected N items`` to stdout after collection. If neither appears +prints ``collected N items`` to stdout after collection (``N workers [M items]`` +under ``pytest-xdist``, once every worker has collected). If none appears within this deadline the process is treated as hung. Kit startup can exceed 60 s on cold CI workers, so this catches real startup hangs without killing legitimate slow launches. """ +PYTEST_WORKERS_ENV_VAR = "TEST_PYTEST_WORKERS" +"""Environment variable naming the number of ``pytest-xdist`` workers for each test file. + +Each file still runs in its own pytest process; the workers split that file's tests between them, each in +its own process with its own simulation (and Kit app, for files that launch one). Unset or ``1`` runs +the file serially. +""" + STARTUP_HANG_RETRIES = 2 """Number of times to retry a test that hangs during startup before giving up.""" @@ -202,6 +211,32 @@ def _drain_ready_output(process, stdout_fd, stderr_fd, timeout=0.1): return stdout_chunk, stderr_chunk +def _pytest_workers(env) -> int: + """Return the ``pytest-xdist`` worker count configured in ``env``, or 0 to run serially.""" + try: + workers = int(env.get(PYTEST_WORKERS_ENV_VAR, "") or 0) + except ValueError: + return 0 + return workers if workers > 1 else 0 + + +def _child_pids(pid: int) -> list[int]: + """Return the direct children of ``pid``, or an empty list where ``/proc`` is unavailable.""" + children = [] + for entry in os.listdir("/proc") if os.path.isdir("/proc") else []: + if not entry.isdigit(): + continue + try: + with open(f"/proc/{entry}/stat") as handle: + # The command name is parenthesized and may contain spaces; the parent PID follows it. + parent = int(handle.read().rsplit(")", 1)[1].split()[1]) + except (OSError, IndexError, ValueError): + continue + if parent == pid: + children.append(int(entry)) + return sorted(children) + + def _dump_hung_process_stacks(process, stdout_fd, stderr_fd, env): """Ask a hung process for a stack of every thread, and collect what it writes. @@ -211,7 +246,9 @@ def _dump_hung_process_stacks(process, stdout_fd, stderr_fd, env): The signal goes to the test process itself rather than its group. The handler is registered there, and a standalone script the test launched as a grandchild has no handler -- ``SIGUSR1`` would simply kill it, - losing it from the process tree the caller has already recorded. + losing it from the process tree the caller has already recorded. Under ``pytest-xdist`` the tests run in + the worker processes, the controller's direct children, so each worker is asked in turn as well; one at + a time, because they all append to the same dump file. Args: process: The hung child. @@ -233,23 +270,28 @@ def _dump_hung_process_stacks(process, stdout_fd, stderr_fd, env): if hang_dump.DUMP_SIGNAL is None or not dump_file: return "", stdout_data, stderr_data + targets = [process.pid] + if _pytest_workers(env): + targets += _child_pids(process.pid) + for _ in range(HANG_DUMP_PASSES): - # Only this pass's share of the file is the dump it asked for. - start = hang_dump.size(dump_file) - try: - os.kill(process.pid, hang_dump.DUMP_SIGNAL) - except OSError: - break + for pid in targets: + # Only this request's share of the file is the dump it asked for. + start = hang_dump.size(dump_file) + try: + os.kill(pid, hang_dump.DUMP_SIGNAL) + except OSError: + continue - # Keep draining while the handler runs, so a full pipe cannot be what stops it answering. - deadline = time.time() + HANG_DUMP_GRACE - while time.time() < deadline: - stdout_chunk, stderr_chunk = _drain_ready_output(process, stdout_fd, stderr_fd) - stdout_data += stdout_chunk - stderr_data += stderr_chunk + # Keep draining while the handler runs, so a full pipe cannot be what stops it answering. + deadline = time.time() + HANG_DUMP_GRACE + while time.time() < deadline: + stdout_chunk, stderr_chunk = _drain_ready_output(process, stdout_fd, stderr_fd) + stdout_data += stdout_chunk + stderr_data += stderr_chunk - if dumped := hang_dump.read_since(dump_file, start): - dumps.append(dumped) + if dumped := hang_dump.read_since(dump_file, start): + dumps.append(dumped if len(targets) == 1 else f"(pid {pid})\n{dumped}") # exit early if the process died if process.poll() is not None: break @@ -320,7 +362,11 @@ def capture_test_output_with_timeout(cmd, timeout, env, startup_deadline=0, repo elapsed = time.time() - start_time if not startup_done: - if b"AppLauncher initialization complete" in stderr_data or b"collected " in stdout_data: + if ( + b"AppLauncher initialization complete" in stderr_data + or b"collected " in stdout_data + or b" workers [" in stdout_data + ): startup_done = True if report_file and not shutdown_deadline and os.path.exists(report_file): @@ -993,6 +1039,8 @@ def _run_one_pass( cmd += ["-p", "mgpu_shard_select"] if ctx.ci_marker: cmd += ["-m", ctx.ci_marker] + if workers := _pytest_workers(ctx.env): + cmd += ["-n", str(workers)] if k_expr is not None: cmd += ["-k", k_expr] cmd += ctx.pytest_targets diff --git a/tools/hang_dump.py b/tools/hang_dump.py index ad519be57c59..9fef9b8b4b95 100644 --- a/tools/hang_dump.py +++ b/tools/hang_dump.py @@ -107,8 +107,10 @@ def register(): path = dump_path() if not path or not is_supported(): return False + # pytest-xdist workers share the controller's dump file, which the controller already truncated. + mode = "a" if os.environ.get("PYTEST_XDIST_WORKER") else "w" try: - _dump_file = open(path, "w") # noqa: SIM115 (held open for the process lifetime, see above) + _dump_file = open(path, mode) # noqa: SIM115 (held open for the process lifetime, see above) except OSError: return False faulthandler.register(DUMP_SIGNAL, file=_dump_file, all_threads=True, chain=False) diff --git a/tools/test_crash_journal.py b/tools/test_crash_journal.py index 86eb6531c1c8..a07e6f471b8d 100644 --- a/tools/test_crash_journal.py +++ b/tools/test_crash_journal.py @@ -429,6 +429,42 @@ def test_drop(): assert read_journal(str(journal_file)).collected == [f"{_FILE}::test_keep"] +def test_an_xdist_run_journals_each_event_once(tmp_path): + """Regression test for ``pytest-xdist`` runs journaling every event twice. + + The workers and the controller both fire the per-test hooks, and every worker fires the + collection hook. Duplicated starts turn a worker crash that xdist recovered from into an + unmatched start, so a later session crash would be blamed on a test that already reported. + """ + pytest.importorskip("xdist") + _write_test_module( + tmp_path, + """ + def test_a(): + pass + + def test_b(): + assert 1 == 2 + + def test_c(): + pass + """, + ) + journal_file = tmp_path / "journal.jsonl" + junit_file = tmp_path / "report.xml" + _run_pytest(tmp_path, journal_file, junit_file, "-p", "xdist.plugin", "-n", "2") + + records = [json.loads(line) for line in journal_file.read_text(encoding="utf-8").splitlines()] + node_ids = [f"{_FILE}::test_{name}" for name in "abc"] + assert [record["event"] for record in records].count("collected") == 1 + for event in ("start", "result", "finish"): + assert sorted(record["node_id"] for record in records if record["event"] == event) == node_ids + + journal = read_journal(str(journal_file)) + assert journal.collected == node_ids + assert journal.culprit is None + + # -- artificial crashes in a real pytest run -------------------------------------------------- diff --git a/uv.lock b/uv.lock index 20a39638029a..b341a9b1cbc8 100644 --- a/uv.lock +++ b/uv.lock @@ -1105,6 +1105,15 @@ epath = [ { name = "zipp" }, ] +[[package]] +name = "execnet" +version = "2.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/bf/89/780e11f9588d9e7128a3f87788354c7946a9cbb1401ad38a48c4db9a4f07/execnet-2.1.2.tar.gz", hash = "sha256:63d83bfdd9a23e35b9c6a3261412324f964c2ec8dcd8d3c6916ee9373e0befcd", size = 166622, upload-time = "2025-11-12T09:56:37.75Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ab/84/02fc1827e8cdded4aa65baef11296a9bbe595c474f0d6d758af082d849fd/execnet-2.1.2-py3-none-any.whl", hash = "sha256:67fba928dd5a544b783f6056f449e5e3931a5c378b128bc18501f7ea79e296ec", size = 40708, upload-time = "2025-11-12T09:56:36.333Z" }, +] + [[package]] name = "executing" version = "2.2.1" @@ -1818,6 +1827,7 @@ dev = [ { name = "myst-parser" }, { name = "pytest" }, { name = "pytest-mock" }, + { name = "pytest-xdist" }, { name = "sphinx" }, { name = "sphinx-book-theme" }, { name = "sphinx-copybutton" }, @@ -1903,6 +1913,7 @@ test = [ { name = "junitparser" }, { name = "pytest" }, { name = "pytest-mock" }, + { name = "pytest-xdist" }, ] tetrahedralization = [ { name = "pytetwild", extra = ["all"] }, @@ -2007,6 +2018,7 @@ requires-dist = [ { name = "pyopengl-accelerate", specifier = ">=3.1.0" }, { name = "pytest", marker = "extra == 'test'" }, { name = "pytest-mock", marker = "extra == 'test'" }, + { name = "pytest-xdist", marker = "extra == 'test'" }, { name = "pytetwild", extras = ["all"], marker = "extra == 'tetrahedralization'", specifier = ">=0.3.0,<0.4" }, { name = "ray", extras = ["default"], marker = "extra == 'rlinf'", specifier = ">=2.47.0" }, { name = "requests", specifier = ">=2.25.0" }, @@ -4405,6 +4417,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/5a/cc/06253936f4a7fa2e0f48dfe6d851d9c56df896a9ab09ac019d70b760619c/pytest_mock-3.15.1-py3-none-any.whl", hash = "sha256:0a25e2eb88fe5168d535041d09a4529a188176ae608a6d249ee65abc0949630d", size = 10095, upload-time = "2025-09-16T16:37:25.734Z" }, ] +[[package]] +name = "pytest-xdist" +version = "3.8.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "execnet" }, + { name = "pytest" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/78/b4/439b179d1ff526791eb921115fca8e44e596a13efeda518b9d845a619450/pytest_xdist-3.8.0.tar.gz", hash = "sha256:7e578125ec9bc6050861aa93f2d59f1d8d085595d6551c2c90b6f4fad8d3a9f1", size = 88069, upload-time = "2025-07-01T13:30:59.346Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ca/31/d4e37e9e550c2b92a9cbc2e4d0b7420a27224968580b5a447f420847c975/pytest_xdist-3.8.0-py3-none-any.whl", hash = "sha256:202ca578cfeb7370784a8c33d6d05bc6e13b4f25b5053c30a152269fd10f0b88", size = 46396, upload-time = "2025-07-01T13:30:56.632Z" }, +] + [[package]] name = "pytetwild" version = "0.3.0" From 7ed4a17bab8d136857630292422d2f2c6a372c1e Mon Sep 17 00:00:00 2001 From: Mustafa Haiderbhai Date: Thu, 24 Sep 2026 11:34:34 -0700 Subject: [PATCH 6/8] Backport backend writer signature fixes (#7991) (cherry picked from commit 2b62263437524bcbdb210a62a991fce632b052d6) --- .../core-cleanup-backend-signatures.skip | 1 + .../test/assets/test_articulation_iface.py | 13 ++++++ .../test_rigid_object_collection_iface.py | 41 ++++++++----------- .../core-cleanup-backend-signatures.rst | 7 ++++ .../assets/articulation/articulation.py | 21 ---------- .../core-cleanup-backend-signatures.rst | 15 +++++++ .../rigid_object_collection.py | 36 ++++++++++++++-- 7 files changed, 84 insertions(+), 50 deletions(-) create mode 100644 source/isaaclab/changelog.d/core-cleanup-backend-signatures.skip create mode 100644 source/isaaclab_newton/changelog.d/core-cleanup-backend-signatures.rst create mode 100644 source/isaaclab_physx/changelog.d/core-cleanup-backend-signatures.rst diff --git a/source/isaaclab/changelog.d/core-cleanup-backend-signatures.skip b/source/isaaclab/changelog.d/core-cleanup-backend-signatures.skip new file mode 100644 index 000000000000..2d429f207a4c --- /dev/null +++ b/source/isaaclab/changelog.d/core-cleanup-backend-signatures.skip @@ -0,0 +1 @@ +Test coverage for backend signature fixes. diff --git a/source/isaaclab/test/assets/test_articulation_iface.py b/source/isaaclab/test/assets/test_articulation_iface.py index f2f74075ede6..e0064b220ae7 100644 --- a/source/isaaclab/test/assets/test_articulation_iface.py +++ b/source/isaaclab/test/assets/test_articulation_iface.py @@ -1636,6 +1636,19 @@ def test_write_root_velocity_to_sim_mask( method(root_velocity=_make_bad_data_warp((num_instances,), device, wp.spatial_vectorf)) +@_backends +@pytest.mark.parametrize("num_instances, num_joints, num_bodies", [(2, 4, 5)]) +@pytest.mark.parametrize("device", ["cpu"]) +def test_deprecated_joint_friction_writers(backend, num_instances, num_joints, num_bodies, device, articulation_iface): + """The deprecated joint friction writers forward to the index writer on every backend.""" + art, _ = articulation_iface + friction = torch.full((num_instances, num_joints), 0.5, device=device) + with pytest.warns(DeprecationWarning): + art.write_joint_friction_coefficient_to_sim(friction) + with pytest.warns(DeprecationWarning): + art.write_joint_friction_to_sim(friction) + + # --------------------------------------------------------------------------- # Tests: Joint writers — torch/warp × index/mask × all/subset × negative # --------------------------------------------------------------------------- diff --git a/source/isaaclab/test/assets/test_rigid_object_collection_iface.py b/source/isaaclab/test/assets/test_rigid_object_collection_iface.py index 039140c3f229..b91e8d1b3128 100644 --- a/source/isaaclab/test/assets/test_rigid_object_collection_iface.py +++ b/source/isaaclab/test/assets/test_rigid_object_collection_iface.py @@ -1003,8 +1003,6 @@ def test_write_body_velocity_to_sim_index( method(body_velocities=_make_bad_data_warp((num_instances, num_bodies), device, wp.spatial_vectorf)) # -- mask variants for pose -- - # Note: write_body_pose_to_sim_mask accepts body_mask, but write_body_link_pose_to_sim_mask - # and write_body_com_pose_to_sim_mask use body_ids instead. We only test body_mask on body_pose. @_backends @_default_dims @@ -1018,8 +1016,6 @@ def test_write_body_pose_to_sim_mask( obj.data.update(dt=0.01) method = getattr(obj, f"write_{method_suffix}_to_sim_mask") - has_body_mask = method_suffix == "body_pose" - # torch, no mask (all) method(body_poses=_make_data_torch((num_instances, num_bodies), device, wp.transformf)) # torch, partial env_mask @@ -1027,18 +1023,17 @@ def test_write_body_pose_to_sim_mask( body_poses=_make_data_torch((num_instances, num_bodies), device, wp.transformf), env_mask=_make_env_mask(num_instances, device, True), ) - if has_body_mask: - # torch, partial body_mask - method( - body_poses=_make_data_torch((num_instances, num_bodies), device, wp.transformf), - body_mask=_make_item_mask(num_bodies, [0], device), - ) - # torch, both masks - method( - body_poses=_make_data_torch((num_instances, num_bodies), device, wp.transformf), - env_mask=_make_env_mask(num_instances, device, True), - body_mask=_make_item_mask(num_bodies, [0], device), - ) + # torch, partial body_mask + method( + body_poses=_make_data_torch((num_instances, num_bodies), device, wp.transformf), + body_mask=_make_item_mask(num_bodies, [0], device), + ) + # torch, both masks + method( + body_poses=_make_data_torch((num_instances, num_bodies), device, wp.transformf), + env_mask=_make_env_mask(num_instances, device, True), + body_mask=_make_item_mask(num_bodies, [0], device), + ) # warp, no mask method(body_poses=_make_data_warp((num_instances, num_bodies), device, wp.transformf)) # warp, partial env_mask @@ -1054,7 +1049,6 @@ def test_write_body_pose_to_sim_mask( method(body_poses=_make_bad_data_warp((num_instances, num_bodies), device, wp.transformf)) # -- mask variants for velocity -- - # Note: write_body_velocity_to_sim_mask accepts body_mask, but the _link_/_com_ variants use body_ids. @_backends @_default_dims @@ -1068,8 +1062,6 @@ def test_write_body_velocity_to_sim_mask( obj.data.update(dt=0.01) method = getattr(obj, f"write_{method_suffix}_to_sim_mask") - has_body_mask = method_suffix == "body_velocity" - # torch, no mask method(body_velocities=_make_data_torch((num_instances, num_bodies), device, wp.spatial_vectorf)) # torch, partial env_mask @@ -1077,12 +1069,11 @@ def test_write_body_velocity_to_sim_mask( body_velocities=_make_data_torch((num_instances, num_bodies), device, wp.spatial_vectorf), env_mask=_make_env_mask(num_instances, device, True), ) - if has_body_mask: - # torch, partial body_mask - method( - body_velocities=_make_data_torch((num_instances, num_bodies), device, wp.spatial_vectorf), - body_mask=_make_item_mask(num_bodies, [0], device), - ) + # torch, partial body_mask + method( + body_velocities=_make_data_torch((num_instances, num_bodies), device, wp.spatial_vectorf), + body_mask=_make_item_mask(num_bodies, [0], device), + ) # warp, no mask method(body_velocities=_make_data_warp((num_instances, num_bodies), device, wp.spatial_vectorf)) # warp, partial env_mask diff --git a/source/isaaclab_newton/changelog.d/core-cleanup-backend-signatures.rst b/source/isaaclab_newton/changelog.d/core-cleanup-backend-signatures.rst new file mode 100644 index 000000000000..f56968d1867a --- /dev/null +++ b/source/isaaclab_newton/changelog.d/core-cleanup-backend-signatures.rst @@ -0,0 +1,7 @@ +Fixed +^^^^^ + +* Fixed the deprecated :meth:`~isaaclab_newton.assets.Articulation.write_joint_friction_coefficient_to_sim` and + :meth:`~isaaclab_newton.assets.Articulation.write_joint_friction_to_sim` always raising ``TypeError``. The Newton + override passed arguments that :meth:`~isaaclab_newton.assets.Articulation.write_joint_friction_coefficient_to_sim_index` + does not accept; it is removed in favor of the base class implementation. diff --git a/source/isaaclab_newton/isaaclab_newton/assets/articulation/articulation.py b/source/isaaclab_newton/isaaclab_newton/assets/articulation/articulation.py index 5784e07a5d56..39ed08aa3194 100644 --- a/source/isaaclab_newton/isaaclab_newton/assets/articulation/articulation.py +++ b/source/isaaclab_newton/isaaclab_newton/assets/articulation/articulation.py @@ -3841,27 +3841,6 @@ def _resolve_mask(self, mask: wp.array | torch.Tensor | None, full_mask: wp.arra Deprecated methods. """ - def write_joint_friction_coefficient_to_sim( - self, - joint_friction_coeff: torch.Tensor | wp.array | float, - joint_ids: Sequence[int] | torch.Tensor | wp.array | None = None, - env_ids: Sequence[int] | torch.Tensor | wp.array | None = None, - full_data: bool = False, - ): - """Deprecated, same as :meth:`write_joint_friction_coefficient_to_sim_index`.""" - warnings.warn( - "The function 'write_joint_friction_coefficient_to_sim' will be deprecated in a future release. Please" - " use 'write_joint_friction_coefficient_to_sim_index' instead.", - DeprecationWarning, - stacklevel=2, - ) - self.write_joint_friction_coefficient_to_sim_index( - joint_friction_coeff, - joint_ids=joint_ids, - env_ids=env_ids, - full_data=full_data, - ) - def write_root_state_to_sim( self, root_state: torch.Tensor, diff --git a/source/isaaclab_physx/changelog.d/core-cleanup-backend-signatures.rst b/source/isaaclab_physx/changelog.d/core-cleanup-backend-signatures.rst new file mode 100644 index 000000000000..139ebf412368 --- /dev/null +++ b/source/isaaclab_physx/changelog.d/core-cleanup-backend-signatures.rst @@ -0,0 +1,15 @@ +Fixed +^^^^^ + +* Fixed :meth:`~isaaclab_physx.assets.RigidObjectCollection.write_body_link_pose_to_sim_mask`, + :meth:`~isaaclab_physx.assets.RigidObjectCollection.write_body_com_pose_to_sim_mask`, + :meth:`~isaaclab_physx.assets.RigidObjectCollection.write_body_com_velocity_to_sim_mask`, and + :meth:`~isaaclab_physx.assets.RigidObjectCollection.write_body_link_velocity_to_sim_mask` not accepting the + ``body_mask`` argument that the base class and the other backends declare. + +Deprecated +^^^^^^^^^^ + +* Deprecated the ``body_ids`` argument of the :class:`~isaaclab_physx.assets.RigidObjectCollection` mask writers + listed above. Pass a boolean ``body_mask`` of shape (num_bodies,) instead, or use the ``*_index`` writers with + ``body_ids``. diff --git a/source/isaaclab_physx/isaaclab_physx/assets/rigid_object_collection/rigid_object_collection.py b/source/isaaclab_physx/isaaclab_physx/assets/rigid_object_collection/rigid_object_collection.py index 05936dee34ea..188b68f7951a 100644 --- a/source/isaaclab_physx/isaaclab_physx/assets/rigid_object_collection/rigid_object_collection.py +++ b/source/isaaclab_physx/isaaclab_physx/assets/rigid_object_collection/rigid_object_collection.py @@ -481,6 +481,7 @@ def write_body_link_pose_to_sim_mask( self, *, body_poses: torch.Tensor | wp.array, + body_mask: wp.array | None = None, env_mask: wp.array | None = None, body_ids: Sequence[int] | torch.Tensor | wp.array | slice | None = None, skip_forward: bool = False, @@ -502,15 +503,17 @@ def write_body_link_pose_to_sim_mask( Args: body_poses: Body link poses in simulation frame. Shape is (num_instances, num_bodies, 7) or (num_instances, num_bodies) with dtype wp.transformf. + body_mask: Body mask. If None, then all bodies are updated. Shape is (num_bodies,). env_mask: Environment mask. If None, then all the instances are updated. Shape is (num_instances,). skip_forward: Whether to skip invalidating cached data after the write. When True, the caller must invalidate stale cached data before reading it back. Defaults to False. - body_ids: Body indices. If None, then all indices are used. + body_ids: Deprecated, use :attr:`body_mask` instead. Body indices. Defaults to None. """ if env_mask is not None: env_ids = self._resolve_env_mask(env_mask) else: env_ids = self._ALL_ENV_INDICES + body_ids = self._resolve_mask_writer_body_ids(body_mask, body_ids) self.write_body_link_pose_to_sim_index( body_poses=body_poses, env_ids=env_ids, body_ids=body_ids, full_data=True, skip_forward=skip_forward ) @@ -592,6 +595,7 @@ def write_body_com_pose_to_sim_mask( self, *, body_poses: torch.Tensor | wp.array, + body_mask: wp.array | None = None, env_mask: wp.array | None = None, body_ids: Sequence[int] | torch.Tensor | wp.array | slice | None = None, skip_forward: bool = False, @@ -614,15 +618,17 @@ def write_body_com_pose_to_sim_mask( Args: body_poses: Body center of mass poses in simulation frame. Shape is (num_instances, num_bodies, 7) or (num_instances, num_bodies) with dtype wp.transformf. + body_mask: Body mask. If None, then all bodies are updated. Shape is (num_bodies,). env_mask: Environment mask. If None, then all the instances are updated. Shape is (num_instances,). skip_forward: Whether to skip invalidating cached data after the write. When True, the caller must invalidate stale cached data before reading it back. Defaults to False. - body_ids: Body indices. If None, then all indices are used. + body_ids: Deprecated, use :attr:`body_mask` instead. Body indices. Defaults to None. """ if env_mask is not None: env_ids = self._resolve_env_mask(env_mask) else: env_ids = self._ALL_ENV_INDICES + body_ids = self._resolve_mask_writer_body_ids(body_mask, body_ids) self.write_body_com_pose_to_sim_index( body_poses=body_poses, env_ids=env_ids, body_ids=body_ids, full_data=True, skip_forward=skip_forward ) @@ -706,6 +712,7 @@ def write_body_com_velocity_to_sim_mask( self, *, body_velocities: torch.Tensor | wp.array, + body_mask: wp.array | None = None, env_mask: wp.array | None = None, body_ids: Sequence[int] | torch.Tensor | wp.array | slice | None = None, skip_forward: bool = False, @@ -731,15 +738,17 @@ def write_body_com_velocity_to_sim_mask( body_velocities: Body center of mass velocities in simulation frame. Shape is (num_instances, num_bodies, 6) or (num_instances, num_bodies) with dtype wp.spatial_vectorf. + body_mask: Body mask. If None, then all bodies are updated. Shape is (num_bodies,). env_mask: Environment mask. If None, then all the instances are updated. Shape is (num_instances,). skip_forward: Whether to skip invalidating cached data after the write. When True, the caller must invalidate stale cached data before reading it back. Defaults to False. - body_ids: Body indices. If None, then all indices are used. + body_ids: Deprecated, use :attr:`body_mask` instead. Body indices. Defaults to None. """ if env_mask is not None: env_ids = self._resolve_env_mask(env_mask) else: env_ids = self._ALL_ENV_INDICES + body_ids = self._resolve_mask_writer_body_ids(body_mask, body_ids) self.write_body_com_velocity_to_sim_index( body_velocities=body_velocities, env_ids=env_ids, @@ -831,6 +840,7 @@ def write_body_link_velocity_to_sim_mask( self, *, body_velocities: torch.Tensor | wp.array, + body_mask: wp.array | None = None, env_mask: wp.array | None = None, body_ids: Sequence[int] | torch.Tensor | wp.array | slice | None = None, skip_forward: bool = False, @@ -855,15 +865,17 @@ def write_body_link_velocity_to_sim_mask( Args: body_velocities: Body link velocities in simulation frame. Shape is (num_instances, num_bodies, 6) or (num_instances, num_bodies) with dtype wp.spatial_vectorf. + body_mask: Body mask. If None, then all bodies are updated. Shape is (num_bodies,). env_mask: Environment mask. If None, then all the instances are updated. Shape is (num_instances,). skip_forward: Whether to skip invalidating cached data after the write. When True, the caller must invalidate stale cached data before reading it back. Defaults to False. - body_ids: Body indices. If None, then all indices are used. + body_ids: Deprecated, use :attr:`body_mask` instead. Body indices. Defaults to None. """ if env_mask is not None: env_ids = self._resolve_env_mask(env_mask) else: env_ids = self._ALL_ENV_INDICES + body_ids = self._resolve_mask_writer_body_ids(body_mask, body_ids) self.write_body_link_velocity_to_sim_index( body_velocities=body_velocities, env_ids=env_ids, @@ -1328,6 +1340,22 @@ def _resolve_env_mask(self, env_mask: wp.array | None) -> torch.Tensor | wp.arra env_ids = self._ALL_ENV_INDICES return env_ids + def _resolve_mask_writer_body_ids( + self, body_mask: wp.array | None, body_ids: Sequence[int] | torch.Tensor | wp.array | slice | None + ) -> Sequence[int] | torch.Tensor | wp.array | slice: + """Resolve the bodies of a mask writer, accepting the deprecated ``body_ids`` in place of ``body_mask``.""" + if body_ids is None: + return self._resolve_body_mask(body_mask) + if body_mask is not None: + raise ValueError("Pass either 'body_mask' or the deprecated 'body_ids', not both.") + warnings.warn( + "The 'body_ids' argument of the rigid object collection mask writers is deprecated. Please use" + " 'body_mask' instead.", + DeprecationWarning, + stacklevel=3, + ) + return body_ids + def _resolve_body_mask(self, body_mask: wp.array | None) -> torch.Tensor | wp.array: """Resolve body mask to indices via torch.nonzero.""" if body_mask is not None: From 51214a8d8ecd71abb24eb91e4f8bf56e474471c7 Mon Sep 17 00:00:00 2001 From: Mustafa Haiderbhai Date: Thu, 24 Sep 2026 11:36:20 -0700 Subject: [PATCH 7/8] Backport consolidated Newton tests (#8003) (cherry picked from commit d696cd0cedd02c240417e63f97725485cbebf218) --- .../test-consolidate-newton-tests.skip | 1 + .../test/assets/test_articulation.py | 272 ++++-------------- .../assets/test_newton_actuators_newton.py | 16 -- .../test/assets/test_rigid_object.py | 38 --- .../assets/test_rigid_object_collection.py | 39 --- .../test_newton_manager_abstraction.py | 58 +--- .../test/sensors/test_contact_sensor.py | 30 +- 7 files changed, 81 insertions(+), 373 deletions(-) create mode 100644 source/isaaclab_newton/changelog.d/test-consolidate-newton-tests.skip diff --git a/source/isaaclab_newton/changelog.d/test-consolidate-newton-tests.skip b/source/isaaclab_newton/changelog.d/test-consolidate-newton-tests.skip new file mode 100644 index 000000000000..fe6c803a9b4b --- /dev/null +++ b/source/isaaclab_newton/changelog.d/test-consolidate-newton-tests.skip @@ -0,0 +1 @@ +Consolidated redundant Newton tests and trimmed device and shape matrices that repeated the same code paths. diff --git a/source/isaaclab_newton/test/assets/test_articulation.py b/source/isaaclab_newton/test/assets/test_articulation.py index b9e0534062cb..fac76841724c 100644 --- a/source/isaaclab_newton/test/assets/test_articulation.py +++ b/source/isaaclab_newton/test/assets/test_articulation.py @@ -910,7 +910,7 @@ def test_mjwarp_ordering_resolver_matches_newton_backend_names(sim, device, grav assert get_articulation_name_ordering(articulation, "mjwarp", kind="body") == tuple(articulation.backend_body_names) -@pytest.mark.parametrize("device", ["cuda:0", "cpu"]) +@pytest.mark.parametrize("device", test_devices(DeviceScope.CUDA)) @pytest.mark.parametrize("gravity_enabled", [False]) @pytest.mark.parametrize("articulation_type", ["single_joint_explicit"]) def test_branching_fixture_physx_ordering_reorders_newton_to_bfs(sim, device, gravity_enabled, articulation_type): @@ -974,7 +974,7 @@ class _ShapeCountSurface: assert articulation.num_shapes_per_body == [3, 0, 2] -@pytest.mark.parametrize("device", ["cuda:0", "cpu"]) +@pytest.mark.parametrize("device", test_devices(DeviceScope.CUDA)) @pytest.mark.parametrize("articulation_type", ["anymal"]) # consumed by the sim fixture @pytest.mark.parametrize("use_newton_actuators", [True]) # consumed by the sim fixture def test_newton_native_actuator_gain_write_maps_public_joint_subset_to_backend( @@ -1032,14 +1032,13 @@ def gather_stiffness() -> torch.Tensor: @pytest.mark.parametrize("num_articulations", [1]) -@pytest.mark.parametrize("device", ["cuda:0", "cpu"]) +@pytest.mark.parametrize("device", test_devices(DeviceScope.CUDA)) @pytest.mark.parametrize("gravity_enabled", [False]) @pytest.mark.parametrize("articulation_type", ["anymal"]) -@pytest.mark.parametrize("state_kind", ["pose", "velocity"]) def test_newton_ordered_body_state_cache_invalidates_on_same_timestamp_root_write( - sim, num_articulations, device, gravity_enabled, articulation_type, state_kind + sim, num_articulations, device, gravity_enabled, articulation_type ): - """Refresh ordered body state after a root write at the current simulation timestamp.""" + """Refresh ordered body pose and velocity after root writes at the current simulation timestamp.""" articulation_cfg = generate_articulation_cfg(articulation_type=articulation_type).replace( body_ordering=_ANYMAL_C_ROOT_PRESERVING_REVERSED_BODY_NAMES ) @@ -1054,32 +1053,29 @@ def test_newton_ordered_body_state_cache_invalidates_on_same_timestamp_root_writ root_body_idx = articulation.find_bodies("base")[0][0] sim_timestamp = data._sim_timestamp - if state_kind == "pose": - cached_body_state = data.body_link_pose_w.torch[:, root_body_idx].clone() - written_root_state = data.root_link_pose_w.torch.clone() - written_root_state[:, 0] += 0.25 - articulation.write_root_link_pose_to_sim_index(root_pose=written_root_state) - - assert data._sim_timestamp == sim_timestamp - torch.testing.assert_close(data.root_link_pose_w.torch, written_root_state) - refreshed_body_state = data.body_link_pose_w.torch[:, root_body_idx] - else: - cached_body_state = data.body_com_vel_w.torch[:, root_body_idx].clone() - written_root_state = torch.tensor( - [[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]], device=device, dtype=cached_body_state.dtype - ) - articulation.write_root_com_velocity_to_sim_index(root_velocity=written_root_state) - - assert data._sim_timestamp == sim_timestamp - torch.testing.assert_close(data.root_com_vel_w.torch, written_root_state) - refreshed_body_state = data.body_com_vel_w.torch[:, root_body_idx] - - torch.testing.assert_close(refreshed_body_state, written_root_state) - assert not torch.equal(refreshed_body_state, cached_body_state) + cached_body_pose = data.body_link_pose_w.torch[:, root_body_idx].clone() + written_root_pose = data.root_link_pose_w.torch.clone() + written_root_pose[:, 0] += 0.25 + articulation.write_root_link_pose_to_sim_index(root_pose=written_root_pose) + assert data._sim_timestamp == sim_timestamp + torch.testing.assert_close(data.root_link_pose_w.torch, written_root_pose) + refreshed_body_pose = data.body_link_pose_w.torch[:, root_body_idx] + torch.testing.assert_close(refreshed_body_pose, written_root_pose) + assert not torch.equal(refreshed_body_pose, cached_body_pose) + + # Populate the velocity cache after the pose write so only the velocity write can invalidate it. + cached_body_vel = data.body_com_vel_w.torch[:, root_body_idx].clone() + written_root_vel = torch.tensor([[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]], device=device, dtype=cached_body_vel.dtype) + articulation.write_root_com_velocity_to_sim_index(root_velocity=written_root_vel) + assert data._sim_timestamp == sim_timestamp + torch.testing.assert_close(data.root_com_vel_w.torch, written_root_vel) + refreshed_body_vel = data.body_com_vel_w.torch[:, root_body_idx] + torch.testing.assert_close(refreshed_body_vel, written_root_vel) + assert not torch.equal(refreshed_body_vel, cached_body_vel) @pytest.mark.parametrize("num_articulations", [1]) -@pytest.mark.parametrize("device", ["cuda:0", "cpu"]) +@pytest.mark.parametrize("device", test_devices(DeviceScope.CUDA)) @pytest.mark.parametrize("gravity_enabled", [False]) @pytest.mark.parametrize("articulation_type", ["panda"]) @pytest.mark.parametrize("ordering_mode", ["none", "reversed"]) @@ -1283,7 +1279,7 @@ def test_newton_ordered_state_caches_invalidate_on_rebind( @pytest.mark.parametrize("num_articulations", [1]) -@pytest.mark.parametrize("device", ["cuda:0", "cpu"]) +@pytest.mark.parametrize("device", test_devices(DeviceScope.CUDA)) @pytest.mark.parametrize("gravity_enabled", [True]) @pytest.mark.parametrize("articulation_type", ["anymal"]) @pytest.mark.parametrize("ordering_mode", ["none", "reversed"]) @@ -1358,10 +1354,10 @@ def test_newton_rebind_preserves_lab_owned_actuator_gains( @pytest.mark.parametrize("num_articulations", [1]) -@pytest.mark.parametrize("device", ["cuda:0", "cpu"]) +@pytest.mark.parametrize("device", test_devices(DeviceScope.CUDA)) @pytest.mark.parametrize("gravity_enabled", [True]) @pytest.mark.parametrize("articulation_type", ["anymal"]) -def test_newton_post_step_hook_publishes_ordered_state_inside_step( +def test_newton_post_step_hook_publishes_ordered_state_and_deregisters( sim, num_articulations, device, gravity_enabled, articulation_type ): """Republish the user-order Tier-1 shadows inside the sim step, without any read. @@ -1376,6 +1372,8 @@ def test_newton_post_step_hook_publishes_ordered_state_inside_step( Ships the eager-mode invariant variant: CUDA-graph capture is not reliably reachable from this CPU test harness, and this invariant directly proves the in-step republish. + + Finally checks that ``_clear_callbacks`` deregisters the hook without touching other callbacks. """ articulation_cfg = generate_articulation_cfg(articulation_type=articulation_type).replace( actuators={"legs": ImplicitActuatorCfg(joint_names_expr=[".*"], stiffness=40.0, damping=5.0)}, @@ -1409,47 +1407,18 @@ def test_newton_post_step_hook_publishes_ordered_state_inside_step( ) np.testing.assert_allclose(data._body_com_vel_w_user.numpy(), data._sim_bind_body_com_vel_w.numpy()[:, body_u2b]) - -@pytest.mark.parametrize("num_articulations", [1]) -@pytest.mark.parametrize("device", ["cpu"]) -@pytest.mark.parametrize("gravity_enabled", [True]) -@pytest.mark.parametrize("articulation_type", ["anymal"]) -def test_newton_clear_callbacks_deregisters_post_step_hook( - sim, num_articulations, device, gravity_enabled, articulation_type -): - """Deregister the ordered post-step republish hook so it does not leak on the manager. - - ``_create_buffers`` registers the backend-to-user state republish on - ``NewtonManager._post_step_callbacks`` for non-identity ordering. Without a - matching deregistration the bound method lingers on the class-level list - after the articulation is gone. ``_clear_callbacks`` must remove exactly that - callback and leave any other registered callback untouched. - """ - articulation_cfg = generate_articulation_cfg(articulation_type=articulation_type).replace( - actuators={"legs": ImplicitActuatorCfg(joint_names_expr=[".*"], stiffness=40.0, damping=5.0)}, - joint_ordering=tuple(reversed(ANYMAL_C_PHYSX_JOINT_NAMES)), - body_ordering=_ANYMAL_C_ROOT_PRESERVING_REVERSED_BODY_NAMES, - ) - articulation, _ = generate_articulation(articulation_cfg, num_articulations, device=sim.device) - sim.reset() - assert articulation.is_initialized - assert articulation.data.joint_ordering is not None - assert articulation.data.body_ordering is not None - - # The republish hook is registered on the manager for non-identity ordering. + # ``_clear_callbacks`` must deregister exactly this hook so it does not leak on the class-level list, + # and leave an unrelated callback (standing in for another articulation's hook) untouched. registered_callback = articulation._post_step_callback assert registered_callback is not None assert registered_callback in SimulationManager._post_step_callbacks - # A second, independent callback stands in for another articulation's hook. def _other_callback() -> None: return None SimulationManager.register_post_step_callback(_other_callback) - articulation._clear_callbacks() - # The articulation's own hook is gone; the unrelated callback survives. assert articulation._post_step_callback is None assert registered_callback not in SimulationManager._post_step_callbacks assert _other_callback in SimulationManager._post_step_callbacks @@ -1887,19 +1856,12 @@ def test_initialization_fixed_base_single_joint(sim, num_articulations, device, @pytest.mark.parametrize("num_articulations", [2]) @pytest.mark.parametrize("device", test_devices()) @pytest.mark.parametrize("articulation_type", ["shadow_hand"]) -def test_initialization_hand_with_tendons(sim, num_articulations, device, articulation_type): - """Test initialization for fixed base articulated hand with tendons. +def test_hand_with_tendons_initializes_and_targets_only_given_envs(sim, num_articulations, device, articulation_type): + """Initialize a fixed-base hand with tendons; a tendon command for one environment must leave the others alone. - This test verifies that: - 1. The articulation is properly initialized - 2. The articulation is fixed base - 3. All buffers have correct shapes - 4. The articulation can be simulated - - Args: - sim: The simulation fixture - num_articulations: Number of articulations to test - device: The device to run the simulation on + ``set_fixed_tendon_position_target_index`` is declared backend-neutral and documented to accept + partial data. Newton took ``env_ids`` and never forwarded it, so a partial command was sized + against every instance and raised rather than commanding the environment asked for. """ articulation_cfg = generate_articulation_cfg(articulation_type=articulation_type) articulation, _ = generate_articulation(articulation_cfg, num_articulations, device=device) @@ -1907,49 +1869,19 @@ def test_initialization_hand_with_tendons(sim, num_articulations, device, articu # Check that the framework doesn't hold excessive strong references. assert sys.getrefcount(articulation) < 10 - # Play sim sim.reset() - # Check if articulation is initialized assert articulation.is_initialized - # Check that fixed base assert articulation.is_fixed_base - # Check buffers that exists and have correct shapes + assert articulation.num_fixed_tendons > 0 assert articulation.data.root_pos_w.torch.shape == (num_articulations, 3) assert articulation.data.root_quat_w.torch.shape == (num_articulations, 4) assert articulation.data.joint_pos.torch.shape == (num_articulations, 24) assert articulation.data.body_mass.torch.shape == (num_articulations, articulation.num_bodies) assert articulation.data.body_inertia.torch.shape == (num_articulations, articulation.num_bodies, 9) - - # -- actuator type for actuator_name, actuator in articulation.actuators.items(): is_implicit_model_cfg = isinstance(articulation_cfg.actuators[actuator_name], ImplicitActuatorCfg) assert getattr(actuator, "is_implicit_model", False) == is_implicit_model_cfg - # Simulate physics - for _ in range(10): - # perform rendering - sim.step() - # update articulation - articulation.update(sim.cfg.dt) - - -@pytest.mark.parametrize("num_articulations", [2]) -@pytest.mark.parametrize("device", test_devices()) -@pytest.mark.parametrize("articulation_type", ["shadow_hand"]) -def test_fixed_tendon_position_target_reaches_only_given_envs(sim, num_articulations, device, articulation_type): - """A tendon command for one environment must leave the others alone. - - ``set_fixed_tendon_position_target_index`` is declared backend-neutral and documented to accept - partial data. Newton took ``env_ids`` and never forwarded it, so a partial command was sized - against every instance and raised rather than commanding the environment asked for. - """ - articulation_cfg = generate_articulation_cfg(articulation_type=articulation_type) - articulation, _ = generate_articulation(articulation_cfg, num_articulations, device=device) - - sim.reset() - assert articulation.is_initialized - assert articulation.num_fixed_tendons > 0 - target = torch.full((1, articulation.num_fixed_tendons), 1.0, dtype=torch.float32, device=device) articulation.set_fixed_tendon_position_target_index(target=target, env_ids=[0]) @@ -2712,7 +2644,7 @@ def test_setting_gains_from_cfg(sim, num_articulations, device, add_ground_plane @pytest.mark.parametrize("num_articulations", [2]) -@pytest.mark.parametrize("device", test_devices()) +@pytest.mark.parametrize("device", test_devices(DeviceScope.CUDA)) @pytest.mark.parametrize("joint_velocity_limit", [1e5, None]) @pytest.mark.parametrize("vel_limit", [1e2, None]) @pytest.mark.parametrize("articulation_type", ["single_joint_implicit"]) # consumed by the sim fixture @@ -2772,7 +2704,7 @@ def test_setting_velocity_limit_implicit( @pytest.mark.parametrize("num_articulations", [2]) -@pytest.mark.parametrize("device", test_devices()) +@pytest.mark.parametrize("device", test_devices(DeviceScope.CUDA)) @pytest.mark.parametrize("joint_velocity_limit", [1e5, None]) @pytest.mark.parametrize("vel_limit", [1e2, None]) @pytest.mark.parametrize("articulation_type", ["single_joint_explicit"]) # consumed by the sim fixture @@ -2830,7 +2762,7 @@ def test_setting_velocity_limit_explicit( @pytest.mark.parametrize("num_articulations", [2]) -@pytest.mark.parametrize("device", test_devices()) +@pytest.mark.parametrize("device", test_devices(DeviceScope.CUDA)) @pytest.mark.parametrize("joint_effort_limit", [1e5, None]) @pytest.mark.parametrize("articulation_type", ["single_joint_implicit"]) # consumed by the sim fixture def test_setting_effort_limit_implicit(sim, articulation_type, num_articulations, device, joint_effort_limit): @@ -2877,7 +2809,7 @@ def test_setting_effort_limit_implicit(sim, articulation_type, num_articulations @pytest.mark.parametrize("num_articulations", [2]) -@pytest.mark.parametrize("device", test_devices()) +@pytest.mark.parametrize("device", test_devices(DeviceScope.CUDA)) @pytest.mark.parametrize("joint_effort_limit", [1e5, None]) @pytest.mark.parametrize("actuator_effort_limit", [1e2, None]) @pytest.mark.parametrize("articulation_type", ["single_joint_explicit"]) # consumed by the sim fixture @@ -3634,63 +3566,6 @@ def _patched_simulate(cls): ) -@pytest.mark.parametrize("add_ground_plane", [True]) -@pytest.mark.parametrize("num_articulations", [2]) -@pytest.mark.parametrize("device", test_devices()) -@pytest.mark.parametrize("articulation_type", ["panda"]) -def test_set_material_properties(sim, num_articulations, device, add_ground_plane, articulation_type): - """Test getting and setting material properties (friction/restitution) via view-level APIs.""" - articulation_cfg = generate_articulation_cfg(articulation_type=articulation_type) - articulation, _ = generate_articulation( - articulation_cfg=articulation_cfg, num_articulations=num_articulations, device=device - ) - - # Play the simulator - sim.reset() - - # Get friction/restitution bindings via view-level API - model = SimulationManager.get_model() - friction_binding = articulation._root_view.get_attribute("shape_material_mu", model)[:, 0] - restitution_binding = articulation._root_view.get_attribute("shape_material_restitution", model)[:, 0] - num_shapes = friction_binding.shape[1] - - # Test 1: Set all shapes via in-place writes to the warp binding - friction = torch.empty(num_articulations, num_shapes, device=device).uniform_(0.4, 0.8) - restitution = torch.empty(num_articulations, num_shapes, device=device).uniform_(0.0, 0.2) - - wp.to_torch(friction_binding)[:] = friction - wp.to_torch(restitution_binding)[:] = restitution - SimulationManager.add_model_change(ModelFlags.SHAPE_PROPERTIES) - - # Simulate physics - sim.step() - articulation.update(sim.cfg.dt) - - # Verify by reading back from the binding - mu = wp.to_torch(friction_binding) - restitution_check = wp.to_torch(restitution_binding) - torch.testing.assert_close(mu, friction) - torch.testing.assert_close(restitution_check, restitution) - - # Test 2: Set subset of shapes (only shape 0) - if num_shapes > 1: - subset_friction = torch.empty(num_articulations, device=device).uniform_(0.1, 0.2) - subset_restitution = torch.empty(num_articulations, device=device).uniform_(0.5, 0.6) - - wp.to_torch(friction_binding)[:, 0] = subset_friction - wp.to_torch(restitution_binding)[:, 0] = subset_restitution - SimulationManager.add_model_change(ModelFlags.SHAPE_PROPERTIES) - - sim.step() - articulation.update(sim.cfg.dt) - - # Check only the subset was updated - mu_updated = wp.to_torch(friction_binding) - restitution_updated = wp.to_torch(restitution_binding) - torch.testing.assert_close(mu_updated[:, 0], subset_friction) - torch.testing.assert_close(restitution_updated[:, 0], subset_restitution) - - @pytest.mark.parametrize("num_articulations", [2]) @pytest.mark.parametrize("device", test_devices(DeviceScope.CUDA)) @pytest.mark.parametrize("add_ground_plane", [True]) @@ -3755,7 +3630,7 @@ def test_randomize_rigid_body_collider_offsets(sim, num_articulations, device, a ## -@pytest.mark.parametrize("num_articulations", [1, 4]) +@pytest.mark.parametrize("num_articulations", [4]) @pytest.mark.parametrize("device", test_devices(DeviceScope.CUDA)) @pytest.mark.parametrize("add_ground_plane", [True]) @pytest.mark.parametrize("articulation_type", ["panda", "anymal"]) @@ -3763,6 +3638,8 @@ def test_randomize_rigid_body_collider_offsets(sim, num_articulations, device, a def test_dynamics_accessor_shapes(sim, num_articulations, device, add_ground_plane, articulation_type): """Pin the per-articulation shapes of the Jacobian, mass matrix and gravity compensation accessors. + Also checks that the mass matrix is symmetric and positive-definite. + Fixed-base (panda): ``body_link_jacobian_w`` drops the fixed-root row, so its shape is ``(N, num_bodies - 1, 6, num_joints)``; ``mass_matrix`` is ``(N, num_joints, num_joints)`` and ``gravity_compensation_forces`` is ``(N, num_joints)``. @@ -3802,6 +3679,14 @@ def test_dynamics_accessor_shapes(sim, num_articulations, device, add_ground_pla diag = M.diagonal(dim1=-2, dim2=-1) assert (diag > 1e-6).all(), f"mass matrix has non-positive diagonal entries: min={diag.min()}" + # The joint-space inertia is symmetric by construction; asymmetry means a wrong-axis gather or a + # half-populated buffer. OSC inverts ``J M^-1 J^T`` every step, so ``M`` must also be positive-definite. + asym = (M - M.transpose(-1, -2)).abs().max().item() + assert asym < 1e-4, f"|M - M^T|_max = {asym:.3e} — mass matrix is not symmetric" + # A tiny jitter tolerates the float32 eigenvalue floor without masking real non-PD bugs. + eye = torch.eye(M.shape[-1], device=M.device, dtype=M.dtype).expand_as(M) + torch.linalg.cholesky(M + 1e-6 * eye) + @pytest.mark.parametrize("device", test_devices(DeviceScope.CUDA)) @pytest.mark.parametrize("add_ground_plane", [True]) @@ -3982,57 +3867,6 @@ def test_get_jacobians_link_origin_contract(sim, num_articulations, device, arti torch.testing.assert_close(v_pred_lin, v_origin_expected, atol=5e-3, rtol=1e-2) -@pytest.mark.parametrize("num_articulations", [4]) -@pytest.mark.parametrize("device", test_devices(DeviceScope.CUDA)) -@pytest.mark.parametrize("articulation_type", ["panda", "anymal"]) -@pytest.mark.parametrize("gravity_enabled", [False]) -@pytest.mark.isaacsim_ci -def test_get_mass_matrix_symmetry_pd(sim, num_articulations, device, articulation_type, gravity_enabled): - """The joint-space mass matrix ``M(q)`` must be square, symmetric, and positive-definite. - - This pins three structural properties of - :attr:`~isaaclab.assets.BaseArticulationData.mass_matrix`: - - * **Square**: shape ``(N, num_joints + num_base_dofs, num_joints + num_base_dofs)``. - A transposed gather or a non-square scratch buffer would be caught - here before downstream OSC inversion silently propagates garbage. - * **Symmetric**: ``M == M.T`` to numerical precision. The joint- - space inertia tensor is symmetric by construction; an asymmetric - result indicates a wrong-axis gather, half-populated buffer, or - Cholesky-input bug. - * **Positive-definite**: ``torch.linalg.cholesky(M)`` succeeds. OSC - computes ``M_b = (J · M^-1 · J^T)^-1`` which requires PD on every - step. A non-PD M would fail downstream as ``LinAlgError``; this - test catches it earlier and pinpoints the source. - - Parameterized on both fixed-base (panda) and floating-base (anymal). - Both backends include the floating-base DoF rows/cols on the front of - the DoF axis for floating-base assets. - """ - articulation_cfg = generate_articulation_cfg(articulation_type=articulation_type) - articulation, _ = generate_articulation(articulation_cfg, num_articulations, device=device) - sim.reset() - assert articulation.is_initialized - - sim.step() - articulation.update(sim.cfg.dt) - - M = articulation.data.mass_matrix.torch # (N, J, J) - assert M.dim() == 3, f"expected 3-D mass matrix, got shape {tuple(M.shape)}" - assert M.shape[0] == num_articulations - assert M.shape[1] == M.shape[2], f"mass matrix is not square: {tuple(M.shape)}" - - # Symmetric to numerical precision. - asym = (M - M.transpose(-1, -2)).abs().max().item() - assert asym < 1e-4, f"|M - M^T|_max = {asym:.3e} — mass matrix is not symmetric" - - # Positive-definite via Cholesky. Adds a tiny diagonal jitter to - # tolerate the floor of float32 PD eigenvalues without masking real - # non-PD bugs (the jitter is well below realistic inertia scales). - eye = torch.eye(M.shape[-1], device=M.device, dtype=M.dtype).expand_as(M) - torch.linalg.cholesky(M + 1e-6 * eye) - - @pytest.mark.parametrize("num_articulations", [4]) @pytest.mark.parametrize("device", test_devices(DeviceScope.CUDA)) @pytest.mark.parametrize("add_ground_plane", [True]) diff --git a/source/isaaclab_newton/test/assets/test_newton_actuators_newton.py b/source/isaaclab_newton/test/assets/test_newton_actuators_newton.py index debdd0103ae8..bdf081d965a7 100644 --- a/source/isaaclab_newton/test/assets/test_newton_actuators_newton.py +++ b/source/isaaclab_newton/test/assets/test_newton_actuators_newton.py @@ -585,22 +585,6 @@ class TestDelayedPDEquivalence(_EquivalenceTestBase): actuators = DELAYED_PD_ACTUATORS -class TestDelayedPDAuthoring(unittest.TestCase): - """Verify DelayedPDActuatorCfg is authored with NewtonActuatorDelayAPI.""" - - @classmethod - def setUpClass(cls): - cls.result = _run_authoring_introspection(DELAYED_PD_ACTUATORS) - - def test_has_delay(self): - for a in self.result["actuator_info"]: - self.assertTrue(a["has_delay"], "Delay not found on delayed PD actuator") - - def test_controller_is_pd(self): - for a in self.result["actuator_info"]: - self.assertEqual(a["controller_type"], "DrivePD") - - # --------------------------------------------------------------------------- # Decimation tests: re-run equivalence with decimation > 1 + CUDA graph capture # --------------------------------------------------------------------------- diff --git a/source/isaaclab_newton/test/assets/test_rigid_object.py b/source/isaaclab_newton/test/assets/test_rigid_object.py index 69babf66f89a..6d05dd4577ba 100644 --- a/source/isaaclab_newton/test/assets/test_rigid_object.py +++ b/source/isaaclab_newton/test/assets/test_rigid_object.py @@ -513,44 +513,6 @@ def test_reset_rigid_object(num_cubes, device): assert torch.count_nonzero(cube_object._permanent_wrench_composer.out_torque_b.torch) == 0 -@pytest.mark.isaacsim_ci -@pytest.mark.parametrize("num_cubes", [2]) -@pytest.mark.parametrize("device", test_devices()) -def test_rigid_body_set_material_properties(num_cubes, device): - """Test getting and setting material properties of rigid object via view-level APIs.""" - with _newton_sim_context(device, gravity_enabled=True, add_ground_plane=True, auto_add_lighting=True) as sim: - sim._app_control_on_stop_handle = None - # Generate cubes scene - cube_object, _ = generate_cubes_scene(num_cubes=num_cubes, device=device) - - # Play sim - sim.reset() - - # Get friction/restitution bindings via view-level API - model = SimulationManager.get_model() - friction_binding = cube_object._root_view.get_attribute("shape_material_mu", model)[:, 0] - restitution_binding = cube_object._root_view.get_attribute("shape_material_restitution", model)[:, 0] - num_shapes = friction_binding.shape[1] - - # Set material properties via in-place writes to the warp binding - friction = torch.empty(num_cubes, num_shapes, device=device).uniform_(0.4, 0.8) - restitution = torch.empty(num_cubes, num_shapes, device=device).uniform_(0.0, 0.2) - - wp.to_torch(friction_binding)[:] = friction - wp.to_torch(restitution_binding)[:] = restitution - SimulationManager.add_model_change(ModelFlags.SHAPE_PROPERTIES) - - # Simulate physics - sim.step() - cube_object.update(sim.cfg.dt) - - # Verify by reading back from the binding - mu = wp.to_torch(friction_binding) - restitution_check = wp.to_torch(restitution_binding) - torch.testing.assert_close(mu, friction) - torch.testing.assert_close(restitution_check, restitution) - - @pytest.mark.isaacsim_ci @pytest.mark.parametrize("num_cubes", [2]) @pytest.mark.parametrize("device", test_devices()) diff --git a/source/isaaclab_newton/test/assets/test_rigid_object_collection.py b/source/isaaclab_newton/test/assets/test_rigid_object_collection.py index 10ac865f076c..2abff5889673 100644 --- a/source/isaaclab_newton/test/assets/test_rigid_object_collection.py +++ b/source/isaaclab_newton/test/assets/test_rigid_object_collection.py @@ -564,45 +564,6 @@ def test_reset_object_collection(num_envs, num_cubes, device): assert torch.count_nonzero(object_collection._permanent_wrench_composer.out_torque_b.torch) == 0 -@pytest.mark.parametrize("num_envs", [3]) -@pytest.mark.parametrize("num_cubes", [2]) -@pytest.mark.parametrize("device", test_devices()) -def test_set_material_properties(num_envs, num_cubes, device): - """Test getting and setting material properties of rigid object collection via view-level APIs.""" - with _newton_sim_context(device, auto_add_lighting=True) as sim: - sim._app_control_on_stop_handle = None - object_collection, _ = generate_cubes_scene(num_envs=num_envs, num_cubes=num_cubes, device=device) - sim.reset() - - # Get friction/restitution bindings via view-level API - # The collection's _root_view stores data in flat view order: (num_envs * num_cubes, ...) - model = SimulationManager.get_model() - friction_raw = object_collection._root_view.get_attribute("shape_material_mu", model) - restitution_raw = object_collection._root_view.get_attribute("shape_material_restitution", model) - - # Shape is (num_envs * num_cubes, num_shapes_per_body, 1) — slice off trailing dim - friction_binding = friction_raw[:, :, 0] - restitution_binding = restitution_raw[:, :, 0] - - # Generate random values matching the flat view shape - friction = torch.empty_like(wp.to_torch(friction_binding)).uniform_(0.4, 0.8) - restitution = torch.empty_like(wp.to_torch(restitution_binding)).uniform_(0.0, 0.2) - - wp.to_torch(friction_binding)[:] = friction - wp.to_torch(restitution_binding)[:] = restitution - SimulationManager.add_model_change(ModelFlags.SHAPE_PROPERTIES) - - # Perform simulation - sim.step() - object_collection.update(sim.cfg.dt) - - # Verify by reading back from the binding - mu = wp.to_torch(friction_binding) - restitution_check = wp.to_torch(restitution_binding) - torch.testing.assert_close(mu, friction) - torch.testing.assert_close(restitution_check, restitution) - - @pytest.mark.parametrize("num_envs", [3]) @pytest.mark.parametrize("num_cubes", [2]) @pytest.mark.parametrize("device", test_devices()) diff --git a/source/isaaclab_newton/test/physics/test_newton_manager_abstraction.py b/source/isaaclab_newton/test/physics/test_newton_manager_abstraction.py index f2731a773e41..da1ac15c6e30 100644 --- a/source/isaaclab_newton/test/physics/test_newton_manager_abstraction.py +++ b/source/isaaclab_newton/test/physics/test_newton_manager_abstraction.py @@ -767,21 +767,6 @@ def test_mpm_prepare_builder_converts_convex_mesh_before_solver_construction(): assert isinstance(solver, SolverImplicitMPM) -def test_active_manager_create_builder_registers_mpm_attributes(): - """The active MPM manager registers solver-specific builder attributes.""" - sim_cfg = SimulationCfg( - dt=1.0 / 120.0, - device="cuda:0", - gravity=(0.0, 0.0, -9.81), - physics=NewtonCfg(solver_cfg=MPMSolverCfg(max_iterations=2, voxel_size=0.05), use_cuda_graph=False), - ) - - with build_simulation_context(sim_cfg=sim_cfg) as sim: - builder = sim.physics_manager.create_builder() - - assert builder.has_custom_attribute("mpm:young_modulus") - - @pytest.mark.parametrize("import_path", ["clone", "standalone"]) @pytest.mark.parametrize( ("manager_cls", "solver_cfg", "expected_friction", "expected_damping"), @@ -915,46 +900,6 @@ def test_schema_resolver_policy_and_precedence(manager_cls, imports_mujoco, auth assert model.joint_armature.numpy()[-1] == pytest.approx(expected_armature) -def test_mpm_end_to_end_with_particle_custom_attributes(): - """End-to-end MPM step using ``add_particles(custom_attributes=...)`` — the production path.""" - sim_cfg = SimulationCfg( - dt=1.0 / 120.0, - device="cuda:0", - gravity=(0.0, 0.0, -9.81), - physics=NewtonCfg( - solver_cfg=MPMSolverCfg(max_iterations=2, voxel_size=0.05), - use_cuda_graph=False, - ), - ) - - with build_simulation_context(sim_cfg=sim_cfg) as sim: - builder = sim.physics_manager.create_builder() - # MPM custom attrs must exist on the builder before particles use them. - assert builder.has_custom_attribute("mpm:young_modulus") - - positions = [(0.0, 0.0, 0.10), (0.05, 0.0, 0.10), (0.0, 0.05, 0.10)] - builder.add_particles( - pos=positions, - vel=[(0.0, 0.0, 0.0)] * len(positions), - mass=[0.01] * len(positions), - radius=[0.02] * len(positions), - custom_attributes={ - "mpm:viscosity": 50.0, - "mpm:friction": 0.0, - "mpm:tensile_yield_ratio": 1.0, - "mpm:yield_pressure": 1.0e15, - "mpm:yield_stress": 0.0, - "mpm:young_modulus": 1.0e15, - "mpm:damping": 0.0, - }, - ) - NewtonManager.set_builder(builder) - - sim.reset() - assert isinstance(NewtonManager._solver, SolverImplicitMPM) - sim.step(render=False) - - @pytest.mark.parametrize("project_outside", [True, False]) def test_mpm_project_outside_colliders_gates_projection(project_outside): """``project_outside_colliders`` controls whether ``project_outside`` runs per substep. @@ -975,6 +920,8 @@ def test_mpm_project_outside_colliders_gates_projection(project_outside): with build_simulation_context(sim_cfg=sim_cfg) as sim: builder = sim.physics_manager.create_builder() + # MPM custom attrs must exist on the builder before particles use them (the production path). + assert builder.has_custom_attribute("mpm:young_modulus") builder.add_particles( pos=[(0.0, 0.0, 0.10), (0.05, 0.0, 0.10), (0.0, 0.05, 0.10)], vel=[(0.0, 0.0, 0.0)] * 3, @@ -992,6 +939,7 @@ def test_mpm_project_outside_colliders_gates_projection(project_outside): ) NewtonManager.set_builder(builder) sim.reset() + assert isinstance(NewtonManager._solver, SolverImplicitMPM) calls = {"n": 0} original_project = NewtonManager._solver.project_outside diff --git a/source/isaaclab_newton/test/sensors/test_contact_sensor.py b/source/isaaclab_newton/test/sensors/test_contact_sensor.py index 97b5577629c5..5242f0eb71c3 100644 --- a/source/isaaclab_newton/test/sensors/test_contact_sensor.py +++ b/source/isaaclab_newton/test/sensors/test_contact_sensor.py @@ -20,7 +20,7 @@ import sys from pathlib import Path -from isaaclab.test.utils import test_devices +from isaaclab.test.utils import DeviceScope, test_devices sys.path.insert(0, str(Path(__file__).resolve().parents[1])) @@ -84,9 +84,26 @@ class ContactSensorTestSceneCfg(InteractiveSceneCfg): # =================================================================== -@pytest.mark.parametrize("device", test_devices()) -@pytest.mark.parametrize("use_mujoco_contacts", COLLISION_PIPELINES) -@pytest.mark.parametrize("shape_type", STABLE_SHAPES, ids=[shape_type_to_str(s) for s in STABLE_SHAPES]) +def _rotated_lifecycle_cases() -> list: + """Cover every shape once while rotating through every device and collision pipeline. + + The sensor has no shape- or device-specific code path, so the full cartesian product only + repeats the same assertions; each (device, pipeline) pair still runs on several shapes. + """ + devices = test_devices() + cases = [] + for i, shape_type in enumerate(STABLE_SHAPES): + device = devices[i % len(devices)] + pipeline = COLLISION_PIPELINES[(i // len(devices)) % len(COLLISION_PIPELINES)] + cases.append( + pytest.param( + device, pipeline.values[0], shape_type, id=f"{shape_type_to_str(shape_type)}-{pipeline.id}-{device}" + ) + ) + return cases + + +@pytest.mark.parametrize("device, use_mujoco_contacts, shape_type", _rotated_lifecycle_cases()) def test_contact_lifecycle(device: str, use_mujoco_contacts: bool, shape_type: ShapeType): """Test full contact detection lifecycle with varied heights across environments. @@ -1193,8 +1210,9 @@ def test_invalid_expression_raises_regex_error(): _compile_label_pattern("foo(") -@pytest.mark.parametrize("device", test_devices()) -@pytest.mark.parametrize("clock_age", [2.5, 10.0, 30.0]) +# The clock is accumulated identically on every device; the largest age bounds the drift. +@pytest.mark.parametrize("device", test_devices(DeviceScope.CUDA)) +@pytest.mark.parametrize("clock_age", [2.5, 30.0]) @pytest.mark.parametrize("history_length", [1, 0], ids=["substep_refresh", "lazy_refresh"]) def test_first_transition_with_aged_clock(device: str, clock_age: float, history_length: int): """Regression for #7283: transitions must still be reported once the sensor clock has aged. From 7a5c8349cee95fb8bdd6dc86b7da1e988d0c1642 Mon Sep 17 00:00:00 2001 From: Mustafa Haiderbhai Date: Thu, 24 Sep 2026 12:15:15 -0700 Subject: [PATCH 8/8] Revert premature Newton CI backport (#8006) Keep #8006 out of the release backport until its source PR merges. --- .github/actions/run-package-tests/action.yml | 7 -- .github/actions/run-tests/action.yml | 9 +-- .github/actions/run-tests/run_tests.sh | 6 -- .github/workflows/build.yaml | 3 - .github/workflows/tools-tests.yml | 3 +- conftest.py | 9 --- pyproject.toml | 2 - tools/conftest.py | 82 ++++---------------- tools/hang_dump.py | 4 +- tools/test_crash_journal.py | 36 --------- uv.lock | 25 ------ 11 files changed, 20 insertions(+), 166 deletions(-) diff --git a/.github/actions/run-package-tests/action.yml b/.github/actions/run-package-tests/action.yml index f6134bedbfe4..2f32aa5f031c 100644 --- a/.github/actions/run-package-tests/action.yml +++ b/.github/actions/run-package-tests/action.yml @@ -47,12 +47,6 @@ inputs: spawned by tools/conftest.py (combined with device-split selectors). default: '' required: false - pytest-workers: - description: >- - Number of pytest-xdist worker processes each per-file pytest run spawned by - tools/conftest.py splits its tests across. Empty or 1 runs every file serially. - default: '' - required: false shard-index: description: 'Zero-based shard index' default: '' @@ -339,7 +333,6 @@ runs: filter-pattern: ${{ inputs.filter-pattern }} exclude-pattern: ${{ inputs.exclude-pattern }} test-k-expr: ${{ inputs.test-k-expr }} - pytest-workers: ${{ inputs.pytest-workers }} shard-index: ${{ inputs.shard-index }} shard-count: ${{ inputs.shard-count }} curobo-only: ${{ inputs.curobo-only }} diff --git a/.github/actions/run-tests/action.yml b/.github/actions/run-tests/action.yml index 572c81cfb038..510c9e4cc033 100644 --- a/.github/actions/run-tests/action.yml +++ b/.github/actions/run-tests/action.yml @@ -49,12 +49,6 @@ inputs: can deselect parametrized cases (e.g. "not ovphysx"). default: '' required: false - pytest-workers: - description: >- - Number of pytest-xdist worker processes each per-file pytest run spawned by - tools/conftest.py splits its tests across. Empty or 1 runs every file serially. - default: '' - required: false curobo-only: description: 'Run only cuRobo and SkillGen tests (requires the cuRobo Docker image)' default: 'false' @@ -161,14 +155,13 @@ runs: TEST_NODE_IDS_KEY: ${{ inputs.test-node-ids-key }} TEST_PATH: ${{ inputs.test-path }} TEST_K_EXPR_INPUT: ${{ inputs.test-k-expr }} - PYTEST_WORKERS_INPUT: ${{ inputs.pytest-workers }} CI_MARKER_INPUT: ${{ inputs.ci-marker }} VOLUME_MOUNT_SOURCE: ${{ inputs.volume-mount-source }} WARP_CACHE_HOST_DIR: ${{ inputs.warp-cache-host-dir }} WHEELHOUSE_HOST_DIR: ${{ inputs.wheelhouse-host-dir }} WHEELHOUSE_PACKAGES: ${{ inputs.wheelhouse-packages }} run: | - bash .github/actions/run-tests/run_tests.sh "$TEST_PATH" "$RESULT_FILE" "$CONTAINER_NAME" "$IMAGE_TAG" "$REPORTS_DIR" "$PYTEST_OPTIONS" "$FILTER_PATTERN" "$EXCLUDE_PATTERN" "$CUROBO_ONLY" "$INCLUDE_FILES" "$QUARANTINED_ONLY" "$SHARD_INDEX" "$SHARD_COUNT" "$VOLUME_MOUNT_SOURCE" "$EXTRA_PIP_PACKAGES" "$TEST_NODE_IDS_FILE" "$TEST_NODE_IDS_KEY" "$WHEELHOUSE_HOST_DIR" "$WHEELHOUSE_PACKAGES" "$TEST_K_EXPR_INPUT" "$CI_MARKER_INPUT" "$STANDALONE_SCRIPT_SCOPE" "$STANDALONE_SCRIPT_VISUALIZER" "$STANDALONE_SCRIPT_RUNTIME_GROUP" "$WARP_CACHE_HOST_DIR" "$EXTRA_UV_PACKAGES" "$OVRTX_SHADER_CACHE_HOST_DIR" "$PYTEST_WORKERS_INPUT" + bash .github/actions/run-tests/run_tests.sh "$TEST_PATH" "$RESULT_FILE" "$CONTAINER_NAME" "$IMAGE_TAG" "$REPORTS_DIR" "$PYTEST_OPTIONS" "$FILTER_PATTERN" "$EXCLUDE_PATTERN" "$CUROBO_ONLY" "$INCLUDE_FILES" "$QUARANTINED_ONLY" "$SHARD_INDEX" "$SHARD_COUNT" "$VOLUME_MOUNT_SOURCE" "$EXTRA_PIP_PACKAGES" "$TEST_NODE_IDS_FILE" "$TEST_NODE_IDS_KEY" "$WHEELHOUSE_HOST_DIR" "$WHEELHOUSE_PACKAGES" "$TEST_K_EXPR_INPUT" "$CI_MARKER_INPUT" "$STANDALONE_SCRIPT_SCOPE" "$STANDALONE_SCRIPT_VISUALIZER" "$STANDALONE_SCRIPT_RUNTIME_GROUP" "$WARP_CACHE_HOST_DIR" "$EXTRA_UV_PACKAGES" "$OVRTX_SHADER_CACHE_HOST_DIR" - name: Kill container on cancellation if: cancelled() shell: bash diff --git a/.github/actions/run-tests/run_tests.sh b/.github/actions/run-tests/run_tests.sh index fc1fac63f037..655f3a3919f3 100755 --- a/.github/actions/run-tests/run_tests.sh +++ b/.github/actions/run-tests/run_tests.sh @@ -38,7 +38,6 @@ run_tests() { local warp_cache_host_dir="${25}" local extra_uv_packages="${26}" local ovrtx_shader_cache_host_dir="${27}" - local pytest_workers="${28}" local logs_pid="" local wait_pid="" local docker_wait_file="/tmp/.docker_exit_${container_name}" @@ -195,11 +194,6 @@ run_tests() { echo "Setting per-file pytest -k expression: $test_k_expr" fi - if [ -n "$pytest_workers" ]; then - docker_env_args+=(-e "TEST_PYTEST_WORKERS=$pytest_workers") - echo "Setting TEST_PYTEST_WORKERS=$pytest_workers" - fi - if [ -n "$ci_marker" ]; then docker_env_args+=(-e "CI_MARKER=$ci_marker") echo "Setting CI_MARKER=$ci_marker" diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index e4c7ba26d5aa..3ddad1bb5f45 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -631,9 +631,6 @@ jobs: isaacsim-base-image: ${{ needs.config.outputs.isaacsim_image_name }} isaacsim-version: ${{ needs.config.outputs.isaacsim_image_tag }} filter-pattern: "isaaclab_newton" - # Split each test file across worker processes. Four fit the g7.4xlarge runner: - # each worker holds its own simulation, ~1 GB of GPU and ~3.5 GB of host memory. - pytest-workers: "4" warp-cache: restore container-name: isaac-lab-newton-test diff --git a/.github/workflows/tools-tests.yml b/.github/workflows/tools-tests.yml index 950565da45ca..0903ebea5727 100644 --- a/.github/workflows/tools-tests.yml +++ b/.github/workflows/tools-tests.yml @@ -52,9 +52,8 @@ jobs: # flaky, so they run without an Isaac Sim install or a full project sync. flaky drives the # rerun in test_crash_during_a_flaky_retry_is_blamed_on_the_retried_test; without it # installed that test skips itself rather than failing, so keep it in this list. - # pytest-xdist likewise drives test_an_xdist_run_journals_each_event_once. - name: Install test dependencies - run: bash "$GITHUB_WORKSPACE/.github/actions/_lib/with-python-package-retries.sh" python3 -m pip install pytest pytest-xdist junitparser flaky pyyaml + run: bash "$GITHUB_WORKSPACE/.github/actions/_lib/with-python-package-retries.sh" python3 -m pip install pytest junitparser flaky pyyaml - name: Run tools tests env: diff --git a/conftest.py b/conftest.py index 05b3721a43be..d5bf93c8ca11 100644 --- a/conftest.py +++ b/conftest.py @@ -51,19 +51,10 @@ def _journal_write(record: dict) -> None: The per-record flush is the whole point: it puts the data in the OS page cache before the next test starts, so a process killed by a signal cannot take down verdicts it had already reported. Journaling failures are swallowed — losing debug context must never fail a run. - - Under ``pytest-xdist`` the controller receives every worker's start, report and finish, so only - it journals those; each worker journaling too would record every event twice and leave a - worker crash that xdist recovered from looking like an in-flight test. The controller never - collects, so the ``collected`` record comes from the first worker instead (every worker - collects the same items). """ path = os.environ.get(JOURNAL_ENV_VAR) if not path: return - worker = os.environ.get("PYTEST_XDIST_WORKER") - if worker and (record["event"] != "collected" or worker != "gw0"): - return try: with open(path, "a", encoding="utf-8") as handle: handle.write(json.dumps(record, separators=(",", ":")) + "\n") diff --git a/pyproject.toml b/pyproject.toml index 8ecc273c0e98..493b18cd78eb 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -126,8 +126,6 @@ importers = [ test = [ "pytest", "pytest-mock", - # Splits a test file across worker processes; see TEST_PYTEST_WORKERS in tools/conftest.py. - "pytest-xdist", "junitparser", "flaky", # numba subclasses coverage.types.Tracer at import; >=7.6.1 restores that shim diff --git a/tools/conftest.py b/tools/conftest.py index ba93a2c7347c..cd0c2ac9237d 100644 --- a/tools/conftest.py +++ b/tools/conftest.py @@ -49,21 +49,12 @@ def pytest_ignore_collect(collection_path, config): AppLauncher prints ``[ISAACLAB] AppLauncher initialization complete`` to ``sys.__stderr__`` (never suppressed) when Kit finishes initializing, and pytest -prints ``collected N items`` to stdout after collection (``N workers [M items]`` -under ``pytest-xdist``, once every worker has collected). If none appears +prints ``collected N items`` to stdout after collection. If neither appears within this deadline the process is treated as hung. Kit startup can exceed 60 s on cold CI workers, so this catches real startup hangs without killing legitimate slow launches. """ -PYTEST_WORKERS_ENV_VAR = "TEST_PYTEST_WORKERS" -"""Environment variable naming the number of ``pytest-xdist`` workers for each test file. - -Each file still runs in its own pytest process; the workers split that file's tests between them, each in -its own process with its own simulation (and Kit app, for files that launch one). Unset or ``1`` runs -the file serially. -""" - STARTUP_HANG_RETRIES = 2 """Number of times to retry a test that hangs during startup before giving up.""" @@ -211,32 +202,6 @@ def _drain_ready_output(process, stdout_fd, stderr_fd, timeout=0.1): return stdout_chunk, stderr_chunk -def _pytest_workers(env) -> int: - """Return the ``pytest-xdist`` worker count configured in ``env``, or 0 to run serially.""" - try: - workers = int(env.get(PYTEST_WORKERS_ENV_VAR, "") or 0) - except ValueError: - return 0 - return workers if workers > 1 else 0 - - -def _child_pids(pid: int) -> list[int]: - """Return the direct children of ``pid``, or an empty list where ``/proc`` is unavailable.""" - children = [] - for entry in os.listdir("/proc") if os.path.isdir("/proc") else []: - if not entry.isdigit(): - continue - try: - with open(f"/proc/{entry}/stat") as handle: - # The command name is parenthesized and may contain spaces; the parent PID follows it. - parent = int(handle.read().rsplit(")", 1)[1].split()[1]) - except (OSError, IndexError, ValueError): - continue - if parent == pid: - children.append(int(entry)) - return sorted(children) - - def _dump_hung_process_stacks(process, stdout_fd, stderr_fd, env): """Ask a hung process for a stack of every thread, and collect what it writes. @@ -246,9 +211,7 @@ def _dump_hung_process_stacks(process, stdout_fd, stderr_fd, env): The signal goes to the test process itself rather than its group. The handler is registered there, and a standalone script the test launched as a grandchild has no handler -- ``SIGUSR1`` would simply kill it, - losing it from the process tree the caller has already recorded. Under ``pytest-xdist`` the tests run in - the worker processes, the controller's direct children, so each worker is asked in turn as well; one at - a time, because they all append to the same dump file. + losing it from the process tree the caller has already recorded. Args: process: The hung child. @@ -270,28 +233,23 @@ def _dump_hung_process_stacks(process, stdout_fd, stderr_fd, env): if hang_dump.DUMP_SIGNAL is None or not dump_file: return "", stdout_data, stderr_data - targets = [process.pid] - if _pytest_workers(env): - targets += _child_pids(process.pid) - for _ in range(HANG_DUMP_PASSES): - for pid in targets: - # Only this request's share of the file is the dump it asked for. - start = hang_dump.size(dump_file) - try: - os.kill(pid, hang_dump.DUMP_SIGNAL) - except OSError: - continue + # Only this pass's share of the file is the dump it asked for. + start = hang_dump.size(dump_file) + try: + os.kill(process.pid, hang_dump.DUMP_SIGNAL) + except OSError: + break - # Keep draining while the handler runs, so a full pipe cannot be what stops it answering. - deadline = time.time() + HANG_DUMP_GRACE - while time.time() < deadline: - stdout_chunk, stderr_chunk = _drain_ready_output(process, stdout_fd, stderr_fd) - stdout_data += stdout_chunk - stderr_data += stderr_chunk + # Keep draining while the handler runs, so a full pipe cannot be what stops it answering. + deadline = time.time() + HANG_DUMP_GRACE + while time.time() < deadline: + stdout_chunk, stderr_chunk = _drain_ready_output(process, stdout_fd, stderr_fd) + stdout_data += stdout_chunk + stderr_data += stderr_chunk - if dumped := hang_dump.read_since(dump_file, start): - dumps.append(dumped if len(targets) == 1 else f"(pid {pid})\n{dumped}") + if dumped := hang_dump.read_since(dump_file, start): + dumps.append(dumped) # exit early if the process died if process.poll() is not None: break @@ -362,11 +320,7 @@ def capture_test_output_with_timeout(cmd, timeout, env, startup_deadline=0, repo elapsed = time.time() - start_time if not startup_done: - if ( - b"AppLauncher initialization complete" in stderr_data - or b"collected " in stdout_data - or b" workers [" in stdout_data - ): + if b"AppLauncher initialization complete" in stderr_data or b"collected " in stdout_data: startup_done = True if report_file and not shutdown_deadline and os.path.exists(report_file): @@ -1039,8 +993,6 @@ def _run_one_pass( cmd += ["-p", "mgpu_shard_select"] if ctx.ci_marker: cmd += ["-m", ctx.ci_marker] - if workers := _pytest_workers(ctx.env): - cmd += ["-n", str(workers)] if k_expr is not None: cmd += ["-k", k_expr] cmd += ctx.pytest_targets diff --git a/tools/hang_dump.py b/tools/hang_dump.py index 9fef9b8b4b95..ad519be57c59 100644 --- a/tools/hang_dump.py +++ b/tools/hang_dump.py @@ -107,10 +107,8 @@ def register(): path = dump_path() if not path or not is_supported(): return False - # pytest-xdist workers share the controller's dump file, which the controller already truncated. - mode = "a" if os.environ.get("PYTEST_XDIST_WORKER") else "w" try: - _dump_file = open(path, mode) # noqa: SIM115 (held open for the process lifetime, see above) + _dump_file = open(path, "w") # noqa: SIM115 (held open for the process lifetime, see above) except OSError: return False faulthandler.register(DUMP_SIGNAL, file=_dump_file, all_threads=True, chain=False) diff --git a/tools/test_crash_journal.py b/tools/test_crash_journal.py index a07e6f471b8d..86eb6531c1c8 100644 --- a/tools/test_crash_journal.py +++ b/tools/test_crash_journal.py @@ -429,42 +429,6 @@ def test_drop(): assert read_journal(str(journal_file)).collected == [f"{_FILE}::test_keep"] -def test_an_xdist_run_journals_each_event_once(tmp_path): - """Regression test for ``pytest-xdist`` runs journaling every event twice. - - The workers and the controller both fire the per-test hooks, and every worker fires the - collection hook. Duplicated starts turn a worker crash that xdist recovered from into an - unmatched start, so a later session crash would be blamed on a test that already reported. - """ - pytest.importorskip("xdist") - _write_test_module( - tmp_path, - """ - def test_a(): - pass - - def test_b(): - assert 1 == 2 - - def test_c(): - pass - """, - ) - journal_file = tmp_path / "journal.jsonl" - junit_file = tmp_path / "report.xml" - _run_pytest(tmp_path, journal_file, junit_file, "-p", "xdist.plugin", "-n", "2") - - records = [json.loads(line) for line in journal_file.read_text(encoding="utf-8").splitlines()] - node_ids = [f"{_FILE}::test_{name}" for name in "abc"] - assert [record["event"] for record in records].count("collected") == 1 - for event in ("start", "result", "finish"): - assert sorted(record["node_id"] for record in records if record["event"] == event) == node_ids - - journal = read_journal(str(journal_file)) - assert journal.collected == node_ids - assert journal.culprit is None - - # -- artificial crashes in a real pytest run -------------------------------------------------- diff --git a/uv.lock b/uv.lock index b341a9b1cbc8..20a39638029a 100644 --- a/uv.lock +++ b/uv.lock @@ -1105,15 +1105,6 @@ epath = [ { name = "zipp" }, ] -[[package]] -name = "execnet" -version = "2.1.2" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/bf/89/780e11f9588d9e7128a3f87788354c7946a9cbb1401ad38a48c4db9a4f07/execnet-2.1.2.tar.gz", hash = "sha256:63d83bfdd9a23e35b9c6a3261412324f964c2ec8dcd8d3c6916ee9373e0befcd", size = 166622, upload-time = "2025-11-12T09:56:37.75Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/ab/84/02fc1827e8cdded4aa65baef11296a9bbe595c474f0d6d758af082d849fd/execnet-2.1.2-py3-none-any.whl", hash = "sha256:67fba928dd5a544b783f6056f449e5e3931a5c378b128bc18501f7ea79e296ec", size = 40708, upload-time = "2025-11-12T09:56:36.333Z" }, -] - [[package]] name = "executing" version = "2.2.1" @@ -1827,7 +1818,6 @@ dev = [ { name = "myst-parser" }, { name = "pytest" }, { name = "pytest-mock" }, - { name = "pytest-xdist" }, { name = "sphinx" }, { name = "sphinx-book-theme" }, { name = "sphinx-copybutton" }, @@ -1913,7 +1903,6 @@ test = [ { name = "junitparser" }, { name = "pytest" }, { name = "pytest-mock" }, - { name = "pytest-xdist" }, ] tetrahedralization = [ { name = "pytetwild", extra = ["all"] }, @@ -2018,7 +2007,6 @@ requires-dist = [ { name = "pyopengl-accelerate", specifier = ">=3.1.0" }, { name = "pytest", marker = "extra == 'test'" }, { name = "pytest-mock", marker = "extra == 'test'" }, - { name = "pytest-xdist", marker = "extra == 'test'" }, { name = "pytetwild", extras = ["all"], marker = "extra == 'tetrahedralization'", specifier = ">=0.3.0,<0.4" }, { name = "ray", extras = ["default"], marker = "extra == 'rlinf'", specifier = ">=2.47.0" }, { name = "requests", specifier = ">=2.25.0" }, @@ -4417,19 +4405,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/5a/cc/06253936f4a7fa2e0f48dfe6d851d9c56df896a9ab09ac019d70b760619c/pytest_mock-3.15.1-py3-none-any.whl", hash = "sha256:0a25e2eb88fe5168d535041d09a4529a188176ae608a6d249ee65abc0949630d", size = 10095, upload-time = "2025-09-16T16:37:25.734Z" }, ] -[[package]] -name = "pytest-xdist" -version = "3.8.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "execnet" }, - { name = "pytest" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/78/b4/439b179d1ff526791eb921115fca8e44e596a13efeda518b9d845a619450/pytest_xdist-3.8.0.tar.gz", hash = "sha256:7e578125ec9bc6050861aa93f2d59f1d8d085595d6551c2c90b6f4fad8d3a9f1", size = 88069, upload-time = "2025-07-01T13:30:59.346Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/ca/31/d4e37e9e550c2b92a9cbc2e4d0b7420a27224968580b5a447f420847c975/pytest_xdist-3.8.0-py3-none-any.whl", hash = "sha256:202ca578cfeb7370784a8c33d6d05bc6e13b4f25b5053c30a152269fd10f0b88", size = 46396, upload-time = "2025-07-01T13:30:56.632Z" }, -] - [[package]] name = "pytetwild" version = "0.3.0"