[Backport release/3.0.0] Rendering, backend writers, and Newton tests (#7941, #7977, #7983, #7991, #8003) by StafaH · Pull Request #8002 · isaac-sim/IsaacLab

StafaH · 2026-09-24T16:16:28Z

Description

Backports the rendering changes and related release fixes:

[4C/10] Route rigid rendering transforms through SDP #7941 — routes rigid rendering transforms through SceneDataProvider and preserves shared Fabric bindings.
Batch camera rendering and fix moving-camera and cloned-camera updates #7977 — batches due camera captures by renderer and fixes moving and cloned camera updates. This supersedes [Backport] PR #7977 to release/3.0.0 #8001.
Remove camera_path_relative_to_env_0 #7983 — removes CameraRenderSpec.camera_path_relative_to_env_0 and derives and validates the OVRTX source camera path from camera_prim_paths. The rendering changes were developed with this interface, so it is required by their OVRTX tests.
[Core] Refactor and cleanup core subpackage for 3.0 [5C/N] #7991 — fixes Newton's deprecated joint friction writers and the PhysX rigid object collection body_mask writer signatures.
Consolidate redundant Newton tests #8003 — consolidates redundant Newton tests to shorten the Newton CI job without changing runtime code.

#7941 required conflict resolution in the OVRTX legacy and OvStage transform binding functions: the release branch's Newton model path collection was replaced by SDP-published paths. #7983 was then integrated with those SDP paths. Each added commit records its source commit.

Custom scene-data backends must follow the transforms_version publication contract described in the included migration documentation. Direct callers of BaseRenderer.render() must pass a sequence, for example render([data]). Callers constructing CameraRenderSpec must remove the camera_path_relative_to_env_0 argument; OVRTX now derives it from camera_prim_paths.

Validation

Rendering and scene-data focused tests from the original backport passed locally: 27 renderer tests and 50 scene-data / PhysX renderer tests. OVRTX runtime cases were skipped locally because the optional ovrtx module is unavailable.
[Core] Refactor and cleanup core subpackage for 3.0 [5C/N] #7991: the Newton deprecated friction writer test passed, and 72 rigid object collection mask writer cases passed. The new friction test fails without the fix and passes with it.
Consolidate redundant Newton tests #8003: all 8 revised contact lifecycle cases passed. Kit-based Newton asset tests remain for CI.
uv run isaaclab -f passed after all additions with the changelog base set to release/3.0.0, as did git diff --check.
The original backport's documentation build passed with warnings treated as errors. Full GPU and Kit CI is running on the updated PR.

Type of change

Rendering bug fixes and performance improvement
Backend API bug fixes and breaking renderer / scene-data interface changes with migration guidance
Newton test consolidation
Documentation and changelog fragments for affected source packages

Release backport

This PR targets release/3.0.0; no further backport is needed.

Checklist

Contribution guidance reviewed
Focused local tests and formatting checks passed
Public documentation and package changelog fragments included
Full GPU and Kit CI passed

Route rigid-body transforms from physics to renderers through `SceneDataProvider` (SDP). Physics publishes its native buffer and increments its version after changes. SDP binds that buffer when format and ordering match, or converts it once per requested layout and reuses the result until the next change. Readers never reset the producer's version. - **OVRTX:** converts physics poses directly into its matrix format, without first copying them into a Newton render state. - **Newton rendering:** borrows SDP's transform buffer instead of copying into a second buffer. Newton physics and rendering continue sharing their existing model/state. - **Isaac RTX:** shares one registry-owned Fabric binding with Kit, preserves authored scale, and updates the transform hierarchy on GPU. PhysX publishes its native Fabric matrices without fetching packed poses. - **Ownership:** physics refreshes native data; SDP borrows or converts arrays. One `FabricBackend` in `isaaclab_physx` owns the stage/hierarchy handles and shared transform bindings, identified by stage/device. Consumers pass SDP explicitly to updates; it is not part of backend identity. Core `RenderContext` has no Fabric methods or state. `FabricMatrix44` contains only matrix storage. - **Rendering updates:** removes repeated renderer-driven physics refreshes. Writes made between physics steps remain visible on the next render. For Newton → Isaac RTX, cached bindings and GPU hierarchy updates replace repeated binding setup and the CPU hierarchy fallback. This is the main runtime saving measured below. Particle/deformable transport is outside this change. OVRTX retains its existing Newton geometry bridge. Counts are SDP output-writing passes, not total SDK-internal copies. Conversion, reordering, and scale are combined in one pass; unchanged data reuses the result. | Physics | Renderer | Published → requested format | SDP passes | | --- | --- | --- | ---: | | Newton | Newton Warp | `Transform` → `Transform` | 0 | | Newton | OVRTX | `Transform` → `TransposedMatrix44d` | 1 | | Newton | Isaac RTX | `Transform` → `FabricMatrix44` | 1 | | OVPhysX | Newton Warp | `Transform` → `Transform` | 0 if ordering matches; otherwise 1 | | OVPhysX | OVRTX | `Transform` → `TransposedMatrix44d` | 1 | | Isaac PhysX | Newton Warp | `Transform` → `Transform` | 0 if ordering matches; otherwise 1 | | Isaac PhysX | Isaac RTX | Native Fabric → borrowed Fabric | 0 with `use_fabric=True` | OVPhysX and OVRTX cannot run with Kit. Fabric hierarchy propagation and OVRTX's native attribute write happen after the SDP pass and are not included in these counts. Kuka Allegro Camera, 4096 environments, 64×64 RGB, Newton MJWarp → Isaac RTX, RTX 5090, no interactive visualizer. Two warm-cache runs per revision; runtime excludes 25 warmup steps and measures 200 synchronized full environment steps, including rendering and observations. | Metric | PR | Develop | | --- | ---: | ---: | | Warm startup | 135.47 s | 133.19 s | | Runtime step | 172.32 ms | 554.49 ms | | Environment frames/s | 23,770 | 7,387 | The measured step time was **68.9% lower (3.22× throughput)**. No startup improvement was measured. All 4096 camera images were finite and nonconstant. Measured revisions: PR `257e54d5f` and develop `53a7f1c0a`, with identical dependencies. Subsequent cleanup has not been rebenchmarked. Custom scene-data backends must initialize `transforms_version=0` and increment it after native pose writes or buffer swaps. The consumer-facing `get_transforms(output)` API binds shared, read-only arrays, including converted outputs. Pass `allow_passthrough=False` for caller-owned writable or preallocated arrays; conversion writes directly into them. Single-format backends keep their existing `transforms` property; the base `get_transforms(output_format)` delegates to it. Multi-format backends may override that method and `native_transform_formats`. - Focused CPU tests and GPU 0 Newton/PhysX Fabric tests cover pointer sharing, publication versions, transform formats, ordering, authored scale, buffer reallocation, same-step writes, and resets. The existing cache regression now also checks that independent SDP readers cannot hide producer changes from one another. - All 11 native Newton/Fabric tests and the native PhysX Fabric test passed after moving bindings out of core. They also check one shared RTX/Kit resource and explicit Newton-to-Fabric updates without an RTX camera or Kit viewer. - Removed obsolete synchronization tests and duplicate mock-only checks; native and numerical regressions remain. - Native OVPhysX → OVRTX rendered scale, pose changes, and camera calibration passed for both legacy and ovstage APIs. Matched OVRTX timings have not been collected. - Refactor and bug fixes - Breaking custom scene-data backend interface change, with migration above - [x]  Backport to the active release branch - [x] Contribution guidelines reviewed - [x] Changelog fragments and migration documentation updated - [x] Retained focused tests and formatting checks passed after test cleanup - [ ] Full GPU CI passed for the latest revision - [x] Native OVPhysX → OVRTX rendering validated (cherry picked from commit 22bfa0d)

isaac-sim#7977) (cherry picked from commit 1e520e9)

StafaH · 2026-09-24T16:33:22Z

run-ci

(cherry picked from commit ef821d9)

(cherry picked from commit 94aa09f)

(cherry picked from commit 2b62263)

(cherry picked from commit d696cd0)

StafaH · 2026-09-24T19:12:28Z

run-ci

Keep isaac-sim#8006 out of the release backport until its source PR merges.

StafaH · 2026-09-24T19:16:31Z

run-ci

greptile-apps · 2026-09-24T19:30:54Z

Confidence Score: 4/5

The PR appears safe to merge, with non-blocking frame-retry bookkeeping and test-coverage issues worth addressing.

Findings

Frames advance on failed batches ▶
Material-write coverage is removed ▶

Summary

The PR backports shared scene-data transform publication and Fabric bindings, batches camera captures by renderer, fixes backend writer signatures, and consolidates Newton tests.

Batched capture needs consistent frame bookkeeping when a submission fails.
The Newton test reduction drops distinct material-write coverage.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Physics[Physics backend] --> SDP[SceneDataProvider]
  SDP --> Fabric[Shared Fabric binding]
  SDP --> OVRTX[OVRTX transforms]
  Scene[InteractiveScene update] --> Batch[Due camera batch]
  Batch --> Context[RenderContext]
  Context --> Renderers[Renderer batches]
  Fabric --> Renderers
  Renderers --> Outputs[Camera outputs]

_{Reviews (1) · Last reviewed commit: "Revert premature Newton CI backport (#80..."}

greptile-apps · 2026-09-24T19:30:58Z

+        for camera in cameras:
+            if not camera._env_mask_has_any(camera._is_outdated):
+                continue
+            camera._prepare_camera(camera._is_outdated)
+            ready.append(camera)


Frames advance on failed batches When a batched render or output read fails, these cameras have already advanced their frame counters. The batch then leaves them marked for retry, so a retry advances the counters again for the same capture period. This can make camera frame metadata inaccurate, including for a peer camera whose capture completed before another camera failed.

Knowledge Base Used: Simulation, rendering, and sensors

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

greptile-apps · 2026-09-24T19:30:58Z

@@ -0,0 +1 @@
+Consolidated redundant Newton tests and trimmed device and shape matrices that repeated the same code paths.


Material-write coverage is removed This consolidation removes the rigid-object and rigid-object-collection tests that write and read back friction and restitution, without keeping equivalent checks. Those tests cover distinct view-binding behavior, so material-write regressions will no longer be caught.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

isaaclab-review-bot

Isaac Lab Review Bot

Reviewed the camera batching, SDP transform publication, shared Fabric ownership, OVRTX camera-path derivation, and backend writer fixes. The changed producer/consumer paths are migrated consistently in the supplied patch, and no candidate finding identifies a concrete defect requiring maintainer action.

Design and architecture: The additive batching design preserves fallback behavior through BaseRenderer.render_batch(), while SensorBase opt-in grouping and RenderContext renderer grouping keep eager batching separate from lazy camera reads. Fabric synchronization is centralized in a simulation-owned FabricBackend and transform freshness moves to producer versions rather than physics-step cadence.
API: Checked CameraRenderSpec removal, the SceneDataBackend transforms_version contract, shared read-only transform outputs, SensorBase batching APIs, and PhysX collection mask signatures. In-tree callers and backends are migrated, and changelog and migration documentation describe the immediate external caller changes; compatibility therefore relies on consumers following that release guidance.
Implementation: Traced eager camera updates through InteractiveScene, SensorBase._process_batch, Camera preparation, RenderContext grouping, backend render_batch implementations, and output readback. Also traced SDP version-keyed conversions through PhysX, Newton, OVPhysX, OVRTX, Isaac RTX, and Kit Fabric consumers, including authored-scale preservation and writable-output opt-out behavior.

No blocking issues. No inline issue met the actionable-evidence threshold; the assessment above records the review feedback.

Automated review; human maintainers own approval decisions.

ooctipus and others added 3 commits September 24, 2026 09:12

Batch camera rendering and fix moving-camera and cloned-camera updates (

b92d1d9

isaac-sim#7977) (cherry picked from commit 1e520e9)

Format merged OVRTX contract tests

f480362

github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team labels Sep 24, 2026

StafaH mentioned this pull request Sep 24, 2026

[Backport] PR #7977 to release/3.0.0 #8001

Closed

isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Sep 24, 2026

StafaH added 2 commits September 24, 2026 11:29

Backport camera source path cleanup (isaac-sim#7983)

9852926

(cherry picked from commit ef821d9)

Backport parallel Newton CI tests (isaac-sim#8006)

99ac675

(cherry picked from commit 94aa09f)

github-actions Bot added the infrastructure label Sep 24, 2026

StafaH added 2 commits September 24, 2026 11:34

Backport backend writer signature fixes (isaac-sim#7991)

7ed4a17

(cherry picked from commit 2b62263)

Backport consolidated Newton tests (isaac-sim#8003)

51214a8

(cherry picked from commit d696cd0)

isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Sep 24, 2026

StafaH changed the title ~~[Backport release/3.0.0] Route rendering transforms and batch cameras (#7941, #7977)~~ [Backport release/3.0.0] Rendering, backend writers, and Newton CI (#7941, #7977, #7983, #7991, #8003, #8006) Sep 24, 2026

Revert premature Newton CI backport (isaac-sim#8006)

7a5c834

Keep isaac-sim#8006 out of the release backport until its source PR merges.

StafaH changed the title ~~[Backport release/3.0.0] Rendering, backend writers, and Newton CI (#7941, #7977, #7983, #7991, #8003, #8006)~~ [Backport release/3.0.0] Rendering, backend writers, and Newton tests (#7941, #7977, #7983, #7991, #8003) Sep 24, 2026

isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Sep 24, 2026

StafaH marked this pull request as ready for review September 24, 2026 19:17

StafaH requested a review from a team September 24, 2026 19:17

StafaH requested review from AntoineRichard, fatimaanes, kellyguo11, ooctipus and r-schmitt as code owners September 24, 2026 19:17

StafaH requested review from huidongc, hujc7, marcodiiga, matthewtrepte, pbarejko and rilei-nvidia as code owners September 24, 2026 19:17

greptile-apps Bot reviewed Sep 24, 2026

View reviewed changes

isaaclab-review-bot Bot reviewed Sep 24, 2026

View reviewed changes

StafaH merged commit 85c2144 into isaac-sim:release/3.0.0 Sep 24, 2026
54 checks passed

Navigation Menu

[Backport release/3.0.0] Rendering, backend writers, and Newton tests (#7941, #7977, #7983, #7991, #8003) - #8002

[Backport release/3.0.0] Rendering, backend writers, and Newton tests (#7941, #7977, #7983, #7991, #8003)#8002
StafaH merged 8 commits into
isaac-sim:release/3.0.0from
StafaH:backport/release-3-rendering-7941-7977

StafaH commented Sep 24, 2026 •

edited

Loading

Uh oh!

StafaH commented Sep 24, 2026

Uh oh!

StafaH commented Sep 24, 2026

Uh oh!

StafaH commented Sep 24, 2026

Uh oh!

greptile-apps Bot commented Sep 24, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot Sep 24, 2026

Uh oh!

greptile-apps Bot Sep 24, 2026

Uh oh!

isaaclab-review-bot Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1 @@
		Consolidated redundant Newton tests and trimmed device and shape matrices that repeated the same code paths.

Conversation

StafaH commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Validation

Type of change

Release backport

Checklist

Uh oh!

StafaH commented Sep 24, 2026

Uh oh!

StafaH commented Sep 24, 2026

Uh oh!

StafaH commented Sep 24, 2026

Uh oh!

greptile-apps Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Diagram

Uh oh!

greptile-apps Bot Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

isaaclab-review-bot Bot left a comment

Choose a reason for hiding this comment

Isaac Lab Review Bot

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

StafaH commented Sep 24, 2026 •

edited

Loading

greptile-apps Bot commented Sep 24, 2026 •

edited

Loading