[4C/10] Route rigid rendering transforms through SDP

[ooctipus]

Description

Route rigid-body transforms from physics to renderers through SceneDataProvider (SDP). Physics publishes its native buffer and increments its version after changes. SDP binds that buffer when format and ordering match, or converts it once per requested layout and reuses the result until the next change. Readers never reset the producer's version.

What changes

• OVRTX: converts physics poses directly into its matrix format, without first copying them into a Newton render state.
• Newton rendering: borrows SDP's transform buffer instead of copying into a second buffer. Newton physics and rendering continue sharing their existing model/state.
• Isaac RTX: shares one registry-owned Fabric binding with Kit, preserves authored scale, and updates the transform hierarchy on GPU. PhysX publishes its native Fabric matrices without fetching packed poses.
• Ownership: physics refreshes native data; SDP borrows or converts arrays. One FabricBackend in isaaclab_physx owns the stage/hierarchy handles and shared transform bindings, identified by stage/device. Consumers pass SDP explicitly to updates; it is not part of backend identity. Core RenderContext has no Fabric methods or state. FabricMatrix44 contains only matrix storage.
• Rendering updates: removes repeated renderer-driven physics refreshes. Writes made between physics steps remain visible on the next render.

For Newton → Isaac RTX, cached bindings and GPU hierarchy updates replace repeated binding setup and the CPU hierarchy fallback. This is the main runtime saving measured below.

Particle/deformable transport is outside this change. OVRTX retains its existing Newton geometry bridge.

Transform movement

Counts are SDP output-writing passes, not total SDK-internal copies. Conversion, reordering, and scale are combined in one pass; unchanged data reuses the result.

Physics	Renderer	Published → requested format	SDP passes
Newton	Newton Warp	Transform → Transform	0
Newton	OVRTX	Transform → TransposedMatrix44d	1
Newton	Isaac RTX	Transform → FabricMatrix44	1
OVPhysX	Newton Warp	Transform → Transform	0 if ordering matches; otherwise 1
OVPhysX	OVRTX	Transform → TransposedMatrix44d	1
Isaac PhysX	Newton Warp	Transform → Transform	0 if ordering matches; otherwise 1
Isaac PhysX	Isaac RTX	Native Fabric → borrowed Fabric	0 with use_fabric=True

OVPhysX and OVRTX cannot run with Kit. Fabric hierarchy propagation and OVRTX's native attribute write happen after the SDP pass and are not included in these counts.

Performance

Kuka Allegro Camera, 4096 environments, 64×64 RGB, Newton MJWarp → Isaac RTX, RTX 5090, no interactive visualizer. Two warm-cache runs per revision; runtime excludes 25 warmup steps and measures 200 synchronized full environment steps, including rendering and observations.

Metric	PR	Develop
Warm startup	135.47 s	133.19 s
Runtime step	172.32 ms	554.49 ms
Environment frames/s	23,770	7,387

The measured step time was 68.9% lower (3.22× throughput). No startup improvement was measured. All 4096 camera images were finite and nonconstant.

Measured revisions: PR 257e54d5f and develop 53a7f1c0a, with identical dependencies. Subsequent cleanup has not been rebenchmarked.

Migration

Custom scene-data backends must initialize transforms_version=0 and increment it after native pose writes or buffer swaps. The consumer-facing get_transforms(output) API binds shared, read-only arrays, including converted outputs. Pass allow_passthrough=False for caller-owned writable or preallocated arrays; conversion writes directly into them. Single-format backends keep their existing transforms property; the base get_transforms(output_format) delegates to it. Multi-format backends may override that method and native_transform_formats.

Validation

• Focused CPU tests and GPU 0 Newton/PhysX Fabric tests cover pointer sharing, publication versions, transform formats, ordering, authored scale, buffer reallocation, same-step writes, and resets. The existing cache regression now also checks that independent SDP readers cannot hide producer changes from one another.
• All 11 native Newton/Fabric tests and the native PhysX Fabric test passed after moving bindings out of core. They also check one shared RTX/Kit resource and explicit Newton-to-Fabric updates without an RTX camera or Kit viewer.
• Removed obsolete synchronization tests and duplicate mock-only checks; native and numerical regressions remain.
• Native OVPhysX → OVRTX rendered scale, pose changes, and camera calibration passed for both legacy and ovstage APIs. Matched OVRTX timings have not been collected.

Type of change

• Refactor and bug fixes
• Breaking custom scene-data backend interface change, with migration above

Release backport

• Backport to the active release branch

Checklist

• Contribution guidelines reviewed
• Changelog fragments and migration documentation updated
• Retained focused tests and formatting checks passed after test cleanup
• Full GPU CI passed for the latest revision
• Native OVPhysX → OVRTX rendering validated

[ooctipus]

run-ci

[ooctipus]

run-ci

[ooctipus]

run-ci

[ooctipus]

run-ci

[ooctipus]

run-ci

[ooctipus]

run-ci

[ooctipus]

run-ci

[StafaH]

Thanks Octi!

Looking through it at a high level, I noticed a couple of _dirty that is in different places. This might turn into a problem of whack-a-mole with dirty flags. What if we maybe did something like:

• Producers do backend.transforms_version += 1 after a write or pointer swap, and nothing resets it. Each consumer compares against the last version it saw.
• Then we can remove all the _dirty and SceneDatProvider.transform_generator

Second is that for request_transforms, maybe we should set-up in a way that does all the setup once, and then in the future it is just a call.

There could be 2 ways to do it:

Functional:

  read = sdp.transform_reader(SceneDataFormat.TransposedMatrix44d, scales=scales)  # validate, allocate, pick kernel once
  transforms, version = read()  # converts at most once per version

  publish = sdp.fabric_sink(stage, device)  # replaces _prepare_fabric + FabricMatrix44
  publish()
  Wrong arguments now fail at setup instead of mid-render. The Fabric path no longer needs private fields on the provider, and
  FabricMatrix44 goes away since every caller discarded its return value.

Or slightly less functional but maybe more user friendly:

  view = sdp.transform_view(SceneDataFormat.TransposedMatrix44d, scales=scales)  # validate, allocate, pick kernel once
  transforms, version = view.read()                                              # converts at most once per version

  publisher = sdp.fabric_publisher(stage, device)  # replaces _prepare_fabric + FabricMatrix44
  publisher.publish()

[StafaH]

_last_geometry_step is a bit odd of a term, reading this first time I would now know what it refers to

[StafaH]

There is alot of usage of this output._cls, complex to read

[StafaH]

Should we remove the main function here?

[StafaH]

We should have the test use ovrtx preferable, as that path matters more to us

[StafaH]

Let's remove this from AGENTS.md, if such a note is required, leave it near the implementation of SDP refresh publication and agents will naturally pick up the rule when they write code requiring it.

[ooctipus]

@StafaH Agreed on producer-owned versions. Replaced the transform dirty latches and SDP generation counter with backend.transforms_version. Readers never reset it. PhysX retains separate last-read versions for packed poses and native Fabric, since either representation can be requested independently.

This fixed a concrete stale-data case: with two SDP readers, the first previously consumed the flag before the second refreshed its converted buffer. The existing regression now covers both readers. Pending FK flags remain because they represent work that still needs execution, not publication freshness.

The Fabric part has changed since your example: RenderContext now owns its bindings and hierarchy updates, while SDP only borrows or converts array storage. Moving a Fabric publisher back into SDP would undo that ownership correction.

I kept the single get_transforms consumer API for this update. Buffers and mappings are reused, and unchanged conversions now skip kernel lookup too, but this is not yet a setup-once reader factory. That proposal should replace the existing API coherently, not add a second interface alongside it.

Also merged latest develop and resolved the conflict.

Addressed the inline cleanup too: removed the embedded __main__ demo and the SDP-specific AGENTS.md section, renamed the geometry cadence field to _last_geometry_update_step, and named Warp's descriptors source_format / output_format instead of repeatedly reading _cls.

For the test comment, the cited test asserts actual Kit/Fabric matrices, so it needs Isaac RTX; OVRTX cannot run with Kit. I also ran the existing native OVPhysX → OVRTX image tests on GPU 0: both legacy and ovstage passed, checking scale, pose changes, and camera calibration. The 5 PhysX publication checks and 4 Newton/Fabric integration cases also passed, including captured writes followed by rendering without a physics step.

[ooctipus]

run-ci

[ooctipus]

run-ci

[ooctipus]

run-ci

[ooctipus]

run-ci