Skip to content

Fix cloned camera bindings and moving-camera poses for OVRTX batching - #3

Merged
pbarejko merged 1 commit into
pbarejko:pbarejko/batch-renderingfrom
nblauch:fix/ovrtx-batched-camera-updates
Sep 23, 2026
Merged

pbarejko merged 1 commit into
pbarejko:pbarejko/batch-renderingfrom
nblauch:fix/ovrtx-batched-camera-updates

Conversation

@nblauch

@nblauch nblauch commented Sep 22, 2026 •

Copy link
Copy Markdown

Newton can author only the source camera in USD while OVRTX clones it across environments. Additional cameras on this branch were binding only that source path, leaving the other environments' views incorrect. Camera intrinsics had the same path mismatch. Moving cameras also need pose refresh enabled because OVRTX renders from the cached camera pose.

This PR against pbarejko/batch-rendering:

  • Expands source-only camera paths at registration, preserving nested paths and already-expanded inputs. Every camera's pose and intrinsic bindings then cover its logical environments.
  • Enables pose refresh for the shared Kuka-Allegro wrist camera and Galbot wrist/head cameras. Documents the explicit-pose renderer requirement.
  • Extends the existing CPU and GPU camera tests for source-only registration and calibration writes, with and without OVStage.

Validation on an RTX 6000 Ada with Newton 1.6.0 and OVRTX 0.5.0.377615:

  • Regression demonstrated before the fix: both source-only CPU cases failed; the GPU test rendered incorrect depth for the second camera's cloned environment. All four CPU cases and all four GPU cases passed afterward.
  • Renderer contracts: 49 passed, 2 skipped. Render-context scheduling: 19 passed. uv run isaaclab -f: passed.
  • Kuka lift and reorientation camera smokes passed with four environments, including a moving wrist and different camera resolutions. Cached poses matched a frozen-physics pose-refresh reference.
  • Galbot's four-camera smoke passed with replicate_physics=True explicitly overridden. Its default configuration remains blocked as described below.

Remaining issues found by the broader audit:

  • The Kuka visual table is missing in OVRTX. The task makes its physical table invisible and draws a red/green replacement using success markers. Headless OVRTX selects no marker backend, whereas IsaacSim RTX does. This predates batching and the binding fix. The earlier renderer videos therefore have different visible geometry; their throughput comparison is not scene-matched.
  • Mixed camera rates still discard temporal history. OVRTX's native step contract discards history for omitted products. Alternating the submitted product set repeatedly removes/recreates the slower camera. In a four-environment diagnostic with base 64×64 and wrist 96×48, both-product native calls averaged 222.18 ms with unequal rates versus 3.01 ms with equal rates. These are instrumented native-call times, not training throughput. Raw static frames repeat a short exact cycle. The limitation also applies to other temporary subset-only submissions. Resolving it requires separate scheduling/lifetime work; this PR preserves the branch's batching behavior.
  • Galbot's default multi-environment Newton configuration fails before rendering. It inherits replicate_physics=False; the Kit-free clone dispatch skips the Newton replication context, leaving authored prototypes while the ClonePlan still describes multiple environments. The fixed camera then fails while resolving those worlds. Enabling replication allowed the camera smoke above, but does not establish that the unchanged task works.

The mixed-rate diagnostic also exposed pre-existing float32 sensor timestamp drift: a nominal two-step camera period eventually slipped to three steps. Replaying the actual physics-substep timestamp updates explains all observed frame counters; no requested native frames were missing.

@nblauch
nblauch requested a review from pbarejko as a code owner September 22, 2026 23:31
@github-actions github-actions Bot added bug Something isn't working isaac-lab labels Sep 22, 2026
@pbarejko
pbarejko merged commit bb2ad5f into pbarejko:pbarejko/batch-rendering Sep 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working isaac-lab

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants