Skip to content

[USD] Export one initialized deployment environment - #7499

Draft
hujc7 wants to merge 56 commits into
isaac-sim:developfrom
hujc7:jichuanh/newton-usd-export
Draft

hujc7 wants to merge 56 commits into
isaac-sim:developfrom
hujc7:jichuanh/newton-usd-export

Conversation

@hujc7

@hujc7 hujc7 commented Sep 2, 2026 •

Copy link
Copy Markdown
Collaborator

Review Map

  1. The child PR page includes its dependency; “Exact changes” isolates the child's own changes.
  2. SHA links are refreshed after branch updates, starting with this parent PR's copy.
PR Status Depends on Exact changes
#7499 — Environment export Draft -- --
#7857 — Deformables and solver settings Draft #7499 Child changes

Summary

Export one complete selected environment with its initialized physical properties and required scene resources. The training option writes environment zero before startup randomization, then training continues.

PXR_WORK_THREAD_LIMIT=1 uv run --frozen --extra rsl-rl isaaclab train \
  --rl_library rsl_rl --task Isaac-Cartpole \
  physics=newton_mjwarp env.scene.num_envs=1 \
  --headless --export_deployment_usd

Writes deployment.usda into the run log directory. Outside training, set env_cfg.scene.export_usd_path before construction. A standalone initialized scene can call scene.export_to_usd("environment.usda", env_id=0, preserve_source_contacts=True) before startup events or stepping.

Description

  1. The scene selects registered articulations, rigid objects, collections and cables through ClonePlan queries, retains static/shared resources and dependencies, and removes other replicas. ClonePlan supplies replication layout, not runtime override history.
  2. Data-property @usd_field(actuator_config="...") declarations now own configuration-to-data bindings alongside USD targets. The actuator framework discovers those inherited declarations for imported-default reads and fixed override selection, then records field/joint/environment identities after backend assignment succeeds. The parallel binding table is removed. No additional value snapshots are retained. None inheritance, partial-dictionary zero fill, explicit-drive disabling and required PhysX schema semantics are preserved. Export reads only selected values and skips equivalent effective opinions before changing schemas, bindings or instanceability.
  3. The shared writer preserves source mass/inertia/COM, geometry, defaults and material bindings. Backend managers identify necessary contact overrides using source/import precedence; matching material overrides share resources. Public data declarations provide schema targets and two angular conversions; meter/kilogram stages are required. MuJoCo limit provenance retains native defaults. No dependency was added.
  4. Automatic export remains after asset initialization/prestartup and before startup/reset/stepping. It keeps required root placement without baking articulation child poses, sampling joint state, or writing/clearing velocities. Arbitrary later setter/buffer changes are not tracked. The existing CLI and construction flow remain; there is no separate export workflow or nominal-value snapshot.
  5. Fixed locomotion materials use shared spawn configuration (static friction 0.8, dynamic friction 0.6, multiply combine modes). Newton friction changes from 0.8 to 0.6; OVPhysX combine modes become multiply. Startup inertia corrections remain unchanged. Removing fixed event terms changes random-number consumption.
  6. Native Newton import plus existing terrain/cable readers preserve supported geometry and contacts. Optional MJWarp settings remain manager-owned; additional solver settings and deformables are in dependent PR #7857.

See the export contract and loading recipes.

Support matrix

Historical results at bfbc42ef74a, before this cleanup. The full task sweep is deferred during cleanup; these numbers are not a revalidation of the current head. Current validation uses focused unit/round-trip tests and USD diff audits, not another task sweep.

Validation on 2026-09-16: 134/142 task–backend combinations passed across 45 task names; 8 combinations explicitly reject deformable export. These are export limitations, separate from cells where the task has no backend preset.

The full 146-job sweep took 12m 15s on 12 GPUs (eight-GPU and four-GPU nodes). A focused 12-job run took 1m 50s, validating all 10 Newton rough-terrain combinations plus Newton and shared/entrypoint suites after adding the terrain contact reader. The table consolidates both runs. The full sweep covered feature commit 6df8777d400; the focused run covered the subsequent dependency merge and terrain-readback follow-up published at bfbc42ef74a.

# Backend / solver Passed Export unsupported
1 Newton MJWarp 38/38 0
2 Newton Kamino 18/18 0
3 Newton MJWarp/VBD proxy 2/6 4
4 OVPhysX 38/38 0
5 IsaacSim PhysX 38/42 4

Complete task × backend results

  1. ✅ Pass — fresh same-backend physical configuration passed.
  2. ❌ Unsupported (D) — deformable export is unsupported in this revision.
  3. -- — the task declares no backend preset; excluded from tested counts.
# Task MJWarp Kamino MJWarp/VBD proxy OVPhysX IsaacSim PhysX
1 Isaac-Ant ✅ Pass ✅ Pass -- ✅ Pass ✅ Pass
2 Isaac-Ant-Direct ✅ Pass ✅ Pass -- ✅ Pass ✅ Pass
3 Isaac-Cartpole ✅ Pass ✅ Pass -- ✅ Pass ✅ Pass
4 Isaac-Cartpole-Camera ✅ Pass ✅ Pass -- ✅ Pass ✅ Pass
5 Isaac-Cartpole-Camera-Direct ✅ Pass ✅ Pass -- ✅ Pass ✅ Pass
6 Isaac-Cartpole-Direct ✅ Pass ✅ Pass -- ✅ Pass ✅ Pass
7 Isaac-Fourbar-Pole-Swingup -- ✅ Pass -- -- --
8 Isaac-Humanoid ✅ Pass -- -- ✅ Pass ✅ Pass
9 Isaac-Humanoid-Direct ✅ Pass -- -- ✅ Pass ✅ Pass
10 Isaac-Lift-Cable-Franka -- -- ✅ Pass -- --
11 Isaac-Lift-Cable-Franka-Camera -- -- ✅ Pass -- --
12 Isaac-Lift-Cloth-Franka -- -- ❌ Unsupported (D) -- ❌ Unsupported (D)
13 Isaac-Lift-Cloth-Franka-Camera -- -- ❌ Unsupported (D) -- ❌ Unsupported (D)
14 Isaac-Lift-Franka ✅ Pass -- -- ✅ Pass ✅ Pass
15 Isaac-Lift-KukaAllegro ✅ Pass -- -- ✅ Pass ✅ Pass
16 Isaac-Lift-KukaAllegro-Camera ✅ Pass -- -- ✅ Pass ✅ Pass
17 Isaac-Lift-Soft-Franka -- -- ❌ Unsupported (D) -- ❌ Unsupported (D)
18 Isaac-Lift-Soft-Franka-Camera -- -- ❌ Unsupported (D) -- ❌ Unsupported (D)
19 Isaac-Open-Drawer-Franka ✅ Pass -- -- ✅ Pass ✅ Pass
20 Isaac-Open-Drawer-Franka-Direct ✅ Pass -- -- ✅ Pass ✅ Pass
21 Isaac-Pendulum-MARL-Direct ✅ Pass ✅ Pass -- ✅ Pass ✅ Pass
22 Isaac-Reach-Franka ✅ Pass -- -- ✅ Pass ✅ Pass
23 Isaac-Reach-Franka-OSC ✅ Pass -- -- ✅ Pass ✅ Pass
24 Isaac-Reach-UR10 ✅ Pass -- -- ✅ Pass ✅ Pass
25 Isaac-Reorient-Cube-Allegro ✅ Pass -- -- ✅ Pass ✅ Pass
26 Isaac-Reorient-Cube-Allegro-Direct ✅ Pass -- -- ✅ Pass ✅ Pass
27 Isaac-Reorient-Cube-Shadow ✅ Pass -- -- ✅ Pass ✅ Pass
28 Isaac-Reorient-Cube-Shadow-Camera ✅ Pass -- -- ✅ Pass ✅ Pass
29 Isaac-Reorient-Cube-Shadow-Camera-Direct ✅ Pass -- -- ✅ Pass ✅ Pass
30 Isaac-Reorient-Cube-Shadow-Direct ✅ Pass -- -- ✅ Pass ✅ Pass
31 Isaac-Reorient-Franka ✅ Pass -- -- ✅ Pass ✅ Pass
32 Isaac-Reorient-KukaAllegro ✅ Pass -- -- ✅ Pass ✅ Pass
33 Isaac-Reorient-KukaAllegro-Camera ✅ Pass -- -- ✅ Pass ✅ Pass
34 Isaac-Shadow-Handover ✅ Pass -- -- ✅ Pass ✅ Pass
35 Isaac-Shadow-Handover-Direct ✅ Pass -- -- ✅ Pass ✅ Pass
36 Isaac-Velocity-Flat-AnymalD ✅ Pass ✅ Pass -- ✅ Pass ✅ Pass
37 Isaac-Velocity-Flat-Cassie ✅ Pass ✅ Pass -- ✅ Pass ✅ Pass
38 Isaac-Velocity-Flat-G1 ✅ Pass ✅ Pass -- ✅ Pass ✅ Pass
39 Isaac-Velocity-Flat-H1 ✅ Pass ✅ Pass -- ✅ Pass ✅ Pass
40 Isaac-Velocity-Flat-UnitreeGo2 ✅ Pass ✅ Pass -- ✅ Pass ✅ Pass
41 Isaac-Velocity-Rough-AnymalD ✅ Pass ✅ Pass -- ✅ Pass ✅ Pass
42 Isaac-Velocity-Rough-Cassie ✅ Pass ✅ Pass -- ✅ Pass ✅ Pass
43 Isaac-Velocity-Rough-G1 ✅ Pass ✅ Pass -- ✅ Pass ✅ Pass
44 Isaac-Velocity-Rough-H1 ✅ Pass ✅ Pass -- ✅ Pass ✅ Pass
45 Isaac-Velocity-Rough-UnitreeGo2 ✅ Pass ✅ Pass -- ✅ Pass ✅ Pass

Remaining export limitations

Deformable export was removed from this revision; normal simulation remains supported where its task preset exists. Export stops at the asset boundary instead of writing an incomplete scene. Restoring support requires data-owned physical declarations, the corresponding native/schema reader, and fresh-backend validation under the same pattern as rigid assets and cables.

# Task Backend preset Exact export error
1 Isaac-Lift-Cloth-Franka isaacsim_physx NotImplementedError: Fixed USD export is not supported for DeformableObject.
2 Isaac-Lift-Cloth-Franka newton_mjwarp_vbd_proxy NotImplementedError: Fixed USD export is not supported for DeformableObject.
3 Isaac-Lift-Cloth-Franka-Camera isaacsim_physx NotImplementedError: Fixed USD export is not supported for DeformableObject.
4 Isaac-Lift-Cloth-Franka-Camera newton_mjwarp_vbd_proxy NotImplementedError: Fixed USD export is not supported for DeformableObject.
5 Isaac-Lift-Soft-Franka isaacsim_physx NotImplementedError: Fixed USD export is not supported for DeformableObject.
6 Isaac-Lift-Soft-Franka newton_mjwarp_vbd_proxy NotImplementedError: Fixed USD export is not supported for DeformableObject.
7 Isaac-Lift-Soft-Franka-Camera isaacsim_physx NotImplementedError: Fixed USD export is not supported for DeformableObject.
8 Isaac-Lift-Soft-Franka-Camera newton_mjwarp_vbd_proxy NotImplementedError: Fixed USD export is not supported for DeformableObject.

Validation coverage

  1. Fresh backend loading compares stable entity identities, topology, mass/inertia/COM, joint properties, geometry, material values, collision relationships and gravity. Discrete relationships compare exactly; floats use property-appropriate tolerances. Task presets use one environment/env0 and seed 42; separate fixtures cover multiple objects, different per-environment values and selected-replica isolation.
  2. Task diagnostics suppress startup only to capture the pre-startup reference. Ordinary construction/entrypoint tests separately verify automatic export, continued startup/training behavior and source nonmutation. Fresh loading replays no task overrides. The six Shadow regressions pass after preserving native limit provenance.
  3. Newton rough-terrain loading reuses the existing heightfield adapter and declared contact reader. The artifact retains its source mesh; direct mesh-only loading is a different physical representation. Cable contact supplements use the asset owner's reader. Proxy-preset checks validate asset configuration; coupled solver reconstruction is outside this revision.
  4. Experimental USD Exchange 3.0 validation: parent Newton 13 passed, OVPhysX 2 passed, and the actual Franka OVPhysX task passed a fresh native load. The child suite passed 16 Newton and 2 PhysX deformable cases. Final merged shared/actuator/metadata/builder checks: 76 passed, 2 skipped. The schema follow-up overlaps these counts. Required-schema and equal-import regressions fail without their fixes. OVPhysX selection includes 64 environments, not arbitrary heterogeneity.
  5. Eight fresh input/output audits using the experimental USD Exchange 3.0 runtime trace 489 remaining property/relationship edits. Removed 143 redundant writes: 142 Newton aliases and the flat-cloth rest-angle array. The two OV finger-armature aliases are required: adding an axis API for a velocity override otherwise changes fresh-load armature from 0.1 to 0. Backend adapters therefore control imported-default reuse. No new pose, velocity, mass/inertia/COM or geometry-array opinions were found. These audits do not refresh task-support results.
  6. The PR retains USD Exchange 2.3.0, its lockfile and parsing workarounds; the 3.0 experiment remains testing-only. Current decorator correction: 82 passed, 2 CUDA skips in the parent; 88 passed, 2 CUDA skips in the merged child, on local CPU. Tests cover inherited bindings driving actual initialization/selection, dictionary behavior and explicit-drive disabling. The new binding regression fails with the previous control implementation. The dependency test also fails before the resource-namespace collision fix and passes afterward. Formatting and changelog checks passed. Earlier native/backend and importer results belong to the 3.0 experiment; no new full sweep, support-matrix refresh or CI dispatch.

Components outside this export

  1. Controllers and observations: policy/controller code, histories and observation processing require deployment integration.
  2. Sensors: retained camera/sensor prims do not restore sensor sampling, rendering or observation execution.
  3. Transient state: sampled positions/velocities, warm-start caches, active contacts and transient forces are excluded. Existing source state is preserved, including authored velocities; export does not zero it.
  4. Unsupported objects: registered deformables and surface grippers fail explicitly. Cable support remains included.
  5. Backend limits: later setter/buffer mutations, distinct per-cooked-piece PhysX runtime overrides and private geometry/topology changes are not reconstructed. Native Newton/USD alone does not read Isaac Lab's heightfield or cable supplements.
  6. Simulation choices: VBD, XPBD, Kamino and coupled solver settings are not serialized. MJWarp settings are optional; loading ordinary assets does not require them. Cross-backend deployment and arbitrary heterogeneous layouts need separate validation.
  7. External resources: flattening retains references and does not package textures, MDL or native neural-actuator checkpoints. ANYmal’s pre-startup actuator setup authors a temporary checkpoint path; deployment on another host must carry that checkpoint. The eight-file audit is not a self-contained deployment-bundle certification.

Type of change

New feature, bug fixes, task configuration migration and documentation update.

Release backport

  • Backport this pull request to the active release branch after it merges into develop

Screenshots

Not applicable: USD authoring and backend validation.

Checklist

  • Read the contribution guidelines; contributor already listed.
  • Added package changelog fragments and ran formatting/lint checks.
  • Added fresh-backend and initialization regression tests.
  • Updated public documentation.
  • Clean strict Sphinx HTML build with the dev extra.
  • No warnings: existing dependency deprecation and native importer warnings remain.

hujc7 added 9 commits July 29, 2026 13:34
Isaac Lab applies most configuration directly to the solver rather than
to the stage, so a scene that has been loaded and overridden no longer
has a USD file describing what is actually being simulated.

Add export_model_to_usd(), the inverse of ModelBuilder.add_usd(). Core
physics is authored with standard UsdPhysics schemas and Newton-specific
properties as newton:* attributes, at the prim paths the model was
imported from, so a reimport reproduces the same model.

The correctness contract is model idempotence rather than USD fidelity:
the importer normalizes as it reads, so the exported stage differs from
the source by construction, but reimporting it must change nothing.

Known gaps are documented in the module and fail loudly rather than
exporting something wrong: D6/ball/distance joints, planes, heightfields
and SDF shapes, cloned multi-world scenes, and shape ordering, which
follows USD stage traversal.
Comparing model arrays shows the numbers match but not that the export
reproduces the same physics, and it reports spurious differences for
rebuilt acceleration structures and for shape orderings that differ
without changing behavior.

Add a rollout check that steps the source and exported models under
identical initial conditions and compares body trajectories, treating a
quaternion and its negation as the same rotation. The fixture's joint
carries an offset frame so the trajectory is sensitive to joint
geometry; without it the check cannot distinguish a correct export from
one that drops joint frames.

The two layers are complementary. The rollout misses parameter drops
that do not change this particular motion, such as mass in a pendulum
under gravity, which the array comparison catches.
Every existing check starts from an unmodified load, so an exporter that
re-derived its output from the source USD would pass all of them while
reproducing the asset rather than what is being simulated. That is the
property the feature exists for: Isaac Lab applies most configuration by
writing into the model after the stage is parsed.

Add a check that writes distinctive values into the model, as the asset
classes do, and asserts the export carries them rather than the source
file's values.

Also document that free-joint degrees of freedom carry no exportable
state: a floating body is expressed in USD by the absence of a joint, so
per-DOF values written to a free joint have nowhere to be authored.
Isaac Lab's cloner builds one Newton world per environment, all sharing the
source asset's prim paths. The exporter resolved indices as if the model held
a single world, so a cloned scene exported only the first environment's
entities under every environment's prim paths and reported success: a
3-world model of 6 bodies came back as 2.

Select the world explicitly. Entities are chosen by their world membership
plus the worldless global content, which is the same rule for an uncloned
model, and the count of exportable entities is checked against the source's
prim paths so a shortfall fails instead of shipping a partial file. Free
joints and sites are exempt: a floating body is expressed in USD by the
absence of a joint, and a site is a bare frame with no geometry.

Every task's ground is a plane, which raised NotImplementedError; author it
as a Z-axis UsdGeom.Plane, the form the importer reads back. Physics
materials are authored for every shape, since Isaac Lab randomizes friction
per shape regardless of whether it collides.

Also export the entry point from isaaclab_newton.sim; the lazy-export stub
had omitted it, so the documented import path raised AttributeError.
export_model_to_usd() needs the path maps that ModelBuilder.add_usd()
returns, and every build path discarded them: the clone-plan session dropped
the return of ctx.replicate(), and the per-prototype import result never
left _build_source_builder. The exporter could not be called on a real
environment at all.

Keep the results and merge them into one environment's provenance. Global
content is indexed against the model directly; each prototype's indices are
lifted by the landing offset replicate_builder_mapping already computes
before appending it, so a world with several prototypes (robot, object,
table) resolves exactly. NewtonManager.get_stage_info() exposes the result
in the shape add_usd() returns, so it feeds the exporter unchanged.
PhysX and OVPhysX keep the stage authoritative for structure, but every
runtime write (drive gains, masses, armature, friction, limits) goes to the
solver's buffers and never reaches a prim. Saving the stage of a running
scene therefore emits a file that looks complete while carrying the
spawn-time value of everything overridden since.

Author those properties back onto the prims they came from, reading through
BaseArticulationData so the same code serves any backend. Recovering prim
paths is the one backend-specific step, so each backend supplies them
through ArticulationPrimPaths. Values are joined to prims by name, because
the view is in backend order and the data in public order.

Angular drive gains are per degree on the stage and per radian in the
simulation; converting on export matters for every driven revolute joint,
and a task whose only revolute joint is passive cannot reveal it.
The PhysX tensor view records the prim each link and degree of freedom was
built from, so the paths are read straight off it and handed to the shared
exporter.
An OVPhysX binding reports names and the articulation prims it matched, not
per-link paths, so they are resolved by indexing the articulation subtree by
prim name. A body or joint with no prim fails the export rather than leaving
it partial.
@github-actions github-actions Bot added the isaac-lab Related to Isaac Lab team label Sep 2, 2026
Both PhysX and OVPhysX populate the joint friction the solver applies from
PhysxJointAxisAPI (static and dynamic friction effort, viscous coefficient),
not from the legacy physxJoint:jointFriction scalar the exporter wrote, so
randomized friction reimported as zero. The per-axis schema also shadows the
joint-level armature on OVPhysX, and its viscous coefficient is per degree
per second on angular axes: the exporter now writes the triple under the
drive axis, armature in both places, and viscous scaled like drive gains.

Applying a schema to a prim that is itself an articulation root invalidates
every PhysX articulation view on the stage for the rest of the session.
export_articulation_to_usd therefore authors onto a flattened snapshot;
write_articulation_state_to_stage takes the target stage explicitly and
applies schemas only where absent. Reads precede writes, prim paths on
OVPhysX resolve from a typed index that walks up from the root prim, and
each backend wrapper forwards the stage argument.
ModelBuilder.approximate_meshes(keep_visual_shapes=True), which the USD
importer runs on visible meshes with a collision approximation, copies the
mesh as a visual-only shape labelled <label>_visual that has no prim of its
own, so the coverage guard refused every scene with such an asset. The
resolution of a world's entities to prim paths moves into a public
resolve_world_prim_paths returning WorldPrimPaths; twins resolve to a
visual-only <prim>_visual sibling of their source and are authored as such.
Coverage is checked entity by entity rather than by count, unsupported
geometry is rejected by name, and joints outside the articulation are
exported with physics:excludeFromArticulation.
Retaining import provenance made replicate_builder_mapping return the
world-0 landing offsets as a third element against its declared two-tuple,
which broke four develop cloner tests and every caller that unpacks two
values. The offsets are recorded only on request by a private
implementation; replicate_builder_mapping keeps its contract and the two
provenance callers use replicate_builder_mapping_with_provenance.
…usd-export

# Conflicts:
#	source/isaaclab_newton/isaaclab_newton/cloner/newton_clone_utils.py
#	source/isaaclab_newton/isaaclab_newton/cloner/replicate.py
#	source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py
#	source/isaaclab_newton/isaaclab_newton/physics/visualization_builder.py
Newton 1.6 keeps shape_color in sRGB and reads primvars:displayColor as
linear, converting on import, so writing the model colour straight into the
attribute re-applied the transfer curve on every round-trip and the export
stopped being a fixed point. The exporter now authors the inverse. The test
loader imports the way Isaac Lab does, aligning builder colours with the
stage after add_usd, which is where the exported colour is read back.
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 10, 2026
@hujc7 hujc7 changed the title [USD] Export the simulated scene back to USD on Newton, PhysX and OVPhysX [USD] Export selected environments with effective physics configuration Sep 10, 2026
@hujc7

hujc7 commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator Author

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Sep 10, 2026
@hujc7

hujc7 commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator Author

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Sep 14, 2026
Preserve authored scene content and declare initialized buffer mappings through a shared writer and thin source adapters. Export the actual task before events in an isolated rank-zero training worker, and preserve distinct Newton collider material bindings.

Remove the PR-only runtime snapshot, model reconstruction, and clone provenance paths. Validate complete fresh-backend loads, failure atomicity, and unchanged seeded training behavior.
@hujc7 hujc7 changed the title [USD] Export selected environments with effective physics configuration [USD] Export complete fixed deployment scenes before training Sep 14, 2026
@hujc7

hujc7 commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator Author

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Sep 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation infrastructure isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant