Feature proposal: unified parallel evaluation interface for RoboTwin and LIBERO using Labtasker

[luocfprime]

Summary

• add self-contained RoboTwin evaluation using 50 native whole-task Tasks and managed one-server-per-Worker execution
• add native LIBERO task orchestration with managed policy servers, suite/task sharding, retries, and summaries
• share process ownership and resource helpers, and expose both workflows from the benchmarks index
• standardize Task names as benchmark:robotwin and benchmark:libero

Validation

• 67 combined adapter tests passed (26 RoboTwin + 41 LIBERO)
• Ruff, Python byte-compilation, shell syntax, and whitespace checks passed
• real-checkpoint RoboTwin smoke: 50/50 Tasks completed, 48/50 episodes succeeded, 8 peak Workers, 20m05s evaluation span, 7.01x summed-work/span
• real-checkpoint LIBERO smoke: 1/1 task and trial succeeded

These one-episode smoke runs validate GPU compatibility, native Task lifecycle, and parallel scheduling; they are not benchmark-scale accuracy or throughput claims.

Proposal scope

• evaluation adapters only; GPU allocation and artifact storage remain external
• native Task state, retries, and Queue semantics; no custom scheduler or manifest state

[wayrise]

I recommend revising the evaluation granularity and repository placement before merging.

1. Schedule at the episode (or small episode-batch) level

The current design assigns one complete task to one Worker. For RoboTwin, a Task normally contains the full episode set (typically 100 episodes). This creates a long-tail problem: once most tasks finish, the remaining slow task can continue occupying only one Worker while all other Workers are idle. The overall evaluation time is therefore bounded by the slowest task, and idle Workers cannot help with its remaining episodes.

Please consider making the schedulable unit an episode, or at least a small episode batch, for example (task, mode, episode_start, num_episodes). This would allow idle Workers to claim more work from the same benchmark task. The implementation should preserve deterministic episode seeds and ordering, aggregate results back to task-level and benchmark-level success rates, and retry only the failed episode/batch where possible. Each Worker can still use its own exclusive policy endpoint; the endpoint ownership constraint does not require the Labtasker Task to represent an entire benchmark task.

For LIBERO, the existing trial_start / num_trials interface appears to provide a natural foundation for this split. RoboTwin may need an evaluator entry point that accepts an episode range or a single-episode invocation while preserving the original seed semantics.

2. Extend the existing benchmark directories instead of adding parallel copies

Please integrate the Labtasker entry points into the existing benchmarks/robotwin and benchmarks/libero directories as an additional execution mode. Adding separate robotwin-labtasker and libero-labtasker trees duplicates evaluator, interface, configuration, and launcher code, which will make future fixes and protocol changes prone to drift between the old and new paths.

The corresponding tests should be moved to the repository-level tests/ directory (for example, tests/benchmarks/robotwin/ and tests/benchmarks/libero/) rather than living under benchmarks/*/tests. This keeps test discovery and CI ownership consistent with the rest of the repository.

These changes are important for both practical evaluation throughput and long-term maintainability.

[luocfprime]

I recommend revising the evaluation granularity and repository placement before merging.

1. Schedule at the episode (or small episode-batch) level

The current design assigns one complete task to one Worker. For RoboTwin, a Task normally contains the full episode set (typically 100 episodes). This creates a long-tail problem: once most tasks finish, the remaining slow task can continue occupying only one Worker while all other Workers are idle. The overall evaluation time is therefore bounded by the slowest task, and idle Workers cannot help with its remaining episodes.

Please consider making the schedulable unit an episode, or at least a small episode batch, for example (task, mode, episode_start, num_episodes). This would allow idle Workers to claim more work from the same benchmark task. The implementation should preserve deterministic episode seeds and ordering, aggregate results back to task-level and benchmark-level success rates, and retry only the failed episode/batch where possible. Each Worker can still use its own exclusive policy endpoint; the endpoint ownership constraint does not require the Labtasker Task to represent an entire benchmark task.

For LIBERO, the existing trial_start / num_trials interface appears to provide a natural foundation for this split. RoboTwin may need an evaluator entry point that accepts an episode range or a single-episode invocation while preserving the original seed semantics.

2. Extend the existing benchmark directories instead of adding parallel copies

Please integrate the Labtasker entry points into the existing benchmarks/robotwin and benchmarks/libero directories as an additional execution mode. Adding separate robotwin-labtasker and libero-labtasker trees duplicates evaluator, interface, configuration, and launcher code, which will make future fixes and protocol changes prone to drift between the old and new paths.

The corresponding tests should be moved to the repository-level tests/ directory (for example, tests/benchmarks/robotwin/ and tests/benchmarks/libero/) rather than living under benchmarks/*/tests. This keeps test discovery and CI ownership consistent with the rest of the repository.

These changes are important for both practical evaluation throughput and long-term maintainability.

Thanks, this is very helpful. This was intentionally an early version submitted to get feedback before settling on the final implementation.
I agree with both directions. My current interpretation is to apply them consistently to both benchmarks:

• integrate the Labtasker execution paths into the existing benchmarks/libero and benchmarks/robotwin directories, and move their tests under the repository-level tests/benchmarks;
• change the Labtasker scheduling unit from a complete benchmark task (e.g. task in robotwin or task_id in LIBERO) to a configurable small episode/trial batch;
• aggregate batches back into task-level and benchmark-level results, and retry only the failed episode or batch where possible.

I’ll revise the implementation in this direction and follow up here once the changes and tests are ready. Please let me know if I’ve misunderstood the intended scope.

[luocfprime]

Hello, this implements issue #29.

• Partitions RoboTwin and LIBERO evaluation into deterministic episode/trial minibatches.
• Unifies scheduling, elastic Workers, retries, monitoring, and result aggregation through Labtasker.
• Preserves existing evaluation entry points and validates out-of-order execution, retries, and real-checkpoint evaluation.

This supersedes the earlier whole-task implementation in this draft.

[luocfprime]

Here are the reproduced LIBERO and RoboTwin results using this evaluation implementation.

LIBERO:

❯ uv run --isolated --with labtasker python \
    benchmarks/libero/labtasker_summarize.py \
    full-libero-eval-20260915 \
    --queue fastwam \
    --output-dir /tmp/libero-summary
Installed 108 packages in 348ms
[labtasker] connected server=remote transport=https url=https://labtasker.xxxxxxxx.site
Suite summary
Suite           Success/Trials  Success rate  Done/Total  Pending  Running  Failed
--------------  --------------  ------------  ----------  -------  -------  ------
libero_10       492/500         98.40%        100/100     0        0        0
libero_goal     495/500         99.00%        100/100     0        0        0
libero_object   499/500         99.80%        100/100     0        0        0
libero_spatial  496/500         99.20%        100/100     0        0        0

Task summary
Suite           Task  Success/Trials  Success rate  Done/Total  Pending  Running  Failed
--------------  ----  --------------  ------------  ----------  -------  -------  ------
libero_10       0     47/50           94.00%        10/10       0        0        0
libero_10       1     50/50           100.00%       10/10       0        0        0
libero_10       2     49/50           98.00%        10/10       0        0        0
libero_10       3     50/50           100.00%       10/10       0        0        0
libero_10       4     48/50           96.00%        10/10       0        0        0
libero_10       5     50/50           100.00%       10/10       0        0        0
libero_10       6     49/50           98.00%        10/10       0        0        0
libero_10       7     50/50           100.00%       10/10       0        0        0
libero_10       8     49/50           98.00%        10/10       0        0        0
libero_10       9     50/50           100.00%       10/10       0        0        0
libero_goal     0     50/50           100.00%       10/10       0        0        0
libero_goal     1     50/50           100.00%       10/10       0        0        0
libero_goal     2     48/50           96.00%        10/10       0        0        0
libero_goal     3     50/50           100.00%       10/10       0        0        0
libero_goal     4     50/50           100.00%       10/10       0        0        0
libero_goal     5     50/50           100.00%       10/10       0        0        0
libero_goal     6     49/50           98.00%        10/10       0        0        0
libero_goal     7     50/50           100.00%       10/10       0        0        0
libero_goal     8     48/50           96.00%        10/10       0        0        0
libero_goal     9     50/50           100.00%       10/10       0        0        0
libero_object   0     50/50           100.00%       10/10       0        0        0
libero_object   1     50/50           100.00%       10/10       0        0        0
libero_object   2     50/50           100.00%       10/10       0        0        0
libero_object   3     50/50           100.00%       10/10       0        0        0
libero_object   4     50/50           100.00%       10/10       0        0        0
libero_object   5     50/50           100.00%       10/10       0        0        0
libero_object   6     50/50           100.00%       10/10       0        0        0
libero_object   7     49/50           98.00%        10/10       0        0        0
libero_object   8     50/50           100.00%       10/10       0        0        0
libero_object   9     50/50           100.00%       10/10       0        0        0
libero_spatial  0     50/50           100.00%       10/10       0        0        0
libero_spatial  1     48/50           96.00%        10/10       0        0        0
libero_spatial  2     50/50           100.00%       10/10       0        0        0
libero_spatial  3     50/50           100.00%       10/10       0        0        0
libero_spatial  4     49/50           98.00%        10/10       0        0        0
libero_spatial  5     49/50           98.00%        10/10       0        0        0
libero_spatial  6     50/50           100.00%       10/10       0        0        0
libero_spatial  7     50/50           100.00%       10/10       0        0        0
libero_spatial  8     50/50           100.00%       10/10       0        0        0
libero_spatial  9     50/50           100.00%       10/10       0        0        0

Output files
  Suite summary: /tmp/libero-summary/suite_summary.csv
  Task summary:  /tmp/libero-summary/task_summary.csv

RoboTwin (demo_clean):

❯ uv run --isolated --with labtasker python \
    benchmarks/robotwin/labtasker_summarize.py \
    full-robotwin-eval-clean-20260915 \
    --queue fastwam \
    --output-dir /tmp/robotwin-summary/clean
Installed 108 packages in 1.10s
[labtasker] connected server=remote transport=https url=https://labtasker.xxxxxxxx.site
Benchmark summary
Mode        Instructions  Success/Episodes  Success rate  Done/Total  Pending  Running  Failed
----------  ------------  ----------------  ------------  ----------  -------  -------  ------
demo_clean  unseen        4715/5000         94.30%        1000/1000   0        0        0

Task summary
Mode        Instructions  Task                       Success/Episodes  Success rate  Done/Total  Pending  Running  Failed
----------  ------------  -------------------------  ----------------  ------------  ----------  -------  -------  ------
demo_clean  unseen        adjust_bottle              100/100           100.00%       20/20       0        0        0
demo_clean  unseen        beat_block_hammer          99/100            99.00%        20/20       0        0        0
demo_clean  unseen        blocks_ranking_rgb         100/100           100.00%       20/20       0        0        0
demo_clean  unseen        blocks_ranking_size        76/100            76.00%        20/20       0        0        0
demo_clean  unseen        click_alarmclock           98/100            98.00%        20/20       0        0        0
demo_clean  unseen        click_bell                 100/100           100.00%       20/20       0        0        0
demo_clean  unseen        dump_bin_bigbin            96/100            96.00%        20/20       0        0        0
demo_clean  unseen        grab_roller                100/100           100.00%       20/20       0        0        0
demo_clean  unseen        handover_block             98/100            98.00%        20/20       0        0        0
demo_clean  unseen        handover_mic               86/100            86.00%        20/20       0        0        0
demo_clean  unseen        hanging_mug                51/100            51.00%        20/20       0        0        0
demo_clean  unseen        lift_pot                   100/100           100.00%       20/20       0        0        0
demo_clean  unseen        move_can_pot               99/100            99.00%        20/20       0        0        0
demo_clean  unseen        move_pillbottle_pad        99/100            99.00%        20/20       0        0        0
demo_clean  unseen        move_playingcard_away      100/100           100.00%       20/20       0        0        0
demo_clean  unseen        move_stapler_pad           99/100            99.00%        20/20       0        0        0
demo_clean  unseen        open_laptop                100/100           100.00%       20/20       0        0        0
demo_clean  unseen        open_microwave             87/100            87.00%        20/20       0        0        0
demo_clean  unseen        pick_diverse_bottles       87/100            87.00%        20/20       0        0        0
demo_clean  unseen        pick_dual_bottles          93/100            93.00%        20/20       0        0        0
demo_clean  unseen        place_a2b_left             98/100            98.00%        20/20       0        0        0
demo_clean  unseen        place_a2b_right            94/100            94.00%        20/20       0        0        0
demo_clean  unseen        place_bread_basket         94/100            94.00%        20/20       0        0        0
demo_clean  unseen        place_bread_skillet        92/100            92.00%        20/20       0        0        0
demo_clean  unseen        place_burger_fries         98/100            98.00%        20/20       0        0        0
demo_clean  unseen        place_can_basket           79/100            79.00%        20/20       0        0        0
demo_clean  unseen        place_cans_plasticbox      98/100            98.00%        20/20       0        0        0
demo_clean  unseen        place_container_plate      99/100            99.00%        20/20       0        0        0
demo_clean  unseen        place_dual_shoes           96/100            96.00%        20/20       0        0        0
demo_clean  unseen        place_empty_cup            100/100           100.00%       20/20       0        0        0
demo_clean  unseen        place_fan                  98/100            98.00%        20/20       0        0        0
demo_clean  unseen        place_mouse_pad            99/100            99.00%        20/20       0        0        0
demo_clean  unseen        place_object_basket        93/100            93.00%        20/20       0        0        0
demo_clean  unseen        place_object_scale         96/100            96.00%        20/20       0        0        0
demo_clean  unseen        place_object_stand         97/100            97.00%        20/20       0        0        0
demo_clean  unseen        place_phone_stand          94/100            94.00%        20/20       0        0        0
demo_clean  unseen        place_shoe                 100/100           100.00%       20/20       0        0        0
demo_clean  unseen        press_stapler              91/100            91.00%        20/20       0        0        0
demo_clean  unseen        put_bottles_dustbin        95/100            95.00%        20/20       0        0        0
demo_clean  unseen        put_object_cabinet         86/100            86.00%        20/20       0        0        0
demo_clean  unseen        rotate_qrcode              93/100            93.00%        20/20       0        0        0
demo_clean  unseen        scan_object                97/100            97.00%        20/20       0        0        0
demo_clean  unseen        shake_bottle               100/100           100.00%       20/20       0        0        0
demo_clean  unseen        shake_bottle_horizontally  100/100           100.00%       20/20       0        0        0
demo_clean  unseen        stack_blocks_three         99/100            99.00%        20/20       0        0        0
demo_clean  unseen        stack_blocks_two           100/100           100.00%       20/20       0        0        0
demo_clean  unseen        stack_bowls_three          89/100            89.00%        20/20       0        0        0
demo_clean  unseen        stack_bowls_two            97/100            97.00%        20/20       0        0        0
demo_clean  unseen        stamp_seal                 98/100            98.00%        20/20       0        0        0
demo_clean  unseen        turn_switch                77/100            77.00%        20/20       0        0        0

Output files
  Benchmark summary: /tmp/robotwin-summary/clean/benchmark_summary.csv
  Task summary:      /tmp/robotwin-summary/clean/task_summary.csv

Benchmark	Suite / Mode	OpenWAM reported	Reproduced	Difference
LIBERO	Spatial	99.6%	99.2%	−0.4 pp
LIBERO	Object	99.6%	99.8%	+0.2 pp
LIBERO	Goal	99.8%	99.0%	−0.8 pp
LIBERO	Long (libero_10)	98.2%	98.4%	+0.2 pp
LIBERO	Average	99.3%	99.1%	−0.2 pp
RoboTwin2.0-Full	demo_clean / unseen	93.74%	94.30%	+0.56 pp

The results produced by our evaluation implementation are consistent with the OpenWAM originally reported results, with all completed benchmark settings differing by less than 1 percentage point.

[luocfprime]

Hello! I believe the requested changes are now in place, and I’ve verified the implementation by running the full evaluation suite and reproducing the results.

I’d appreciate another review. Please let me know if anything needs further adjustment or clarification!

[wayrise]

Review notes from running this PR on a multi-GPU node

How these were produced: checked out codex/labtasker-native-evals (a577564), installed labtasker==2.4.1 into a clone of our OpenWAM conda env, read the full diff, and ran the suites on a real GPU node (containerized, repo on a WekaFS mount). Results: tests/benchmarks 107/108 passed; the openwam/deploy/server.py refactor passes all 143 deploy-related tests; ruff check benchmarks/ and byte-compilation are clean.

Overall this is solid work — content-addressed sealed manifests with atomic publication, the stdin-lifeline process guardian, RNG domain isolation, and the layered result validation are all well designed, and the integration tests (real Labtasker server + real worker subprocesses + fake simulators) exercise caching, retries, and idempotent resubmission end to end.

Findings, roughly by severity:

1. test_normal_exit_cleans_grandchild fails deterministically in containers whose PID 1 does not reap orphans

Reproduced 3/3 on a dev container (PID 1 = sleep). The guardian actually kills the grandchild correctly, but it becomes a zombie (State: Z, PPid: 1) because nothing reaps it, and the test's gone() uses os.kill(pid, 0), which succeeds for zombies — so wait_for times out. GitHub Actions passes because its PID 1 reaps orphans. Suggest treating zombies as gone in the test helper (e.g. read the state field from /proc/<pid>/stat), so make test also works in docker/K8s environments without --init. The production code in resource_process.py is fine — this is purely test portability.

2. The local Labtasker daemon cannot start when the repo lives on a shared filesystem — needs a documented escape hatch

First real submission from a WekaFS-mounted checkout fails after a 30 s timeout with only Daemon did not become healthy within 30 seconds; the actual cause (sqlite3.OperationalError: database is locked, raised from labtasker-server's PRAGMA synchronous=FULL) is buried in .labtasker/server.log. The identical command on a local disk starts the daemon instantly and submits fine. Cluster checkouts commonly live on NFS/Weka/Lustre, and both LABTASKER.md files promise "On a single machine, no Labtasker server setup is needed", so first contact with this feature can be a dead end. Suggest a short note in both docs: if the repository is on a network filesystem, run a Labtasker server manually and point LABTASKER_URL at it (the auto-started local daemon needs local-disk SQLite locking).

3. CI pins labtasker==2.4.0 while everything else pins 2.4.1

.github/workflows/ci.yml still installs 2.4.0; the pyproject dev extra, both requirements-labtasker.txt, and docker/requirements-cu128.txt all pin 2.4.1 (def298c missed the workflow). Since the dev extra now contains labtasker and pytest anyway, CI could switch to pip install -e .[dev] and drop the separate pin, removing the second source of truth.

4. Tests set env vars that no implementation reads

tests/benchmarks/test_robotwin_labtasker.py (in test_seed_planner_isolated_from_caller_rng) sets ROBOTWIN_MANIFEST_CACHE_DIR and clears ROBOTWIN_EXTEND_MANIFEST; neither name occurs anywhere in the implementation — presumably leftovers from an earlier design. Suggest deleting the two lines.

5. make test without labtasker installed now fails at collection

test_libero_labtasker.py and test_task_policy.py import labtasker at module level, so a plain pip install -e . user running make test gets collection errors instead of skips. pytest.importorskip("labtasker") at the top of the labtasker test modules would keep the core suite runnable for users who never touch this feature.

6. Question: should the deploy Docker image ship the labtasker-server stack?

Because docker/requirements-cu128.txt is compiled including the dev extra, the GPU image now carries fastapi / uvicorn / sqlalchemy / alembic / greenlet etc. If the image is intended to run Labtasker evaluations, that is fine and worth stating; if not, excluding dev extras from the lock compile would keep the runtime image smaller.

[luocfprime]

Thanks for the detailed report. Issues other than problem 2 seem easy to address. I’ll address the other issues as soon as possible.

For problem 2, this appears to be caused by the interaction between the automatic daemon’s filesystem ownership lock (flock) and SQLite’s locking/WAL behavior on NFS-like filesystems. The problem can generally be avoided by placing the Labtasker database on non-NFS storage:

labtasker-server serve --database /path/on/local/storage/server.db

In my own setup, I run the Labtasker Server on an independent CVM with its database on local storage. Clients can connect to it whether they run inside or outside the cluster, so this issue does not occur.

However, that workaround is insufficient when the Labtasker Server itself runs inside a cluster environment. Persistent storage there is often NFS-backed, while local pod or node storage is ephemeral and disappears when the allocation is released.

I’ll review this design shortly, particularly whether Labtasker should use a different SQLite journal mode and ownership-locking mechanism for shared filesystems.

[KraHsu]

This review found three issues that should be addressed before merging: policy endpoint ownership, completion reporting after an interrupted submission, and readiness checks in environments with proxy settings. Reproduction conditions and suggested changes are in the inline comments.

Validation included CPU regression tests and bounded GPU checks covering real LIBERO policy rollouts, policy reuse and reload, process and GPU-memory cleanup, and protocol checks with the official RoboTwin checkpoint. RoboTwin simulator checks used the existing Python environment with freshly checked-out pinned sources. The fresh RoboTwin environment's CUDA error and the trajectory differences observed during replay have not been attributed to this PR, so neither is counted as a confirmed PR defect. These checks do not constitute a full benchmark evaluation.