Hub quality scan tests the v0 API, so a v1 taskset can't pass it

[strickvl]

Pushed a v1 taskset to the Hub today (strickvl/isaf-extraction) and the automated quality scan fails. As far as I can tell no v1 environment can pass it, because the scan is checking for v0.

test_install_and_import FAILED
test_load               FAILED
test_eval               FAILED

test_load runs vf.load_environment(pkg), and test_eval checks isinstance(env, vf.SingleTurnEnv). A v1 taskset has neither, by design. load_environment in verifiers/utils/env_utils.py looks for a module-level load_environment function returning a v0 Environment; a taskset doesn't expose one. (The import failure was a missing verifiers dependency in my own pyproject, since fixed.)

I've worked around it by exposing both APIs from the same module: the v1 taskset, plus a legacy load_environment() returning a SingleTurnEnv built from the same reward functions. The scan passes and anyone still pinned to verifiers <0.2.0 keeps working. But I'd rather not carry the frozen API around indefinitely.

Two questions:

1. Is the scan due to be updated for v1, or is there a way to mark an environment as v1 so it runs different checks?
2. Which repo owns it? I couldn't find test_envs.py anywhere public, so I've filed here. Happy to move it.

Environment: https://app.primeintellect.ai/dashboard/environments/strickvl/isaf-extraction

[strickvl]

Follow-up, and this one is worse than the original report: declaring verifiers>=0.2.0 as a dependency makes a package uninstallable by default.

verifiers 0.2.0 is a stable release that depends on a pre-release: renderers>=0.1.8.dev40. Resolvers won't touch pre-releases unless asked, so the whole install fails:

$ uv pip install ./isaf-extraction
  × No solution found when resolving dependencies:
  ╰─▶ Because only renderers<0.1.8.dev40 is available and verifiers==0.2.0
      depends on renderers>=0.1.8.dev40, we can conclude that verifiers==0.2.0
      cannot be used.

That's what the Hub scan now hits, before any test runs (uv pip install in the venv fixture, exit 1).

So a v1 environment is stuck: it needs verifiers in its dependencies or import verifiers.v1 fails, and it can't have verifiers in its dependencies or the install fails. I suspect this is why the *_v1 examples in this repo don't declare verifiers at all — they don't have to, since they live here.

Workaround, if anyone else hits this: name the pre-release yourself, which opts the resolver in.

dependencies = ["verifiers>=0.2.0", "renderers>=0.1.8.dev40"]

Installs cleanly with default settings, and the Hub scan passes. But nobody should have to pin a transitive dependency they don't use. A stable renderers release, or a stable-compatible floor in verifiers' own pins, would fix it properly.

[xeophon]

@strickvl hey! We will update the platform for better v1 support shortly. Good catch with the pre-release issue, it is fixed on main and will be fixed with 0.2.1 generally.