Skip to content

Fix broken code examples in custom model docs - #1421

Open
singularity-14 wants to merge 1 commit into
huggingface:mainfrom
singularity-14:fix-custom-model-doc-examples
Open

singularity-14 wants to merge 1 commit into
huggingface:mainfrom
singularity-14:fix-custom-model-doc-examples

Conversation

@singularity-14

Copy link
Copy Markdown

Both runnable examples in docs/source/evaluating-a-custom-model.mdx fail as written.

CLI example

The task argument is missing its closing quote, so the shell swallows the following --max-samples 10 line into the string literal:

 lighteval custom \
     "google-translate" \
     "examples/custom_models/google_translate_model.py" \
-    "wmt20:fr-de \
+    "wmt20:fr-de" \
     --max-samples 10

Python API example

tasks=truthfulqa:mc is a bare identifier followed by a colon, which is a SyntaxError — the example cannot run at all:

 pipeline = Pipeline(
-    tasks=truthfulqa:mc,
+    tasks="truthfulqa:mc",
     pipeline_parameters=pipeline_params,
     evaluation_tracker=evaluation_tracker,
     model_config=model_config
 )

Notes

Both task names are valid — wmt20:fr-de in src/lighteval/tasks/tasks/sacrebleu.py and truthfulqa:mc in src/lighteval/tasks/tasks/truthfulqa.py — so only the quoting was wrong, and the examples' intent is unchanged.

Verified by parsing every python block in the file with ast.parse and tokenizing every bash block with shlex.split; both now pass. The CLI example tokenizes to the three positional arguments custom expects (model_name, model_definition_file_path, tasks) plus --max-samples 10.

Relation to #760

Partially addresses #760. The imports originally reported there look already fixed on main (ParallelismManager is imported, and EnvConfig is gone); what remained was the examples still not running, which is what this PR fixes.

The base-class concern raised later in that thread is a separate, larger change and isn't touched here — so I'd suggest leaving #760 open to track it, unless you'd rather close it.

Both runnable examples in evaluating-a-custom-model.mdx fail as written:

- the CLI example is missing the closing quote on the task argument, so the
  shell swallows the following --max-samples 10 line into the string literal;
- the Python API example passes tasks=truthfulqa:mc, a bare identifier
  followed by a colon, which raises SyntaxError; the task spec must be a string.

Both task names (wmt20:fr-de, truthfulqa:mc) are valid, so only the quoting
was wrong.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant