From ec431009144263234eab26e595b6b77f169adb88 Mon Sep 17 00:00:00 2001 From: singualrity-14 Date: Thu, 8 Oct 2026 08:43:28 +0530 Subject: [PATCH] Fix broken code examples in custom model docs Both runnable examples in evaluating-a-custom-model.mdx fail as written: - the CLI example is missing the closing quote on the task argument, so the shell swallows the following --max-samples 10 line into the string literal; - the Python API example passes tasks=truthfulqa:mc, a bare identifier followed by a colon, which raises SyntaxError; the task spec must be a string. Both task names (wmt20:fr-de, truthfulqa:mc) are valid, so only the quoting was wrong. --- docs/source/evaluating-a-custom-model.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/evaluating-a-custom-model.mdx b/docs/source/evaluating-a-custom-model.mdx index df9b3ea17..969feb59d 100644 --- a/docs/source/evaluating-a-custom-model.mdx +++ b/docs/source/evaluating-a-custom-model.mdx @@ -59,7 +59,7 @@ You can evaluate your custom model using either the command-line interface or th lighteval custom \ "google-translate" \ "examples/custom_models/google_translate_model.py" \ - "wmt20:fr-de \ + "wmt20:fr-de" \ --max-samples 10 ``` @@ -94,7 +94,7 @@ model_config = CustomModelConfig( # Create and run the pipeline pipeline = Pipeline( - tasks=truthfulqa:mc, + tasks="truthfulqa:mc", pipeline_parameters=pipeline_params, evaluation_tracker=evaluation_tracker, model_config=model_config