Repository navigation
Fix broken code examples in custom model docs - #1421
Open
singularity-14 wants to merge 1 commit into
Open
singularity-14 wants to merge 1 commit into
singularity-14 wants to merge 1 commit into
Conversation
Both runnable examples in evaluating-a-custom-model.mdx fail as written: - the CLI example is missing the closing quote on the task argument, so the shell swallows the following --max-samples 10 line into the string literal; - the Python API example passes tasks=truthfulqa:mc, a bare identifier followed by a colon, which raises SyntaxError; the task spec must be a string. Both task names (wmt20:fr-de, truthfulqa:mc) are valid, so only the quoting was wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both runnable examples in
docs/source/evaluating-a-custom-model.mdxfail as written.CLI example
The task argument is missing its closing quote, so the shell swallows the following
--max-samples 10line into the string literal:lighteval custom \ "google-translate" \ "examples/custom_models/google_translate_model.py" \ - "wmt20:fr-de \ + "wmt20:fr-de" \ --max-samples 10Python API example
tasks=truthfulqa:mcis a bare identifier followed by a colon, which is aSyntaxError— the example cannot run at all:Notes
Both task names are valid —
wmt20:fr-deinsrc/lighteval/tasks/tasks/sacrebleu.pyandtruthfulqa:mcinsrc/lighteval/tasks/tasks/truthfulqa.py— so only the quoting was wrong, and the examples' intent is unchanged.Verified by parsing every
pythonblock in the file withast.parseand tokenizing everybashblock withshlex.split; both now pass. The CLI example tokenizes to the three positional argumentscustomexpects (model_name,model_definition_file_path,tasks) plus--max-samples 10.Relation to #760
Partially addresses #760. The imports originally reported there look already fixed on
main(ParallelismManageris imported, andEnvConfigis gone); what remained was the examples still not running, which is what this PR fixes.The base-class concern raised later in that thread is a separate, larger change and isn't touched here — so I'd suggest leaving #760 open to track it, unless you'd rather close it.