Skip to content

Speed up evaluate imports with lazy loading - #833

Open
MonikaDvorackova wants to merge 2 commits into
huggingface:mainfrom
MonikaDvorackova:perf/lazy-package-imports
Open

MonikaDvorackova wants to merge 2 commits into
huggingface:mainfrom
MonikaDvorackova:perf/lazy-package-imports

Conversation

@MonikaDvorackova

@MonikaDvorackova MonikaDvorackova commented Oct 20, 2026 •

Copy link
Copy Markdown

Summary

Addresses #674.

  • Replace eager package-level imports with lazy loading.
  • Avoid importing heavy dependencies such as datasets, pandas, and pyarrow during a plain import evaluate.
  • Preserve package-level API exports, including the callable evaluate.evaluator.
  • Add import benchmarks and regression tests.

Benchmark

Representative measurements on macOS:

Import Before After
import evaluate 14.914 s 0.119 s
from evaluate import load 14.099 s 7.238 s
from evaluate import EvaluationSuite 9.995 s 15.325 s
from evaluate import evaluator 11.657 s 11.220 s

Import timings varied substantially between runs. The main improvement is consistently observed for plain import evaluate; other imports may still load heavy dependencies on first access.

Validation

  • 11 regression tests passed.
  • Ruff lint and formatting checks passed.
  • Verified 50 public exports and evaluator compatibility.

The complete test suite was not run because some optional dependencies were not installed.

Additional local benchmark (Python 3.10, macOS)

Seven subprocess measurements per import; median values.

Import Upstream main PR branch
import evaluate 1.419 s 0.024 s
from evaluate import load 1.563 s 0.768 s
from evaluate import EvaluationSuite 1.456 s 1.389 s
from evaluate import evaluator 1.313 s 1.528 s

Plain import evaluate was approximately 59x faster.
The evaluator import was slower in this measurement.
Results may vary across runs and environments.

Additional validation:

  • 6 focused lazy-import tests passed.
  • All 50 original public names remained accessible.
  • Wildcard exports were verified.
  • Regression tests added in commit b6c1136.

The complete project test suite has not yet been verified in this environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant