Open
Description
Describe the bug
The Python API documentation example contains invalid parameters that cause runtime errors. The PipelineParameters constructor doesn't accept env_config or override_batch_size parameters.
https://huggingface.co/docs/lighteval/main/en/using-the-python-api
To Reproduce
- Copy the example code from the Python API documentation
- Run the script
- Get AttributeError for non-existent parameters
pipeline_params = PipelineParameters(
launcher_type=ParallelismManager.ACCELERATE,
env_config=EnvConfig(cache_dir="tmp/"), # ❌ This parameter doesn't exist
custom_task_directory=None,
override_batch_size=1, # ❌ This parameter doesn't exist
max_samples=10
)
Expected behavior
The documentation example should use valid PipelineParameters constructor arguments that actually work.
Version info
- cloned main branch repo (codebase up to date)
- Lighteval Version: 0.9.1.dev0
Suggested Fix
Update the documentation example to use correct parameter names:
pipeline_params = PipelineParameters(
launcher_type=ParallelismManager.ACCELERATE,
custom_tasks_directory=None,
max_samples=10
)