Skip to content

[BUG] Python API documentation #783

Open
@ryantzr1

Description

@ryantzr1

Describe the bug

The Python API documentation example contains invalid parameters that cause runtime errors. The PipelineParameters constructor doesn't accept env_config or override_batch_size parameters.

https://huggingface.co/docs/lighteval/main/en/using-the-python-api

To Reproduce

  1. Copy the example code from the Python API documentation
  2. Run the script
  3. Get AttributeError for non-existent parameters
pipeline_params = PipelineParameters(
    launcher_type=ParallelismManager.ACCELERATE,
    env_config=EnvConfig(cache_dir="tmp/"),  # ❌ This parameter doesn't exist
    custom_task_directory=None,
    override_batch_size=1,  # ❌ This parameter doesn't exist
    max_samples=10
)

Expected behavior

The documentation example should use valid PipelineParameters constructor arguments that actually work.

Version info

  • cloned main branch repo (codebase up to date)
  • Lighteval Version: 0.9.1.dev0

Suggested Fix

Update the documentation example to use correct parameter names:

pipeline_params = PipelineParameters(
    launcher_type=ParallelismManager.ACCELERATE,
    custom_tasks_directory=None, 
    max_samples=10
)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions