[FT] Python API docs using small model that can run on Mac

[jamesbraza]

Issue encountered

https://github.com/huggingface/lighteval/blob/v0.9.2/docs/source/using-the-python-api.mdx uses HuggingFaceH4/zephyr-7b-beta which is too big for a developer machine.

Solution/Feature

Using a model that is much smaller in docs.

Possible alternatives

N/a

[clefourrier]

Hi! Feel free to use a model from the SmolLM family for your use case!
https://huggingface.co/collections/HuggingFaceTB/smollm2-6723884218bcda64b34d7db9

@NathanHB we could add a comment in the doc to suggest a smollm model for devs with less than 16G of RAM, wdyt?

[jamesbraza]

To share, HuggingFaceTB/SmolLM2-135M-Instruct with the task "lighteval|triviaqa|0|0" and PipelineParameters(launcher_type=ParallelismManager.ACCELERATE, max_samples=5) still crashed my Mac with M3 Pro with 36 GB RAM.

[clefourrier]

Hmmmmmmm this is not normal at all - can you share precisely what you launched?

[clefourrier]

and the logs?

[clefourrier]

(For ref to know how much mem you should need for which model size: https://github.com/huggingface/evaluation-guidebook/blob/main/contents/troubleshooting/troubleshooting-inference.md#estimating-memory-requirements)

[jamesbraza]

Yeah it's with this code snippet (and I updated the task):

from datetime import timedelta

from accelerate import Accelerator, InitProcessGroupKwargs
from lighteval.logging.evaluation_tracker import EvaluationTracker
from lighteval.models.transformers.transformers_model import TransformersModelConfig
from lighteval.pipeline import ParallelismManager, Pipeline, PipelineParameters


if __name__ == "__main__":
    accelerator = Accelerator(
        kwargs_handlers=[InitProcessGroupKwargs(timeout=timedelta(seconds=3000))]
    )

    pipeline = Pipeline(
        tasks="original|mmlu:abstract_algebra|5|1",
        pipeline_parameters=PipelineParameters(
            launcher_type=ParallelismManager.ACCELERATE, max_samples=2
        ),
        evaluation_tracker=EvaluationTracker(output_dir="./results"),
        model_config=TransformersModelConfig(
            model_name="HuggingFaceTB/SmolLM2-135M-Instruct", use_chat_template=True
        ),
    )

    pipeline.evaluate()
    pipeline.save_and_push_results()
    pipeline.show_results()

My Mac just crashes during pipeline.evaluate with Python 3.12 and:

accelerate==1.4.0
lighteval==0.9.2
transformers==4.51.3

[clefourrier]

Could you share the full logs you get?

[jamesbraza]

Yeah here they are, then my Mac freezes and becomes non-responsive:

[2025-05-21 10:20:19,128] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to mps (auto detect)
W0521 10:20:19.278000 10400 torch/distributed/elastic/multiprocessing/redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.
INFO 05-21 10:20:20 [__init__.py:239] Automatically detected platform cpu.
--max_samples WAS SET. THESE NUMBERS ARE ONLY PARTIAL AND SHOULD NOT BE USED FOR COMPARISON UNLESS YOU KNOW WHAT YOU ARE DOING.
If you want to use extended_tasks, make sure you installed their dependencies using `pip install -e .[extended_tasks]`.

0it [00:00, ?it/s]