[BUG] Cannot list tasks if the user doesn't install multilingual extras #973
Description
Describe the bug
The following command will show the tasks supported by lighteval
lighteval tasks list
Because [multilingual] is an optional-dependencies in pyproject.toml, the users may expect the list command can work successfully without install with pip install lighteval[multilingual]. However, it will raise ModuleNotFoundError. After install langcodes, the problem still cannot be resolved.
╭────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────╮
│ /home/elichen/projects/lighteval_pr/lighteval-fork/src/lighteval/main_tasks.py:78 in list │
│ │
│ 75 │ """List all tasks""" │
│ 76 │ from lighteval.tasks.registry import Registry │
│ 77 │ │
│ ❱ 78 │ registry = Registry(custom_tasks=custom_tasks, load_community=True, load_extended=Tr │
│ 79 │ registry.print_all_tasks(suites=suites) │
│ 80 │
│ 81 │
│ │
│ /home/elichen/projects/lighteval_pr/lighteval-fork/src/lighteval/tasks/registry.py:172 in __init__ │
│ │
│ 169 │ │ self._activate_loading_of_optional_suite() # we dynamically set the loading par │
│ 170 │ │ │
│ 171 │ │ # We load all task to │
│ ❱ 172 │ │ self._task_registry = self._load_full_registry() │
│ 173 │ │ │
│ 174 │ │ self.task_to_configs = self._update_task_configs() │
│ 175 │
│ │
│ /home/elichen/projects/lighteval_pr/lighteval-fork/src/lighteval/tasks/registry.py:265 in _load_full_registry │
│ │
│ 262 │ │ │
│ 263 │ │ # Need to load multilingual tasks │
│ 264 │ │ if self._load_multilingual: │
│ ❱ 265 │ │ │ import lighteval.tasks.multilingual.tasks as multilingual_tasks │
│ 266 │ │ │ │
│ 267 │ │ │ custom_tasks_module.append(multilingual_tasks) │
│ 268 │
│ │
│ /home/elichen/projects/lighteval_pr/lighteval-fork/src/lighteval/tasks/multilingual/tasks.py:26 in <module> │
│ │
│ 23 from functools import partial │
│ 24 from itertools import permutations │
│ 25 │
│ ❱ 26 from langcodes import Language as LangCodeLanguage │
│ 27 from langcodes import standardize_tag │
│ 28 │
│ 29 from lighteval.metrics.dynamic_metrics import ( │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ModuleNotFoundError: No module named 'langcodes'
To Reproduce
Just install the lighteval with the following commmand with main branch
pip install -e lighteval[vllm,math]
And run the list tasks command
lighteval tasks list
or
lighteval tasks list --suites harness # cannot work too.
Expected behavior
I expect the list command should work without [multilingual] installation
For example,
lighteval tasks list --suites harness # This command should work
Version info
Operating System
Distributor ID: Ubuntu
Description: Ubuntu 22.04.5 LTS
Release: 22.04
Codename: jammy
lighteval version
3a71f68 (current main)