[BUG] Cannot list tasks if the user doesn't install multilingual extras

[elichen3051]

Describe the bug

The following command will show the tasks supported by lighteval

lighteval tasks list

Because [multilingual] is an optional-dependencies in pyproject.toml, the users may expect the list command can work successfully without install with pip install lighteval[multilingual]. However, it will raise ModuleNotFoundError. After install langcodes, the problem still cannot be resolved.

╭────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────╮
│ /home/elichen/projects/lighteval_pr/lighteval-fork/src/lighteval/main_tasks.py:78 in list                                      │
│                                                                                                                                │
│   75 │   """List all tasks"""                                                                                                  │
│   76 │   from lighteval.tasks.registry import Registry                                                                         │
│   77 │                                                                                                                         │
│ ❱ 78 │   registry = Registry(custom_tasks=custom_tasks, load_community=True, load_extended=Tr                                  │
│   79 │   registry.print_all_tasks(suites=suites)                                                                               │
│   80                                                                                                                           │
│   81                                                                                                                           │
│                                                                                                                                │
│ /home/elichen/projects/lighteval_pr/lighteval-fork/src/lighteval/tasks/registry.py:172 in __init__                             │
│                                                                                                                                │
│   169 │   │   self._activate_loading_of_optional_suite()  # we dynamically set the loading par                                 │
│   170 │   │                                                                                                                    │
│   171 │   │   # We load all task to                                                                                            │
│ ❱ 172 │   │   self._task_registry = self._load_full_registry()                                                                 │
│   173 │   │                                                                                                                    │
│   174 │   │   self.task_to_configs = self._update_task_configs()                                                               │
│   175                                                                                                                          │
│                                                                                                                                │
│ /home/elichen/projects/lighteval_pr/lighteval-fork/src/lighteval/tasks/registry.py:265 in _load_full_registry                  │
│                                                                                                                                │
│   262 │   │                                                                                                                    │
│   263 │   │   # Need to load multilingual tasks                                                                                │
│   264 │   │   if self._load_multilingual:                                                                                      │
│ ❱ 265 │   │   │   import lighteval.tasks.multilingual.tasks as multilingual_tasks                                              │
│   266 │   │   │                                                                                                                │
│   267 │   │   │   custom_tasks_module.append(multilingual_tasks)                                                               │
│   268                                                                                                                          │
│                                                                                                                                │
│ /home/elichen/projects/lighteval_pr/lighteval-fork/src/lighteval/tasks/multilingual/tasks.py:26 in <module>                    │
│                                                                                                                                │
│     23 from functools import partial                                                                                           │
│     24 from itertools import permutations                                                                                      │
│     25                                                                                                                         │
│ ❱   26 from langcodes import Language as LangCodeLanguage                                                                      │
│     27 from langcodes import standardize_tag                                                                                   │
│     28                                                                                                                         │
│     29 from lighteval.metrics.dynamic_metrics import (                                                                         │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ModuleNotFoundError: No module named 'langcodes'

To Reproduce

Just install the lighteval with the following commmand with main branch

pip install -e lighteval[vllm,math]

And run the list tasks command

lighteval tasks list

or

lighteval tasks list --suites harness  # cannot work too.

Expected behavior

I expect the list command should work without [multilingual] installation

For example,

lighteval tasks list --suites harness # This command should work

Version info

Operating System

Distributor ID: Ubuntu
Description: Ubuntu 22.04.5 LTS
Release: 22.04
Codename: jammy

lighteval version

3a71f68 (current main)

[elichen3051]

I think this problem comes from def _load_full_registry in lighteval-fork/src/lighteval/tasks/registry.py

The workaround could be

        # Need to load multilingual tasks
        if self._load_multilingual:
            try:
                import lighteval.tasks.multilingual.tasks as multilingual_tasks  # <--- Error raised here.
                custom_tasks_module.append(multilingual_tasks)
            except ImportError:
                pass  # leave some message

[mohit-at-delhivery]

i am also facing the same issue. what's the fix? do we really need to install vllm via lighteval only? in my env i already had vllm so it should have worked but it didn't. Then tried creating a new env and getting this error:

python -m pip install -e lighteval[vllm,math]
ERROR: lighteval[vllm,math] is not a valid editable requirement. It should either be a path to a local project or a VCS URL (beginning with bzr+http, bzr+https, bzr+ssh, bzr+sftp, bzr+ftp, bzr+lp, bzr+file, git+http, git+https, git+ssh, git+git, git+file, hg+file, hg+http, hg+https, hg+ssh, hg+static-http, svn+ssh, svn+http, svn+https, svn+svn, svn+file).

[clefourrier]

@elichen3051 good catch! yes, we assume that to list all tasks, all deps are available - do you want to open a PR with your fix?