Search results

[BUG] Faithfulness metric fails because the SummaCZS model is instantiated but never assigned

bugSomething isn't workingSomething isn't working

Status: Open.

#794 In huggingface/lighteval;

· sahilds1 opened on Jun 5, 2025

[BUG] support direct evaluation of local API and local data sets?

bugSomething isn't workingSomething isn't working

Status: Open.

#793 In huggingface/lighteval;

· HemiFate opened on Jun 5, 2025

[FT] Store system prompt in results

Status: Open.

#788 In huggingface/lighteval;

· lewtun opened on Jun 3, 2025

[BUG] OpenAIClient fails when using newer GPT models

bugSomething isn't workingSomething isn't working

Status: Open.

#785 In huggingface/lighteval;

· ryantzr1 opened on May 27, 2025

[BUG] Python API documentation

bugSomething isn't workingSomething isn't working

Status: Open.

#783 In huggingface/lighteval;

· ryantzr1 opened on May 27, 2025

[BUG] Is AIME24 broken?

bugSomething isn't workingSomething isn't working

Status: Open.

#771 In huggingface/lighteval;

· eldarkurtic opened on May 22, 2025

[FT] Add tests for nanotron

feature/enhancementNew feature/requestNew feature/request

good first issueGood for newcomersGood for newcomers

help wantedExtra attention is neededExtra attention is needed

ignore-for-releasewill be ignored when doing release notewill be ignored when doing release note

Status: Open.

Feature

#765 In huggingface/lighteval;

· NathanHB opened on May 21, 2025

[FT] Support computing/displaying stddev in the final table from the variation in sample scores (when using sampling metrics) instead of a bootstrap

Status: Open.

#763 In huggingface/lighteval;

· jamesbraza opened on May 21, 2025

[FT] Python API docs using small model that can run on Mac

bugSomething isn't workingSomething isn't working

documentationImprovements or additions to documentationImprovements or additions to documentation

Status: Open.

Bug

#761 In huggingface/lighteval;

· jamesbraza opened on May 21, 2025

[BUG] custom model docs don't run: missing imports

bugSomething isn't workingSomething isn't working

documentationImprovements or additions to documentationImprovements or additions to documentation

good first issueGood for newcomersGood for newcomers

Status: Open.

Bug

#760 In huggingface/lighteval;

· jamesbraza opened on May 21, 2025

[BUG] incorrect type hints such as <code>callable</code>

bugSomething isn't workingSomething isn't working

documentationImprovements or additions to documentationImprovements or additions to documentation

feature/enhancementNew feature/requestNew feature/request

Status: Open.

Bug

#759 In huggingface/lighteval;

· jamesbraza opened on May 21, 2025

[BUG] Make Probability metric not trigger Loglikehood requests twice when used alongside Accuracy Metrics

bugSomething isn't workingSomething isn't working

Status: Open.

Bug

#751 In huggingface/lighteval;

· hynky1999 opened on May 20, 2025