Skip to content

Normalize multiple choice responses like "0: Yes" to code-only format#2385

Open
Fizza-Mukhtar wants to merge 1 commit intoexpectedparrot:mainfrom
Fizza-Mukhtar:fix-use-code-validation
Open

Normalize multiple choice responses like "0: Yes" to code-only format#2385
Fizza-Mukhtar wants to merge 1 commit intoexpectedparrot:mainfrom
Fizza-Mukhtar:fix-use-code-validation

Conversation

@Fizza-Mukhtar
Copy link

This PR addresses issue #2356.

Problem:

  • When use_code=True for multiple choice questions, the LLM sometimes returns responses like "0: Yes" instead of just "0".
  • This causes QuestionAnswerValidationError because the strict literal validation expects only the numeric code.

Solution:

  • Added normalization in MultipleChoiceResponseValidator.fix() to split responses containing ":" and keep only the numeric code.
  • Applied only for string responses when use_code=True.
  • Existing validation and matching strategies (exact, case-insensitive, prefix matching) remain unchanged.

This ensures that the validator accepts responses like "0: Yes" while keeping the validation strict for other cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant