Skip to content

task: ensure calls to json.dumps allow non-ascii for language support#1630

Open
jmartin-tech wants to merge 1 commit intoNVIDIA:mainfrom
jmartin-tech:task/survey-json-dump-lang-support
Open

task: ensure calls to json.dumps allow non-ascii for language support#1630
jmartin-tech wants to merge 1 commit intoNVIDIA:mainfrom
jmartin-tech:task/survey-json-dump-lang-support

Conversation

@jmartin-tech
Copy link
Collaborator

Updates calls to json.dumps that flow to file writes or other serialized locations to allow non-ascii characters. This improves compatibility with the experimental machine language translation.

When not passed the disabling parameter ensure_ascii=False json dump may escape characters mangling them from the original input or output values. This reduces the accuracy of many components.

Verification

List the steps needed to make sure this thing works

  • Preform various test runs utilizing translation to languages that commonly use non-ascii characters
  • Reprocess results using utility tooling in the analyze package
  • Verify full runs complete without raising json serialization errors
  • Verify reprocessed reports do not modify prompt or response contents for non-ascii characters

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant