Incompatibility between OpenAI and HF's Chat Completion `response_format`

[sadra-barikbin]

Hi there!

OpenAI and HF api seem to have diverged on chat_completion response_format. That of the OpenAI has a type attribute which takes one of three values text, json_object (free-form JSON generation which isn't in TGI yet) and json_schema, and for the last one it has another json_schema attribute which itself contains name, schema, description and strict attributes.

On the other hand, HF's response_format: ChatCompletionInputGrammarType contains type and value attributes.

This might break the OpenAI api usability on HF models. Could we have compatibility between these two on this matter? Or does it need probably breaking changes in TGI and huggingface_hub?

@Wauplin

[Wauplin]

Cc @drbh @Narsil @OlivierDehaene on the TGI side

[Cristy94]

I have the same issue, I could easily add the Gemini and ChatGPT API, but with the Huggingface API I have to change my code and define JSON schemas, because both Gemini and GPT-4 work without a schema defined too.

Can we get JSON responses without passing any value for the response_format ?

Now I do this, I don't need to pass any schema and it works well (for Gemini/ChatGPT):
response_format: json ? { type: "json_object" } : undefined

[Cristy94]

If you try to simply use the OpenAI format, you get this error:

Failed to deserialize the JSON body into the target type: response_format.type: unknown variant `json_schema`, expected one of `json`, `json_object`, `regex`

If you try to pass the required HF format to openai-node:

   return {
                type: "json",
                value: schema,
        }

The request will fail with the error:

UnprocessableEntityError: 422 "Input validation error: cannot compile regex from schema: 'type' must be a string"

So, I assume the schema is still not properly sent.

Any way to use HF chat completion with response_format json or json_schema through openai-node?

EDIT: It seems to work, the problem was with the JSON schema.

This works in OpenAI
description: { "type": ["string", "null"] },

I had to change it to this for HF:

description: { anyOf: [{ type: "string" }, { type: "null" }] },

[lhoestq]

The issue is value is currently mandatory in TGI, see huggingface/text-generation-inference#2899. Hopefully this gets fixed and we can update ChatCompletionInputGrammarType here as well