Realtime agent stuck in a conversational loop after 0.1.0 update

[gregory-buffard]

Hello OpenAI community :

We are facing an issue related to the Realtime Agent configuration using the Agents SDK where after the latest update (0.1.0), our Realtime Agent just runs into a loop, here's a conversation example :

Agent : Good morning! Before starting, are you an individual or a professional?
Agent : Good morning! Before starting, are you an individual or a professional?
User : individual
Agent : Good morning! Before starting, are you an individual or a professional?
User : individual...
Agent: Good, how may I help you ?
User : I need ... { user clearly asks for a specific need }
Agent : Good, how may I help you ?
User : I need ... { user clearly asks for a specific need again }
User : { agent doesn't respond so user asks for need again in a simple wording }
User : { agent doesn't respond again so user asks for need again in a simple wording and then ends the conversation }

As you may conclude, the implementation is very poorly handled, the agent not only repeats itself for no apparent reason and also sometimes doesn't event answer.

We thought the problem might come from the prompt, which we have altered on many iterations but the issue doesn't seem to stem from here since this problem is also observed on many very different prompts. Here's the example of the previous conversation prompt used for the agent (the prompt is translated from French, apologies for possible confusion in wording).

As a result, we have rewinded to the previous 0.17.0 version, which seem to work okay [although it's not really good neither in terms of quality of speech and intelligence compared to the experience we have access to in the playground or the app strangely... (e.g., lack of emotions in the voice, "fake breathing", etc.")].
Additionally, the issue seems to persist even on 0.17.0 only when we select the newly released gpt-realtime model.

We're out of options on how to fix this issue, hence why I'm turning towards the Agent SDK community.
Finally, our very simple code implementation of the agent can be found in the openai.ts file.

Does anyone else face a similar issue ? If so, have you managed to resolve it ? How ?
We'll enormously appreciate any kind of help on this matter since we're desperately out of possibilities...

Do not hesitate to let me know if any additional code resources are needed for better understanding of the matter.

[gregory-buffard]

Update : The loop problem seems not to be occurring on the new 0.1.1 when using legacy models (i.e., GPT 4o Realtime and GPT 4o-mini Realtime) but persists on the new GPT Realtime (gpt-realtime)

[joneriktalvio]

We are facing exactly the same issue in Finnish language. We tested out also English with fast and short answers like "okay" and could replicate it with English also.
We even face the same issue sometimes with longer answers.
One common thing also what we have seen is that it seems to be triggering in most of the cases on the first answers. Meaning;

IF INBOUND CALL (Twilio < WS SERVER > OpenAI)

1. We create response with response.create (Greet the person)
2. Agent: "Hi, thanks for callling..."
3. User: "Hello, wanted to do this and that..."
4. Agent: "Hi, thanks for calling..."

IF OUTBOUND CALL (Twilio < WS SERVER > OpenAI)

1. User: "Hello?"
2. Agent: "Hi, I'm calling you from x y z and about cats."
3. User: "Okay?"
4. Agent: "Hi, I'm calling you from x y z and about cats."

^ This can be replicated even tho having the whole prompt in English and especially saying the prompt suggested by OpenAI:

Unclear audio

• Always respond in the same language the user is speaking in, if unintelligible.
• Only respond to clear audio or text.
• If the user's audio is not clear (e.g. ambiguous input/background noise/silent/unintelligible) or if you did not fully hear or understand the user, ask for clarification using {preferred_language} phrases.

Variety

• Do not repeat the same sentence twice.
• Vary your responses so it doesn't sound robotic.

This all happens even tho we can clearly see in the logs that the audio was transcribed correctly.
So I can confirm you that indeed it looks like an issue with the model itself.

EDIT: (even tho I didn't even post this one yet, lol)
We tested out with the gpt-4o-realtime-preview-2025-06-03 also, it does it less, but in some cases it also triggers the same issue.

I tested out also server_vad and semantic_vad with different settings.

This kinda reminded me of the first versions of whisper model, where it hallucinated some really weird text when the audio was too short for it.

[seratch]

Thanks for sharing the details. It seems you both are using Twilio transport. If possible, may I ask you to check if the same issue occurs with OpenAI connection only transport (meaning the default WebRTC)?

[gregory-buffard]

We have an alternative platform on which we tested GPT Realtime via WebRTC, since it's not possible to use Twilio X WebRTC on the originally provided one because Twilio cannot work with WebRTC on telephony, and it seems to work fine there, however it should be noted that it doesn't use the SDK but manual connection...
Shall we test it rather directly through the Agents SDK or does this clarify the origin of the issue for you enough ?

[joneriktalvio]

I created a really simple WebRTC connection with NextJS to use my microphone.

Works flawlessly with WebRTC.
No issue in there.

[joneriktalvio]

@gregory-buffard have you found any workarounds? I was thinking of LiveKit, but that would kinda then make openai agents sdk obsolete 😅

[gregory-buffard]

Hey @joneriktalvio!

I have no experience with LiveKit but looking at the docs it seems like a good alternative and also provider-modular which could really come handy for such reliability issues as the one we're currently experiencing with OpenAI.

I think we might migrate to LiveKit eventually, but right now, we've just decided to use Google GenAI with Gemini API as our only "workaround" (lol)... Seems a lot more reliable at the moment until OpenAI fixes their SDK.

[mattlanham]

Also experiencing this issue!