openai-realtime: OpenAIRealtimeWebRTC conversation.item.retrieve has no audio for model responses

[leecjohnny]

Describe the bug

When using openai-realtime webrtc transport, conversation.item.retrieve does not retrieve audio for model responses

Debug information

"@openai/agents": "^0.1.6",
"@openai/agents-openai": "^0.1.6",
"@openai/agents-realtime": "^0.1.6",

Repro steps

1. start RealtimeSession with webrtc transport
2. have at least 1 audio model response (assistant) in history
3. get the item id for the model response
4. send conversation.item.retrieve for the id
5. inspect the conversation.item.retrieved event for the item for base64 audio content

Expected behavior

If the model provided a response, the server representation returned by conversation.item.retrieved should have the audio in base64. The model is responding in audio, so the server representation should be in audio.

This is according to your docs:

retrieve event call: https://platform.openai.com/docs/api-reference/realtime-client-events/conversation/item/retrieve

retrieved event response: https://platform.openai.com/docs/api-reference/realtime-server-events/conversation/item/retrieved

[seratch]

Thanks for sharing this. I confirmed the event data includes only transcript but there is no audio data. Will share this with the relevant team members.

Code snippet:

Modified https://github.com/openai/openai-agents-js/tree/main/examples/realtime-next to have the following additional code

    session.current.on('transport_event', (event) => {
      if (event.type === 'response.output_audio.done') {
        console.log('response.output_audio.done', event);
        session.current?.transport.sendEvent({
          type: 'conversation.item.retrieve',
          event_id: event.event_id,
          item_id: event.item_id,
        });
      }
    });

Responses:

{
  "type": "response.output_audio.done",
  "event_id": "event_CKD5wmYgNHDUhB0nbOsFr",
  "item_id": "item_CKD5vRcdZI28pzwKcs2Ze",
  "content_index": 0,
  "output_index": 0,
  "response_id": "resp_CKD5v0JS7hsqEGu8elch7"
}

the returned client event by the above code snippet:

{
  "type": "conversation.item.retrieved",
  "event_id": "event_CKD5waOFRRPHuNmJJc7JU",
  "item": {
    "id": "item_CKD5vRcdZI28pzwKcs2Ze",
    "content": [
      {
        "transcript": "Good morning! How can I help you today?",
        "type": "output_audio"
      }
    ],
    "role": "assistant",
    "status": "completed",
    "type": "message"
  }
}

[juberti-oai]

this appears to be a regression on the GA interface, it works properly when using the beta interface

[leecjohnny]

Looks like this now resolved on the GA interface.

{
  "type": "conversation.item.retrieved",
  "event_id": "event_CbiaqiwoY4Uux7YiHaAU0",
  "item": {
    "id": "item_Cbiao5M5whL6gZWjeqpsW",
    "content": [
      {
        "audio": "BQAEAPv/[...]P3/", // base64 string omitted in full
        "transcript": "Hello, can you hear me?",
        "type": "input_audio"
      }
    ],
    "role": "user",
    "status": "completed",
    "type": "message"
  }
}

[filiphessle]

I'm still facing the issue. Within the same conversation, it works for certain items and doesn't for others. See the example below:

{
      "event_id": "event_CdLGgddk784jpTKyzFbl2",
      "item": {
            "id": "item_CdLGeh9dV36c8blLfBTSr",
            "object": "realtime.item",
            "type": "message",
            "status": "completed",
            "role": "assistant",
            "content": [
                  {
                        "type": "audio",
                        "transcript": "\"Okay\" isn’t enough. What are you going to do about it?",
                        "audio": "AQADAAEAAwAAAAQAAAABAA..."
                        "format": "pcm16"
                  }
            ]
      },
      "timestamp": 1763492583054
}


{
      "event_id": "event_CdLGoZNCgmXg5VmHGrZou",
      "item": {
            "id": "item_CdLGnVUwYJj8E6gUJrmxr",
            "object": "realtime.item",
            "type": "message",
            "status": "completed",
            "role": "assistant",
            "content": [
                  {
                        "type": "audio",
                        "transcript": "That’s not good enough. My daughter can’t wait—what’s your solution right now?"
                  }
            ]
      },
      "timestamp": 1763492590811,
}

[wsk-builds]

I took another pass against current upstream/main and the existing thread. I may be missing some current API-side behavior here, but I think a paired raw trace would make the SDK/API boundary much easier to reason about.

What I observed:

• The SDK has a historyStoreAudio option for whether audio data that is available to local history should be retained.
• The SDK also processes conversation.item.retrieved responses from the API and updates local history from that item.
• If the raw conversation.item.retrieve response does not include audio bytes for an assistant item, the SDK cannot reconstruct that audio locally. If the raw response does include audio but the SDK drops it, that would be a more direct SDK bug.

Would it be possible to share a minimal trace with:

• SDK version.
• Realtime model name.
• Whether this is WebRTC-only, or whether WebSocket reproduces it too.
• Whether historyStoreAudio is enabled.
• Confirmation that the WebRTC data channel is otherwise working normally.
• Raw response.output_audio.done events for an assistant item that later has audio and one that does not, if both happen in the same session.
• Raw conversation.item.created / conversation.item.added events for those assistant items.
• The exact conversation.item.retrieve request and raw conversation.item.retrieved response for the item with missing audio.
• The same retrieve trace for an item that does include audio, if available.

My read:

• If the raw conversation.item.retrieved response has only transcript/content metadata and no audio, this looks like API/server behavior or a docs gap rather than something the SDK can fix after the fact.
• If the raw retrieve response includes audio and historyStoreAudio: true is set, but the SDK history still loses it, then this points to an SDK-side handling bug.

Suggested next step:

• Attach a paired raw trace from one working item and one missing-audio item in the same session. That should make it clear whether the missing audio is already absent from the retrieve API response, or whether the SDK is dropping it after retrieval.