Realtime SDK + Azure: unsupported fields; initial config ignored

[AlexRRR]

Summary

When using the Agents Realtime SDK with Azure OpenAI Realtime (WebRTC), the SDK emits OpenAI-specific session fields that Azure rejects (e.g., session.type, output_modalities). Also, with WebRTC the session is created during SDP, so initial agent config (instructions/tools) is ignored until the data channel opens; subsequent SDK-driven session.update calls include fields Azure doesn’t accept, preventing config from applying cleanly.

Notes

I recognize these incompatibilities stem primarily from Azure OpenAI’s Realtime implementation differing from OpenAI’s GA interface (for example, rejecting session.type, output_modalities, and disallowing ["audio"]-only modalities). Still, I wanted to open this issue so the community has a clear record of the symptoms, error messages, and a practical workaround. Even if the “fix” ultimately belongs on the Azure side (which btw I have no idea where to report), a provider-aware path or extensibility hook in the SDK would help developers avoid common pitfalls and ship faster across providers.

Environment

• @openai/agents-realtime: 0.1.0
• @openai/agents: 0.1.0
• openai: 5.19.1
• Next.js: 15.5.2
• React: 19.1.1
• Node.js: 20.x
• Client: Chrome (WebRTC, ephemeral client key)
• Provider: Azure OpenAI Realtime (region WebRTC endpoint)
• gpt-realtime

Repro steps

1. Browser, WebRTC:

   import { RealtimeAgent, RealtimeSession } from "@openai/agents-realtime";
   
   const agent = new RealtimeAgent({ name: "Assistant", instructions: "..." });
   const session = new RealtimeSession(agent, {
     model: "gpt-realtime",
     // baseUrl points to Azure WebRTC endpoint
     // e.g. https://<region>.realtimeapi-preview.ai.azure.com/v1/realtimertc
   });
   
   await session.connect({
     apiKey: "<ephemeral>",
     url: "https://<region>.realtimeapi-preview.ai.azure.com/v1/realtimertc",
   });

2. Observe transport events (session.created, then session.update).

Observed behavior

• session.created shows Azure defaults (expected for WebRTC).
• SDK sends session.update with OpenAI-only fields:
  • Azure error: Unknown parameter: session.type.
  • Azure error: Unknown parameter: session.output_modalities.
• Attempting to set modalities: ["audio"]:
  • Azure error: Invalid modalities. Supported: ["text"] or ["audio","text"].
• Agent instructions/tools often don’t take effect because the follow-up session.update is rejected due to unsupported fields.

Example error events (from logs):

• Unknown parameter: 'session.type'
• Unknown parameter: 'session.output_modalities'
• Invalid modalities: ['audio']

Expected behavior

• SDK should allow provider-aware session.update payloads and avoid sending fields the provider does not support.
• Post-connect updates should apply minimal, Azure-accepted changes (e.g., instructions, tools, tool_choice, voice, model) without error.

Workarounds that work

• Custom WebRTC transport that strips OpenAI-only fields and allowlists Azure-accepted keys for session.update. Also avoid sending session.type in tracing updates.

  import { OpenAIRealtimeWebRTC } from "@openai/agents-realtime";
  
  class AzureRealtimeWebRTC extends OpenAIRealtimeWebRTC {
    protected _updateTracingConfig(tracing: unknown) {
      this.sendEvent({
        type: "session.update",
        session: { tracing: tracing === "auto" ? "auto" : tracing },
      });
    }
  
    updateSessionConfig(config: Partial<Record<string, unknown>>) {
      const merged = this._getMergedSessionConfig(config) as Record<string, unknown>;
      const session: Record<string, unknown> = {};
      if (typeof merged.model === "string") session.model = merged.model;
      if (typeof merged.instructions === "string") session.instructions = merged.instructions;
      if (Array.isArray(merged.tools)) session.tools = merged.tools;
      if (typeof merged.tool_choice !== "undefined") session.tool_choice = merged.tool_choice;
      if (typeof merged.voice === "string") session.voice = merged.voice;
      this.sendEvent({ type: "session.update", session }); // no type/output_modalities/audio/tracing
    }
  
    async connect(options: any) {
      // Ensure Azure SDP URL includes ?model=<deployment>
      const baseUrl = options?.url;
      const model = options?.model;
      let urlToUse = baseUrl;
      if (baseUrl && model) {
        try {
          const u = new URL(baseUrl);
          if (!u.searchParams.has("model")) u.searchParams.set("model", model);
          urlToUse = u.toString();
        } catch { /* keep baseUrl */ }
      }
      return await super.connect({ ...options, url: urlToUse });
    }
  }

• Use minimal initial agent; apply instructions/tools right after connect via a minimal session.update.
• Keep tracing: null to avoid tracing-shaped updates that include unsupported fields.

Usage of workaround:

// usage in a browser (WebRTC)
import { RealtimeAgent, RealtimeSession } from "@openai/agents-realtime";
import { AzureRealtimeWebRTC } from "./azure-realtime-transport";

// Keep initial agent minimal; Azure creates session during SDP, then we update
const agent = new RealtimeAgent({
  name: "Assistant",
  instructions: "",
});

// Azure region WebRTC endpoint (no model param needed here; transport adds it)
const baseUrl = "https://<region>.realtimeapi-preview.ai.azure.com/v1/realtimertc";
const model = "gpt-realtime"; // your Azure deployment name

const transport = new AzureRealtimeWebRTC({
  baseUrl,
  useInsecureApiKey: true, 
});

const session = new RealtimeSession(agent, {
  model,
  transport,
  config: { tracing: null }, // avoid tracing-shaped updates Azure may reject
});

// Obtain ephemeral client key from your backend first
await session.connect({
  apiKey: "<ephemeral_client_key>",
  url: baseUrl,
  model,
  initialSessionConfig: {
    instructions: "You are a helpful assistant. Always use tools when available.",
    tools: [ your_tools],
  },
});

Why this matters

• The SDK defaults are tuned for OpenAI GA. Azure’s Realtime (preview) differs and rejects certain fields. Without a provider-aware path, common SDK flows (initialSessionConfig, updateAgent) fail on Azure.

Proposed solutions

• Add a provider adapter/flag (provider: "openai" | "azure") that adjusts session.update payloads.
• Or expose a hook to transform outgoing session.update before sendEvent.
• Make tracing updates optional and avoid session.type when disabled; avoid output_modalities for non-OpenAI providers.
• Document provider differences (Azure WebRTC: session created at SDP; valid modalities; minimal post-connect updates).

[seratch]

Thanks for sharing the details of the incompatibility with Azure. You're right that our GA interface is different from the one Azure currently provides. For now, please continue using version 0.0.17 for your use case. In the long run, if Azure endpoint won't support our GA interface, we may consider adding something to the extension package (or place something under examples directory, like we did for ai-sdk v1).

[AlexRRR]

Thank you for the quick response. I can confirm that version 0.0.17 is working as expected.
However, this feels quite limiting. If I understand correctly, the Azure-hosted version won't support images or MCP Servers, and what I find most disappointing is the restriction to not having audio as the only modality. I've encountered an issue in the past where, after several tool calls, the model sometimes switches to text mode and remains there until a new session.update is triggered when having multiple modalities

[AdamosHadjivasiliou]

Do we have any updates on the mcp servers in the azure version?

[amitevski]

I hit the same limitation today, but using the WebSocket API. It looks like there were changes in the session.update request structure which is not accepted by Azure OpenAI realtime endpoint.

new version at: https://github.com/openai/openai-agents-js/blob/v0.1.0/packages/agents-realtime/src/openaiRealtimeBase.ts#L526

previous version which is supported by azure: https://github.com/openai/openai-agents-js/blob/v0.0.17/packages/agents-realtime/src/openaiRealtimeBase.ts#L386

For german privacy reasons we want to use azure. @seratch Are you planning to support azure in future or should we rather look for other libraries?

[Yingrjimsch]

I am setting the modalities modalities: ["audio", "text"] and tools tools: [{ type: "mcp", server_label: "N8N", server_url: process.env.N8N_ENDPOINT, require_approval: "never"},

But i the history_updated event never contains content for the assistant and if i set the tools i get a connection error that the type mcp is invalid. @AlexRRR did you manage to do that?

[akhromow]

I think it makes sense to mention similar issue in python sdk here openai/openai-agents-python#1748

the last comment from @chris-lindstrom helped me to connect to azure realtime model using js sdk by adjusting url accordingly.

[lateef-k]

In case anyone is having any issues with the session.connect bit, it's supposed to be:

  const agent = new RealtimeAgent({
    name: "assistant",
    instructions: "instructions here",
    tools: [tool1, tool2],
  });

  await session.connect({
    apiKey: session_key,
    url: import.meta.env.VITE_AZURE_OPENAI_WEBRTC_ENDPOINT,
    model: "gpt-realtime-mini",
  });


  session.transport.updateSessionConfig({
    instructions: 'instructions here'
    tools: [tool1, tool2],
  });

Namely:

1. initialSessionConfig is not defined for session.connect for 0.1.11, so you instead call updateSessionConfig after connecting to the session, it will activate your instructions and tool definitions
2. Critically though, you need to define the tools both on the Agent and in the updateSessionConfig. If you only do it for the updateSessionConfig, your agent will be aware of the tools but trying to execute them will throw an error. I believe they need to be part of the agent definition to be callable at all.

[AldrinSalazar]

We have been facing the same issue and even with workarounds I think it is important to be able to use the latest OpenAI SDKs when working with Azure deployed resources, and this prevents that.

[kungfucode-rex]

Same error Missing required parameter: session.type. we encountered when using the Azure GA protocol with:

"@openai/agents": "0.0.5",
"@openai/agents-realtime": "0.3.3", 

Hi @seratch, you said "if Azure endpoint won't support our GA interface, we may consider adding something to the extension package (or place something under examples directory, like we did for ai-sdk v1)". So I have a question:

• How to know "if Azure endpoint won't support our GA interface"? Azure will have an official announcement, or how long do we have to wait before they still don't support it, even if they don't?

[randomkms]

Since today, session.connect just infinitely connecting. Is it just me, or is everyone having the same issue?

UPDATE: Nevermind, seems like it was my internet provider's issue.