RealtimeAgent is not working with Azure OpenAI

[monuminu]

I am trying to use RealtimeAgent with Azure OpenAI .

const agent = new RealtimeAgent({
  name: "Voice Agent",
  model: new OpenAIChatCompletionsModel({
    client,
    model: process.env.AZURE_OPENAI_DEPLOYMENT || "gpt-4o-realtime-preview",
  }),
  instructions:
    "You are a voice agent that can answer questions and help with tasks.",
  handoffs: [weatherAgent],
});

What is the right way to call Azure OpenAI ? i am able to get the empirical tokens.

[dkundel-openai]

Hey @monuminu! The Realtime Agents don't use the ChatCompletions Model.

Instead you should be able to specify a different URL when calling session.connect({ url: 'your-azure-url', apiKey: 'your-azure-api-key }).

Let me know if that works.

[monuminu]

Hi @dkundel-openai , i have done that but unfortunately it still call the https://api.openai.com/v1/realtime . Sharing you the page.tsx code.

import { useRef, useState } from "react";
import {
  RealtimeAgent,
  RealtimeItem,
  RealtimeSession,
  tool,
} from "@openai/agents/realtime";
import { getSessionToken } from "./server/token";
import z from "zod";

const getWeather = tool({
  name: "getWeather",
  description: "Get the weather in a given location",
  parameters: z.object({
    location: z.string(),
  }),
  execute: async ({ location }) => {
    return `The weather in ${location} is sunny`;
  },
});

const weatherAgent = new RealtimeAgent({
  name: "Weather Agent",
  instructions: "Talk with a New York accent",
  handoffDescription: "This agent is an expert in weather",
  tools: [getWeather],
});

const agent = new RealtimeAgent({
  name: "Voice Agent",
  instructions:
    "You are a voice agent that can answer questions and help with tasks.",
  handoffs: [weatherAgent],
});

export default function Home() {
  const session = useRef<RealtimeSession | null>(null);
  const [connected, setConnected] = useState(false);
  const [history, setHistory] = useState<RealtimeItem[]>([]);

  async function onConnect() {
    if (connected) {
      setConnected(false);
      await session.current?.close();
    } else {
      const token = await getSessionToken();
      console.log("Connecting with token:", token);
      session.current = new RealtimeSession(agent, {
        model: "gpt-4o-realtime-preview"
      });
      session.current.on("transport_event", (event) => {
        console.log(event);
      });
      session.current.on("history_updated", (history) => {
        setHistory(history);
      });
      session.current.on(
        "tool_approval_requested",
        async (context, agent, approvalRequest) => {
          const response = prompt("Approve or deny the tool call?");
          session.current?.approve(approvalRequest.approvalItem);
        }
      );
      await session.current.connect({
        apiKey: token,
        url: "https://swedencentral.realtimeapi-preview.ai.azure.com/v1/realtimertc",
      });
      setConnected(true);
    }
  }

  return (
    <div className="p-8">
      <h1 className="text-2xl font-bold mb-4">Voice Agent Demo</h1>
      <button
        onClick={onConnect}
        className="bg-black text-white p-2 rounded-md hover:bg-gray-800 cursor-pointer"
      >
        {connected ? "Disconnect" : "Connect"}
      </button>
      <ul>
        {history
          .filter((item) => item.type === "message")
          .map((item) => (
            <li key={item.itemId}>
              {item.role}: {JSON.stringify(item.content)}
            </li>
          ))}
      </ul>
    </div>
  );
}

[seratch]

When initializing RealtimeSession in your code, can you try passing transport option in addition to model? You can instantiate OpenAIRealtimeWebRTC along with a valid Azure base URL for the argument.

[monuminu]

I used this code , i am now getting CORS issue Access to fetch at 'https://swedencentral.realtimeapi-preview.ai.azure.com/v1' from origin 'http://localhost:3000' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. This is Azure OpenAI webRTC reference documentation: https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/realtime-audio-webrtc

import { useRef, useState } from "react";
import {
  RealtimeAgent,
  RealtimeItem,
  RealtimeSession,
  tool,
} from "@openai/agents/realtime";
import { TransportEvent, OpenAIRealtimeWebRTC } from '@openai/agents/realtime';
import { getSessionToken } from "./server/token";
import z from "zod";

const getWeather = tool({
  name: "getWeather",
  description: "Get the weather in a given location",
  parameters: z.object({
    location: z.string(),
  }),
  execute: async ({ location }) => {
    return `The weather in ${location} is sunny`;
  },
});

const weatherAgent = new RealtimeAgent({
  name: "Weather Agent",
  instructions: "Talk with a New York accent",
  handoffDescription: "This agent is an expert in weather",
  tools: [getWeather],
});

const agent = new RealtimeAgent({
  name: "Voice Agent",
  instructions:
    "You are a voice agent that can answer questions and help with tasks.",
  handoffs: [weatherAgent],
});

export default function Home() {
  const session = useRef<RealtimeSession | null>(null);
  const [connected, setConnected] = useState(false);
  const [history, setHistory] = useState<RealtimeItem[]>([]);

  async function onConnect() {
    if (connected) {
      setConnected(false);
      await session.current?.close();
    } else {
      const token = await getSessionToken();
      console.log("Connecting with token:", token);
      
      // Create the transport instance with Azure base URL
      const transport = new OpenAIRealtimeWebRTC({
        baseUrl: "https://swedencentral.realtimeapi-preview.ai.azure.com/v1",
        useInsecureApiKey: true,
      });

      // Initialize RealtimeSession with transport option
      session.current = new RealtimeSession(agent, {
        model: "gpt-4o-realtime-preview",
        transport: transport
      });

      session.current.on("transport_event", (event) => {
        console.log(event);
      });
      session.current.on("history_updated", (history) => {
        setHistory(history);
      });
      session.current.on(
        "tool_approval_requested",
        async (context, agent, approvalRequest) => {
          const response = prompt("Approve or deny the tool call?");
          session.current?.approve(approvalRequest.approvalItem);
        }
      );
      await session.current.connect({
        apiKey: token,
      });
      setConnected(true);
    }
  }

  return (
    <div className="p-8">
      <h1 className="text-2xl font-bold mb-4">Voice Agent Demo</h1>
      <button
        onClick={onConnect}
        className="bg-black text-white p-2 rounded-md hover:bg-gray-800 cursor-pointer"
      >
        {connected ? "Disconnect" : "Connect"}
      </button>
      <ul>
        {history
          .filter((item) => item.type === "message")
          .map((item) => (
            <li key={item.itemId}>
              {item.role}: {JSON.stringify(item.content)}
            </li>
          ))}
      </ul>
    </div>
  );
}```

[monuminu]

@dkundel-openai I see you have added a new feature to make websocket URL configurable. Is this coming in upcoming release?