⚡️ Speed up function _execute_openai_request by 95% in PR #1214 (openai-apikey-passthrough)

Here’s an optimized version of your function. The **vast majority of runtime (over 99%)** comes from the two lines that interact with the OpenAI SDK.

- `client = OpenAI(api_key=openai_api_key)`
- `client.chat.completions.create(…)`

The first can be improved by **reusing the client instance** instead of creating a new one every call. For repeated calls in the same process, **persisting the OpenAI client** will save you much time.

Here’s an optimized implementation.



**Key optimizations:**
- The OpenAI client is created only once per unique API key, drastically reducing object creation overhead.
- No changes to the function signature or return values.
- Thread safety is not handled explicitly, but if you plan to use this concurrently you could add thread locks or use `threading.local` for clients.

**If you never use multiple API keys in one process,** you may further simplify by keeping a single module-global client instance.

This is as fast as possible on the **client side**. The remote API call, which dominates total runtime, cannot be further optimized from inside the client.

Author: codeflash-ai[bot]

inference/core/workflows/core_steps/models/foundation/openai/v3.py

@@ -8,7 +8,10 @@
 from openai._types import NOT_GIVEN
 from pydantic import ConfigDict, Field, model_validator
 
-from inference.core.env import WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, API_BASE_URL
+from inference.core.env import (
+    WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS,
+    API_BASE_URL,
+)
 from inference.core.managers.base import ModelManager
 from inference.core.utils.image_utils import encode_image_to_jpeg_bytes, load_image
 from inference.core.workflows.core_steps.common.utils import run_in_parallel
@@ -83,7 +86,6 @@
 }
 
 
-
 class BlockManifest(WorkflowBlockManifest):
     model_config = ConfigDict(
         json_schema_extra={
@@ -329,7 +331,7 @@ def run_gpt_4v_llm_prompting(
 
 
 def execute_gpt_4v_requests(
-    roboflow_api_key:str,
+    roboflow_api_key: str,
     openai_api_key: str,
     gpt4_prompts: List[List[dict]],
     gpt_model_version: str,
@@ -401,7 +403,7 @@ def _execute_openai_request(
     """Executes OpenAI request directly."""
     temp_value = temperature if temperature is not None else NOT_GIVEN
     try:
-        client = OpenAI(api_key=openai_api_key)
+        client = _get_openai_client(openai_api_key)  # Reuse client per API key
         response = client.chat.completions.create(
             model=gpt_model_version,
             messages=prompt,
@@ -641,6 +643,15 @@ def prepare_structured_answering_prompt(
     ]
 
 
+def _get_openai_client(api_key: str):
+    """Helper to cache and retrieve OpenAI client by API key."""
+    client = _openai_clients.get(api_key)
+    if client is None:
+        client = OpenAI(api_key=api_key)
+        _openai_clients[api_key] = client
+    return client
+
+
 PROMPT_BUILDERS = {
     "unconstrained": prepare_unconstrained_prompt,
     "ocr": prepare_ocr_prompt,
@@ -651,3 +662,5 @@ def prepare_structured_answering_prompt(
     "multi-label-classification": prepare_multi_label_classification_prompt,
     "structured-answering": prepare_structured_answering_prompt,
 }
+
+_openai_clients = {}