Configurable Inference timeout during Query time #131551

Samiul-TheSoccerFan · 2025-07-18T17:07:57Z

This PR focuses on introducing user configurable inference timeout settings and use that as timeout during inference calls. Currently, it is hardcoded to 10s and the goal is to make it configurable.

Setup

PUT _inference/sparse_embedding/my-elser-model
{
  "service": "elser",
  "service_settings": {
    "num_allocations": 1,
    "num_threads": 1
  },
  "task_settings": {}
}

PUT my-semantic-index-5
{
  "mappings": {
    "properties": {
      "writer": {
        "type": "semantic_text",
        "inference_id": "my-elser-model"
      },
      "reader": {
        "type": "semantic_text"
      }
    }
  }
}

PUT my-semantic-index-6
{
  "mappings": {
    "properties": {
      "writer": {
        "type": "semantic_text",
        "inference_id": "my-elser-model"
      },
      "reader": {
        "type": "semantic_text"
      }
    }
  }
}

POST my-semantic-index-5/_doc/1
{
  "writer": "Little Red Riding Hood",
  "reader": ["inference test", "another inference test"]
}

POST my-semantic-index-5/_doc/2
{
  "writer": "Another Little Red Riding Hood",
  "reader": ["inference test", "another inference test"]
}

POST my-semantic-index-5/_doc/3
{
   "writer": "Another Little Red Riding Hood",
  "reader": ["inference test", "another inference test"]
}

POST my-semantic-index-6/_doc/1
{
  "writer": "Little Red Riding Hood",
  "reader": ["inference test", "another inference test"]
}

POST my-semantic-index-6/_doc/2
{
  "writer": "Another Little Red Riding Hood",
  "reader": ["inference test", "another inference test"]
}

POST my-semantic-index-6/_doc/3
{
   "writer": "Another Little Red Riding Hood",
  "reader": ["inference test", "another inference test"]
}

GET the default settings:

GET /my-semantic-index-5/_settings

GET /my-semantic-index-5/_settings?include_defaults=true

GET /my-semantic-index-6/_settings

GET /my-semantic-index-6/_settings?include_defaults=true

Update the inference timeout value:

PUT /my-semantic-index-6/_settings
{
  "index": {
    "semantic_text": {
      "inference_timeout": "1s"
    }
  }
}

GET the updated settings:

GET /my-semantic-index-5/_settings

GET /my-semantic-index-5/_settings?include_defaults=true

GET /my-semantic-index-6/_settings

GET /my-semantic-index-6/_settings?include_defaults=true

elasticsearchmachine · 2025-07-18T17:08:24Z

Pinging @elastic/search-eng (Team:SearchOrg)

elasticsearchmachine · 2025-07-18T17:08:24Z

Pinging @elastic/search-relevance (Team:Search - Relevance)

elasticsearchmachine · 2025-07-18T17:08:25Z

Hi @Samiul-TheSoccerFan, I've created a changelog YAML for you.

Samiul-TheSoccerFan · 2025-07-18T17:19:56Z

@Mikep86 Do we need to ping ML team in the PR too?

Mikep86 · 2025-07-18T17:46:12Z

@Samiul-TheSoccerFan Yes, we should ping the ML team since it touches code they own

elasticsearchmachine · 2025-07-18T17:53:26Z

Pinging @elastic/ml-core (Team:ML)

jonathan-buttner

I left a comment, if you could take a look that'd be great!

jonathan-buttner · 2025-07-18T18:02:28Z

...plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/SenderService.java

@@ -73,6 +74,9 @@ public void infer(
        TimeValue timeout,
        ActionListener<InferenceServiceResults> listener
    ) {
+        if (timeout == null) {


If the request is external via the REST api, the timeout field should always be provided and won't be null. If the timeout isn't provided via a query parameter, we default it to 30 seconds. It seems like we're forcing it to be null in SemanticQueryBuilder. We were intentionally trying to avoid timeout being null.

Can you explain the reasoning behind why we want to retrieve the timeout from the settings vs getting it from a query parameter?

If semantic text would like to retrieve it from the settings, why don't we retrieve it from the settings in that code prior to constructing the InferenceAction.Request and pass the value in?

If we did that, we'd only need to change it in SemanticQueryBuilder right?

@jonathan-buttner We actually started with the approach you're suggesting, having this setting at the index level and passing the timeout value from SemanticQueryBuilder. But moved away from it after realizing it introduced more complexity. Switching to a cluster-level setting helped simplify the overall design.

During the query rewrite phase, we do not have access to cluster services in the current context of the SemanticQueryBuilder. However, the inference service does have access to cluster services during inference (specifically in the infer function), which is why the logic was moved there.

The goal is not to override the passed timeout value, but to use the timeout from cluster settings (only if it is null) so that we can support a user-defined inference timeout at query time.

so that we can support a user-defined inference timeout at query time.

If the goal is to provide a user-defined inference timeout at query time wouldn't we want to leverage the timeout parameter on the search request itself? Is there a way to plumb that through if we can't get access to the cluster service in SemanticQueryBuilder?

https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search#operation-search-timeout

@jonathan-buttner : The timeout parameter at the root level of the search request is only intended to control timeouts getting results from shards, potentially allowing for partial search results if desired. Our goal with this setting is to give users control over how long to wait for inference results, especially in cases where a model deployment is required. This allows users to tailor behavior based on their tolerance for latency or urgency of the query.

Gotcha, thanks for explaining that. I'm good with this approach then. There are a few comments I'll add in another review 👍

There's an issue for this that has been open for a while #107077. Pat had a go at it then came to the same conclusion

kderusso

Change looks good to me, but some more tests need to be updated

kderusso · 2025-07-18T18:07:14Z

...ugin/core/src/main/java/org/elasticsearch/xpack/core/ml/search/SparseVectorQueryBuilder.java

@@ -279,7 +278,7 @@ protected QueryBuilder doRewrite(QueryRewriteContext queryRewriteContext) {
            List.of(query),
            TextExpansionConfigUpdate.EMPTY_UPDATE,
            false,
-            InferModelAction.Request.DEFAULT_TIMEOUT_FOR_API
+            null


It looks like this change broke the SparseVectorQueryBuilderTests.

I will probably have to look in depth why this is happening. I remember running the tests locally and found no issues before creating this PR.

jonathan-buttner

I left a few suggestions.

jonathan-buttner · 2025-07-18T20:28:20Z

x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/InferencePlugin.java

@@ -179,6 +180,12 @@ public class InferencePlugin extends Plugin
        Setting.Property.NodeScope,
        Setting.Property.Dynamic
    );
+    public static final Setting<TimeValue> INFERENCE_QUERY_TIMEOUT = Setting.timeSetting(
+        "xpack.inference.query_timeout",
+        TimeValue.timeValueSeconds(TimeUnit.SECONDS.toSeconds(10)),


We've typically used 30 seconds as the default. Is that too long, is that why we're using 10 here for semantic text?

The default timeout for query-time inference is 10 seconds, as defined here: https://github.com/Mikep86/elasticsearch/blob/main/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ml/action/InferModelAction.java

That timeout is what's used by the semantic, sparse_vector, and knn query_vector_builder.

@jonathan-buttner Do we have an issue with differing default timeouts depending on the source of the request (query vs. external REST request)?

Ah ok, 10 seconds sounds good. No I don't believe we have an issue. I believe we followed the 30 second precedent from elsewhere in the stack for requests.

jonathan-buttner · 2025-07-18T20:30:06Z

...plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/SenderService.java

@@ -73,6 +74,9 @@ public void infer(
        TimeValue timeout,


Can you add @Nullable here and add a comment explaining when null will be passed and the motivation (basically just what we discussed in the previous thread).

Mikep86

Partial review, I think we have unhandled edge cases and potentially divergent default values to manage.

Mikep86 · 2025-07-21T13:19:48Z

x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/InferencePlugin.java

@@ -179,6 +180,12 @@ public class InferencePlugin extends Plugin
        Setting.Property.NodeScope,
        Setting.Property.Dynamic
    );
+    public static final Setting<TimeValue> INFERENCE_QUERY_TIMEOUT = Setting.timeSetting(
+        "xpack.inference.query_timeout",
+        TimeValue.timeValueSeconds(TimeUnit.SECONDS.toSeconds(10)),


The default timeout for query-time inference is 10 seconds, as defined here: https://github.com/Mikep86/elasticsearch/blob/main/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ml/action/InferModelAction.java

That timeout is what's used by the semantic, sparse_vector, and knn query_vector_builder.

@jonathan-buttner Do we have an issue with differing default timeouts depending on the source of the request (query vs. external REST request)?

Mikep86 · 2025-07-21T13:26:34Z

...plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/SenderService.java

+        if (timeout == null) {
+            timeout = clusterService.getClusterSettings().get(InferencePlugin.INFERENCE_QUERY_TIMEOUT);
+        }


We only want to apply this timeout if the input type is SEARCH or INTERNAL_SEARCH. Which brings up another edge case: If we allow timeout to be null now, we need to set default timeouts for the other input types as well.

Samiul-TheSoccerFan added 6 commits July 18, 2025 08:55

introducing timeout as cluster settings

7b1f1da

forcing null to be send instead of default value

95066c7

applying timeout in infer level

e60c409

removing unused variable

846f6c2

adding unit tests for cluster timeout values

74dcc03

fix linting issues

5be1e11

Samiul-TheSoccerFan added >enhancement :SearchOrg/Relevance Label for the Search (solution/org) Relevance team Team:Search - Relevance The Search organization Search Relevance team v9.2.0 labels Jul 18, 2025

elasticsearchmachine added the Team:SearchOrg Meta label for the Search Org (Enterprise Search) label Jul 18, 2025

Update docs/changelog/131551.yaml

29d5b7c

Samiul-TheSoccerFan mentioned this pull request Jul 18, 2025

Configurable Inference Timeout #129880

Closed

github-actions bot deployed to docs-preview July 18, 2025 17:09 View deployment

update changelog

d7b8116

github-actions bot deployed to docs-preview July 18, 2025 17:20 View deployment

Samiul-TheSoccerFan added Team:ML Meta label for the ML team :ml Machine learning labels Jul 18, 2025

Samiul-TheSoccerFan requested review from a team July 18, 2025 17:53

jonathan-buttner requested changes Jul 18, 2025

View reviewed changes

kderusso reviewed Jul 18, 2025

View reviewed changes

fix ml core SparseVectorQueryBuilder unit test

bc67010

github-actions bot deployed to docs-preview July 18, 2025 20:14 View deployment

jonathan-buttner requested changes Jul 18, 2025

View reviewed changes

Mikep86 reviewed Jul 21, 2025

View reviewed changes

davidkyle mentioned this pull request Jul 22, 2025

[ML] Associate text_expansion subsearch with parent search request #107077

Closed

Configurable Inference timeout during Query time #131551

Are you sure you want to change the base?

Configurable Inference timeout during Query time #131551

Conversation

Samiul-TheSoccerFan commented Jul 18, 2025

Setup

Uh oh!

elasticsearchmachine commented Jul 18, 2025

Uh oh!

elasticsearchmachine commented Jul 18, 2025

Uh oh!

elasticsearchmachine commented Jul 18, 2025

Uh oh!

Samiul-TheSoccerFan commented Jul 18, 2025

Uh oh!

Mikep86 commented Jul 18, 2025

Uh oh!

elasticsearchmachine commented Jul 18, 2025

Uh oh!

jonathan-buttner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Samiul-TheSoccerFan Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kderusso left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonathan-buttner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonathan-buttner Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mikep86 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Samiul-TheSoccerFan Jul 18, 2025 •

edited

Loading

jonathan-buttner Jul 18, 2025 •

edited

Loading