ESQL: Pushdown constructs doing case-insensitive regexes #128393

bpintea · 2025-05-23T14:53:40Z

This introduces an optimization to pushdown to Lucense those language constructs that aim at case-insensitive regular expression matching, used with LIKE and RLIKE operators, such as:

| WHERE TO_LOWER(field) LIKE "abc*"
| WHERE TO_UPPER(field) RLIKE "ABC.*"

These are now pushed as case-insensitive wildcard and regexp respectively queries down to Lucene.

Closes #127479

This introduces an optimization to pushdown to Lucense those language constructs that aim case-insensitive regular expression matching, used with LIKE and RLIKE operators, such as: * `| WHERE TO_LOWER(field) LIKE "abc*"` * `| WHERE TO_UPPER(field) RLIKE `ABC.*` These are now pushed as case-insensitive `regexp` and `wildcard` respectively queries down to Lucene.

elasticsearchmachine · 2025-05-23T14:54:04Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-05-23T14:54:05Z

Hi @bpintea, I've created a changelog YAML for you.

bpintea · 2025-05-23T15:04:54Z

...esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/string/RLike.java

+        if (caseInsensitive() && out.getTransportVersion().before(TransportVersions.ESQL_REGEX_MATCH_WITH_CASE_INSENSITIVITY)) {
+            // The plan has been optimized to run a case-insensitive match, which the remote peer cannot be notified of. Simply avoiding
+            // the serialization of the boolean would result in wrong results.
+            throw new EsqlIllegalArgumentException(
+                NAME + " with case insensitivity is not supported in peer node's version [{}]. Upgrade to version [{}] or newer.",
+                out.getTransportVersion(),
+                TransportVersions.ESQL_REGEX_MATCH_WITH_CASE_INSENSITIVITY
+            );
+        }


Pointing out this fast-failure here // bwc issue.

We haven't been serialising the case-insensitivity of the regexp operators, so far. We now need to.

Not sure we have an alternative to failing the query in this case, without introducing some compensating mechanisms like rerouting queries to old nodes for planning. The planner isn't currently aware of the versions it should plan for.

Introducing new operators (RLIKE2?) would still fail on old nodes; but it would maybe allow the user to opt out of the optimisation, so that the query on mixed-versions won't fail?

Another case for us needing to know if we can enable the optimization based on the minimum node version, or something like it.

We can't make these decisions on the local node only?

In this case we could be safe, see my comment below, but in general I agree that having minimum node version (or node features, or whatever) at planning level would help a lot.

We can't make these decisions on the local node only?

We probably should be able to pull that into a rule that applies locally-only, yes.. Thanks!
That would allows to apply this optimisation, tough the serialisation issue itself will remain.

luigidellaquila

I had a quick look and left a first round of comments, the implementation looks good in general.

My real concern is on the breaking change: TO_LOWER and TO_UPPER are the most used functions in ES|QL, it means that introducing a breaking change here would impact most of our users.
Probably we have an escape hatch though, see my comment below.

luigidellaquila · 2025-05-23T15:31:09Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/where-like.csv-spec

+FROM employees
+| KEEP emp_no, first_name
+| SORT emp_no
+| WHERE TO_UPPER(first_name) LIKE "GEOR*"


Could you please add a couple of negative tests, eg. where TO_UPPER tries to match a lowercase pattern

FROM employees | KEEP emp_no, first_name | SORT emp_no | WHERE TO_UPPER(first_name) LIKE "geor*"

luigidellaquila · 2025-05-23T15:33:31Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/where-like.csv-spec

+10055          |Georgy
+;
+
+likeWithLower


If we decide to accept the breaking change, you'll need at least a new capability, otherwise these tests will fail on mixed clusters.

luigidellaquila · 2025-05-23T15:45:13Z

...gin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java

@@ -78,6 +78,8 @@
 import org.elasticsearch.xpack.esql.expression.function.scalar.multivalue.MvSum;
 import org.elasticsearch.xpack.esql.expression.function.scalar.nulls.Coalesce;
 import org.elasticsearch.xpack.esql.expression.function.scalar.string.Concat;
+import org.elasticsearch.xpack.esql.expression.function.scalar.string.RLike;


I think we'll also need more unit tests, ie. a few more cases in WildcardLikeTests and RLikeTests

luigidellaquila · 2025-05-23T15:51:29Z

...esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/string/RLike.java

    }

    @Override
    public void writeTo(StreamOutput out) throws IOException {
        source().writeTo(out);
        out.writeNamedWriteable(field());
        out.writeString(pattern().asJavaRegex());
+        if (caseInsensitive() && out.getTransportVersion().before(TransportVersions.ESQL_REGEX_MATCH_WITH_CASE_INSENSITIVITY)) {


Can't we just re-add the to_upper/lower() function on the fly if we are on an old transport version?

We'll have to be careful with layouts and NameIDs, but probably it's not impossible.

This would also give us a bwc advantage if we decide to add case insensitive operators to the grammar.

Clever idea: serialize foo rlike "aAa" to --> to_lower(foo) rlike to_lower "aAa".
This is best done in the planner but since we don't have the versioning in place, we can do this locally for rlike and like.

Another suggestion that might be cleaner would be to perform the optimization on the data nodes only, not on the coordinator to avoid the difference in serialization.

costin

Thanks for picking this one up! LGTM to me once the serialization/compatibility stuff gets sorted out.

costin · 2025-05-23T21:02:32Z

...src/main/java/org/elasticsearch/xpack/esql/core/expression/predicate/regex/RLikePattern.java

@@ -21,9 +21,10 @@ public RLikePattern(String regexpPattern) {
    }

    @Override
-    public Automaton createAutomaton() {
+    public Automaton createAutomaton(boolean ignoreCase) {


Expose ignoreCase as a property in StringPattern since it affects both the Automaton and javaRegex. The former can contain the mode but the latter doesn't so we need a way to bubble it.

costin · 2025-05-23T21:03:50Z

.../main/java/org/elasticsearch/xpack/esql/core/expression/predicate/regex/WildcardPattern.java

Same comment as above - make the parameter a class property.

costin · 2025-05-23T21:04:48Z

x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/util/StringUtils.java

+    public static String luceneWildcardToRegExp(String wildcard) {
+        StringBuilder regex = new StringBuilder();
+
+        for (int i = 0, wcLen = wildcard.length(); i < wcLen; i++) {


costin · 2025-05-23T21:11:49Z

...esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/string/RLike.java

    }

    @Override
    public void writeTo(StreamOutput out) throws IOException {
        source().writeTo(out);
        out.writeNamedWriteable(field());
        out.writeString(pattern().asJavaRegex());
+        if (caseInsensitive() && out.getTransportVersion().before(TransportVersions.ESQL_REGEX_MATCH_WITH_CASE_INSENSITIVITY)) {


Clever idea: serialize foo rlike "aAa" to --> to_lower(foo) rlike to_lower "aAa".
This is best done in the planner but since we don't have the versioning in place, we can do this locally for rlike and like.

Another suggestion that might be cleaner would be to perform the optimization on the data nodes only, not on the coordinator to avoid the difference in serialization.

bpintea added >enhancement :Analytics/ES|QL AKA ESQL v9.1.0 labels May 23, 2025

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label May 23, 2025

Update docs/changelog/128393.yaml

3525880

github-actions bot deployed to docs-preview May 23, 2025 14:54 View deployment

[CI] Auto commit changes from spotless

f9b8e79

github-actions bot deployed to docs-preview May 23, 2025 15:02 View deployment

bpintea commented May 23, 2025

View reviewed changes

luigidellaquila self-requested a review May 23, 2025 15:22

luigidellaquila reviewed May 23, 2025

View reviewed changes

costin reviewed May 23, 2025

View reviewed changes

costin self-requested a review May 23, 2025 21:12

costin approved these changes May 23, 2025

View reviewed changes

ESQL: Pushdown constructs doing case-insensitive regexes #128393

Are you sure you want to change the base?

ESQL: Pushdown constructs doing case-insensitive regexes #128393

Conversation

bpintea commented May 23, 2025

Uh oh!

elasticsearchmachine commented May 23, 2025

Uh oh!

elasticsearchmachine commented May 23, 2025

Uh oh!

bpintea May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luigidellaquila left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luigidellaquila May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

costin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bpintea May 23, 2025 •

edited

Loading

luigidellaquila May 23, 2025 •

edited

Loading

costin left a comment •

edited

Loading