[Lens] Enable ESQL multi-terms charts #244743

markov00 · 2025-12-01T11:32:51Z

Summary

This PR enables the possibility to configure ESQL charts with a multi-field breakdown.

The code changes are the following:

I've renamed and retyped the splitAccessor?: string visualization state parameter of XY charts into splitAccesors?: string[]. This allows adding multiple separate column/dimension into that parameter, that was already available at the expression level to describe a breakdown formed by multiple data columns.
A runtime migration was added before the color mapping runtime migration to support correctly existing charts
The color mapping was one of the main problem in the refactoring due to the need of specifying more than one column to describe a color. This was done by reusing the same structure of the multi-terms class used for multi-terms aggregation. This class has already everything in place to contain 2 different values and also 2 different formatting options.
The changes are applied only for ESQL, maintaining unchanged the existing behaviour with DSL. (even if enabling multi-fields would work too in DSL world, for this PR I tried to limit the logic to just the ESQL part.
During one review it was discovered that multi-fields title aggregation where contatenated in a different way than in the color mapping/multi-term. I changed the concatenation to use the same > simbol.
A strange behaviour was discovered when testing the breakdown dimension reordering: in the Visualization state the logic was a bit off and caused new duplicated dimensions to popup: I've added a specific method that support reordering the dimension in the visualization state to correctly fix that issue.

Fix #235743

Part of #236682
What is currently missing, and will be added in a subsequent PR:

A way to warn the user when not every field returned by the query is used in the chart configuration.

stratoula

Marco I tested it a lot, especially the suggestions logic and I don't see any regressions. It actually works really well!

stratoula · 2025-12-08T08:10:19Z

src/platform/packages/shared/chart-expressions-common/color_categories.test.ts

Even the filtering works!

elasticmachine · 2025-12-17T08:59:59Z

Pinging @elastic/kibana-visualizations (Team:Visualizations)

elasticmachine · 2025-12-17T09:00:28Z

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

jughosta

Data Discovery changes LGTM

elasticmachine · 2025-12-17T11:54:01Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: dd93a7f

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #82 / cases security and spaces enabled: trial Common analytics indexes backfill task should backfill the cases attachments index

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`lens`	1662	1665	+3

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/lens-common`	1347	1349	+2
`lens`	600	602	+2
total			+4

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`dashboard`	721.4KB	721.4KB	+21.0B
`datasetQuality`	588.3KB	588.3KB	+6.0B
`discover`	1.3MB	1.3MB	+85.0B
`eventAnnotationListing`	206.8KB	206.8KB	+18.0B
`exploratoryView`	138.8KB	138.8KB	+3.0B
`expressionPartitionVis`	37.5KB	37.5KB	-78.0B
`expressionTagcloud`	17.2KB	17.2KB	+11.0B
`expressionXY`	98.5KB	98.9KB	+414.0B
`infra`	1.1MB	1.1MB	+21.0B
`lens`	1.9MB	1.9MB	+3.6KB
`observability`	1.7MB	1.7MB	+21.0B
`observabilityAIAssistantApp`	267.9KB	267.9KB	+21.0B
`onechat`	519.8KB	519.9KB	+21.0B
`securitySolution`	10.8MB	10.8MB	+21.0B
`unifiedDocViewer`	370.6KB	370.6KB	+21.0B
`visTypeTimeseries`	440.7KB	440.8KB	+14.0B
`visTypeXy`	43.7KB	43.7KB	+12.0B
total			+4.2KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`expressionPartitionVis`	26.7KB	27.0KB	+275.0B
`expressionTagcloud`	11.8KB	12.1KB	+275.0B
`expressionXY`	41.5KB	41.8KB	+277.0B
`lens`	64.5KB	64.6KB	+39.0B
total			+866.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/lens-common`	1491	1493	+2
`lens`	703	705	+2
total			+4

History

💚 Build #374142 succeeded 57d61f3
💔 Build #373201 failed f573b09
💔 Build #372759 failed c93bd4f
💔 Build #372606 failed e25e357
💔 Build #372129 failed 7312fb5
💔 Build #371865 failed 1f18732

walterra · 2025-12-18T10:54:26Z

...orm/packages/shared/kbn-coloring/src/shared_components/color_mapping/color/color_handling.ts

+            // remove categories one maching an assignment
            return !assignmentMatcher.hasMatch(category);
          })
+          // setting the Map keys as the strinified version of the rawValue


nit: typo stringified

walterra · 2025-12-18T10:59:38Z

src/platform/plugins/shared/chart_expressions/expression_xy/public/helpers/data_layers.tsx

    }
-    return `${splitValues.join(' - ')}${yAccessorTitle ? ' - ' + yAccessorTitle : ''}`;
+    return `${splitValues.join(MULTI_FIELD_KEY_SEPARATOR)}${
+      yAccessorTitle ? ' - ' + yAccessorTitle : ''


should this ' - ' also be MULTI_FIELD_KEY_SEPARATOR?

No that's a different separator, this separates the breakdown categories (called splitValues and concatenated with the from the MULTI_FIELD_KEY_SEPARATOR) from the metric name

baileycash-elastic

actionable-obs changes lgtm

PhilippeOberti

Code review only, LGTM for the @elastic/security-threat-hunting-investigations team

walterra · 2025-12-29T13:35:18Z

...k/platform/plugins/shared/lens/common/content_management/v1/transforms/split_accessors/xy.ts

+  layers: Array<DeprecatedSplitAccessorLayer | XYLayerConfig>;
+}
+
+export function convertToSplitAccessorsFn(state: DeprecatedSplitAccessorState | XYState): XYState {


looks like this is an exact duplicate of the one in split_accessor.ts - should we move to a shared utility?

walterra · 2025-12-29T13:40:15Z

src/platform/packages/shared/chart-expressions-common/color_categories.ts

  legacyMode: boolean = false // stringifies raw values
 ): SerializedValue[] {
-  if (!accessor) return [];
+  if (!accessors) return [];


With the default being now an array, should we consider it like this: (!accessors || accessors.length === 0) return [];

walterra · 2025-12-29T13:46:37Z

x-pack/platform/plugins/shared/lens/public/visualizations/xy/color_assignment.ts

+
      const splits =
-        !column || !data.tables[layer.layerId]
+        !columns || !data.tables[layer.layerId]


We could add a check for array length here too: !columns || columns.length === 0 | !data.tables[layer.layerId]

walterra

Did some local testing, LGTM! Tested:

Used sample logs/flights
In query mode, was able to add additional breakdown fields
In form mode, only a single breakdown field can be used
Reordering breakdown fields works
ES|QL with multiple BYs will add multiple fields to breakdowns

kpatticha · 2026-01-07T11:47:19Z

Hi @markov00

I'm mainly testing the UI in Discover, and it works really nicely overall.

I do have a few questions, though.

In the code, I don't see any explicit limitation on the number of dimensions. However, when using ES|QL, the charts seem to return a maximum of three dimensions. (am I missing something?)

Screen.Recording.2026-01-07.at.1.38.14.PM.mov

Additionally, with the following query:

FROM metrics-*
| STATS AVG(metrics.system.cpu.load_average.`15m`)
  BY BUCKET(@timestamp, 100, ?_tstart, ?_tend),
     host.ip,
     cloud.region,
     host.id

Screen.Recording.2026-01-07.at.1.39.27.PM.mov

The legend appears in the order cloud.region → host.id → host.ip, as shown in recording above.
How is the legend order determined?

markov00 added 6 commits November 27, 2025 18:07

Merge branch 'main' into 2025_11_26-multi-terms-final-branch

86b8302

Initial implementation

8703da5

fix tests

2ae0649

Merge branch 'main' into 2025_11_26-multi-terms-final-branch

921a020

fix type checks

060fa7b

Merge branch 'main' into 2025_11_26-multi-terms-final-branch

eb33f5b

miguel-sanchez-elastic mentioned this pull request Dec 3, 2025

[Metrics][Discover] Support multi-dimension breakdown #239346

Open

14 tasks

markov00 and others added 10 commits December 3, 2025 14:19

Fix sorting order

f2d4d93

Merge branch 'main' into 2025_11_26-multi-terms-final-branch

f440544

fix suggestions

f612f02

support mult breakdown only in textbased layers

da06dd1

Merge branch 'main' into 2025_11_26-multi-terms-final-branch

d6640fd

Changes from node scripts/eslint_all_files --no-cache --fix

8fe9a28

Fix multi-terms name separator

5fb4a63

temp reorder fix

199cbe6

add runtime migrations

2ada266

Merge branch 'main' into 2025_11_26-multi-terms-final-branch

7b3b514

stratoula reviewed Dec 8, 2025

View reviewed changes

src/platform/packages/shared/chart-expressions-common/color_categories.test.ts

Copy link

Contributor

stratoula Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even the filtering works!

markov00 and others added 12 commits December 9, 2025 12:54

fix runtime conversion

05ed486

Merge branch 'main' into 2025_11_26-multi-terms-final-branch

bac4713

wip

2ef1f8b

update config_builder tests

1f8a3e3

Merge branch 'main' into 2025_11_26-multi-terms-final-branch

7daf93d

[Scout] Automated config manifest updates

7eeae57

fix i18n and test

4b99f48

Merge branch 'main' into 2025_11_26-multi-terms-final-branch

1f18732

update tests

7312fb5

fix different legend entries concatenation

097ae09

Fix on drop

e25e357

remove console log

c93bd4f

markov00 added 2 commits December 16, 2025 19:11

fix test

b65c28e

Cleanup PR code

57d61f3

markov00 marked this pull request as ready for review December 17, 2025 08:58

markov00 requested review from a team as code owners December 17, 2025 08:58

markov00 requested a review from CAWilson94 December 17, 2025 08:58

markov00 added release_note:enhancement Team:Visualizations Team label for Lens, elastic-charts, Graph, legacy editors (TSVB, Visualize, Timelion) t// Feature:Lens backport:skip This PR does not require backporting labels Dec 17, 2025

Merge branch 'main' into 2025_11_26-multi-terms-final-branch

dd93a7f

botelastic bot added the Team:obs-ux-management label Dec 17, 2025

jughosta approved these changes Dec 17, 2025

View reviewed changes

walterra reviewed Dec 18, 2025

View reviewed changes

kpatticha self-requested a review December 19, 2025 14:37

baileycash-elastic approved these changes Dec 19, 2025

View reviewed changes

PhilippeOberti approved these changes Dec 26, 2025

View reviewed changes

walterra reviewed Dec 29, 2025

View reviewed changes

walterra approved these changes Dec 29, 2025

View reviewed changes

Merge branch 'main' into 2025_11_26-multi-terms-final-branch

b8cc9f2

Merge branch 'main' into 2025_11_26-multi-terms-final-branch

2dffad1

[Lens] Enable ESQL multi-terms charts #244743

Are you sure you want to change the base?

[Lens] Enable ESQL multi-terms charts #244743

Conversation

markov00 commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

stratoula left a comment

Choose a reason for hiding this comment

Uh oh!

stratoula Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Dec 17, 2025

Uh oh!

elasticmachine commented Dec 17, 2025

Uh oh!

jughosta left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Dec 17, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

History

Uh oh!

walterra Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

walterra Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

markov00 Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

baileycash-elastic left a comment

Choose a reason for hiding this comment

Uh oh!

PhilippeOberti left a comment

Choose a reason for hiding this comment

Uh oh!

walterra Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

walterra Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

walterra Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

walterra left a comment

Choose a reason for hiding this comment

Uh oh!

kpatticha commented Jan 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

markov00 commented Dec 1, 2025 •

edited

Loading