Add shard write-load to cluster info #131496

nicktindall · 2025-07-18T03:30:21Z

We already use TransportIndicesStatsAction to retrieve size on disk for the disk threshold decider.

This PR just adds the Indexing stats to that request when the write load decider is enabled, and adds the returned shard write loads to the ClusterInfo.

We should be able to feed these into the simulator to properly account for shard movement.

Relates: ES-12419 & ES-12420

elasticsearchmachine · 2025-07-18T03:30:45Z

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

nicktindall · 2025-07-18T03:40:49Z

server/src/main/java/org/elasticsearch/cluster/InternalClusterInfoService.java

@@ -527,8 +538,6 @@ public ClusterInfo getClusterInfo() {
                estimatedHeapUsages.put(nodeId, new EstimatedHeapUsage(nodeId, maxHeapSize.getBytes(), estimatedHeapUsage));
            }
        });
-        final Map<String, NodeUsageStatsForThreadPools> nodeThreadPoolUsageStats = new HashMap<>();
-        nodeThreadPoolUsageStatsPerNode.forEach((nodeId, nodeWriteLoad) -> { nodeThreadPoolUsageStats.put(nodeId, nodeWriteLoad); });


This appeared to be just a copy, which already happens in the ClusterInfo constructor, I assume remnants of something that was since refactored away.

mhl-b

LGTM, nice

DiannaHohensee

I think all my comments are straightforward to address, so I'll go ahead and approve now 👍

DiannaHohensee · 2025-07-18T16:06:40Z

...va/org/elasticsearch/cluster/routing/allocation/allocator/DesiredBalanceReconcilerTests.java

@@ -621,6 +621,7 @@ public void testUnassignedAllocationPredictsDiskUsage() {
            ImmutableOpenMap.of(),
            ImmutableOpenMap.of(),
            ImmutableOpenMap.of(),
+            ImmutableOpenMap.of(),
            ImmutableOpenMap.of()
        );


Looks like I missed a handful of these in the builder conversion 😏 Oops..

DiannaHohensee · 2025-07-18T16:15:28Z

server/src/test/java/org/elasticsearch/cluster/DiskUsageTests.java

-        InternalClusterInfoService.buildShardLevelInfo(stats, shardSizes, shardDataSetSizes, routingToPath, new HashMap<>());
+        InternalClusterInfoService.buildShardLevelInfo(
+            stats,
+            shardWriteLoads,


Below there are checks on the results. Are there some checks that can be added for shardWriteLoads results?

Added in c5ceb92

DiannaHohensee · 2025-07-18T16:20:32Z

server/src/main/java/org/elasticsearch/cluster/ClusterInfo.java

@@ -59,9 +59,10 @@ public class ClusterInfo implements ChunkedToXContent, Writeable {
    final Map<NodeAndPath, ReservedSpace> reservedSpace;


The class comment further up ^ could use an update at this point.

Fixed in cfe5759

DiannaHohensee · 2025-07-18T16:58:36Z

server/src/main/java/org/elasticsearch/cluster/ClusterInfoSimulator.java

@@ -159,7 +159,8 @@ public ClusterInfo getClusterInfo() {
            dataPath,
            Map.of(),
            estimatedHeapUsages,
-            nodeThreadPoolUsageStats
+            nodeThreadPoolUsageStats,
+            allocation.clusterInfo().getShardWriteLoads()


Should this be saved as a private variable initialized in the ClusterInfoSimulator constructor? Similar to estimatedHeapUsages and nodeThreadPoolUsageStats

No we won't use it at this level in the ClusterInfoSimulator and I don't think we'll modify it as part of the simulation either so I think it's fine to pass it through.

DiannaHohensee · 2025-07-18T17:02:45Z

server/src/main/java/org/elasticsearch/cluster/ClusterInfo.java

+    public Map<ShardId, Double> getShardWriteLoads() {
+        return shardWriteLoads;
+    }
+


update the #equals, #hashCode methods below.

I don't like that we add equals and hashCode just for testing, but I updated them in cd3169f

DiannaHohensee · 2025-07-18T17:16:31Z

server/src/internalClusterTest/java/org/elasticsearch/index/shard/IndexShardIT.java

+        );
+
+        try {
+            // Force a ClusterInfo refresh to run collection of the node thread pool usage stats.


copy-paste failed you here :)

Fixed in 7886659

DiannaHohensee · 2025-07-18T17:17:37Z

server/src/internalClusterTest/java/org/elasticsearch/index/shard/IndexShardIT.java

+                }
+            }
+            // And that at least one is greater than zero
+            assertThat(maximumLoadRecorded, greaterThan(0.0));


shouldn't this assert be per index? Since all the indices received writes.

No we insert between 1 and 500 documents in indices with between 1 and 3 shards, so it's possible some shards will not be written to. This is just a sanity check anyhow.

I don't think I follow. Some shards won't receive writes, I agree. But all indices will have writes (to at least one shard), and thus maximumLoadRecorded would have a value per index.

Oh sorry, you're right. For some reason I read that as per shard. Indeed it could be tightened up to be per index. I'll put up a small PR to do that.

server/src/internalClusterTest/java/org/elasticsearch/index/shard/IndexShardIT.java

Co-authored-by: Dianna Hohensee <[email protected]>

…to_cluster_info

Add shard write-load to cluster info

6d1093f

nicktindall requested a review from DiannaHohensee July 18, 2025 03:30

nicktindall added >non-issue :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) labels Jul 18, 2025

nicktindall removed the request for review from DiannaHohensee July 18, 2025 03:30

elasticsearchmachine added Team:Distributed Coordination Meta label for Distributed Coordination team v9.2.0 labels Jul 18, 2025

nicktindall added 3 commits July 18, 2025 13:34

Fix ClusterInfo serialization

4fcea56

Remove redundant copy

db99e30

Fix build break

b24aa23

nicktindall commented Jul 18, 2025

View reviewed changes

Merge branch 'main' into add_shard_write_load_to_cluster_info

62d47f2

nicktindall mentioned this pull request Jul 18, 2025

Simulate impact of shard movement using shard-level write load #131406

Open

nicktindall requested review from DiannaHohensee and mhl-b July 18, 2025 04:06

Note that we're not rendering shard write loads yet

529cba3

mhl-b approved these changes Jul 18, 2025

View reviewed changes

DiannaHohensee approved these changes Jul 18, 2025

View reviewed changes

nicktindall and others added 7 commits July 21, 2025 13:27

Apply suggestion from @DiannaHohensee

1bb0203

Co-authored-by: Dianna Hohensee <[email protected]>

Add missing fields to equals/hashCode

cd3169f

Type

45020c5

Test that shard write load is extracted

c5ceb92

Merge remote-tracking branch 'origin/main' into add_shard_write_load_…

6f2ea34

…to_cluster_info

Fix copy/paste comment

7886659

Add write loads to class comment

cfe5759

nicktindall merged commit b371590 into elastic:main Jul 21, 2025
33 checks passed

nicktindall deleted the add_shard_write_load_to_cluster_info branch July 21, 2025 06:08

		@@ -59,9 +59,10 @@ public class ClusterInfo implements ChunkedToXContent, Writeable {
		final Map<NodeAndPath, ReservedSpace> reservedSpace;

Add shard write-load to cluster info #131496

Add shard write-load to cluster info #131496

Uh oh!

Conversation

nicktindall commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 18, 2025

Uh oh!

nicktindall Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhl-b left a comment

Choose a reason for hiding this comment

Uh oh!

DiannaHohensee left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nicktindall commented Jul 18, 2025 •

edited

Loading

nicktindall Jul 18, 2025 •

edited

Loading