Skip to content

[server] Add resource-based KV leader replica capacity control#3614

Open
swuferhong wants to merge 1 commit into
apache:mainfrom
swuferhong:kv-bucket-limit
Open

[server] Add resource-based KV leader replica capacity control#3614
swuferhong wants to merge 1 commit into
apache:mainfrom
swuferhong:kv-bucket-limit

Conversation

@swuferhong

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #3613

Add tabletServer resource reporting and a coordinator-side KvLeaderReplicaCapacityManager to protect clusters from creating too many KV leader replicas.

TabletServers now report optional CPU and memory information during registration. The coordinator derives kvLeaderReplicaCapacity from live tabletServer memory and kv.leader-replica.memory-reserved, tracks the current KV leader replica count in memory, rebuilds it on coordinator leadership startup, and checks/reserves capacity when creating KV tables or partitions.

The capacity limit is disabled unless more than half of live tablet servers report memory. Unknown-memory TabletServersuse the average known memory when the limit is enabled.

Also add dynamic config support, metrics, RPC error mapping, serde compatibility, auto-partition accounting, and tests for resource probing, capacity calculation, manual create/drop, and auto partition create/drop.

Brief change log

Tests

API and Format

Documentation

DropDatabaseResponse response = new DropDatabaseResponse();
metadataManager.dropDatabase(
request.getDatabaseName(), request.isIgnoreIfNotExists(), request.isCascade());
if (request.isCascade()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the change! One concern about rebuildCurrentCount(metadataManager) in dropDatabase: it rescans all databases/tables, but admin operations run concurrently on the coordinator. If another request drops a database mid-scan, listTables() throws DatabaseNotExistException, which propagates out and fails this dropDatabase request — even with ignoreIfNotExists=true.

This is what fails the Rust IT test_error_database_already_exist in CI: its drop_database(..., cascade=true) gets a DatabaseNotExist error for a different database (test_create_database) that a parallel test was dropping at the same time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[server] Add resource-based KV leader replica capacity control

2 participants