Search before asking
Motivation
Primary-key tables often keep the latest value for each key, but many workloads only need those values within a bounded lifetime. Without table-level row TTL, users must either issue explicit deletes, maintain cleanup jobs, or tolerate unbounded KV state growth.
This is inefficient for common use cases such as session state, device heartbeat state, deduplicated event state, and temporary lookup dimensions. Fluss should provide a native row-level TTL mechanism so stale KV rows can be cleaned up automatically with clear, best-effort semantics.
Solution
Introduce row-level TTL for primary-key tables through table.kv.row.ttl. The table.kv.row.ttl namespace explicitly scopes row-level expiration to KV state and distinguishes it from log or partition retention. Row TTL is disabled by default. When enabled, expired rows are removed asynchronously during RocksDB compaction. Cleanup is best-effort: a row is not guaranteed to become inaccessible immediately after its expiration time.
Fluss owns the versioned KV value layout used by compaction policies. The layout carries a generic value tag, which row TTL uses as its TTL reference timestamp. Writers, readers, recovery paths, and compaction filters interpret values through the same layout contract. This avoids relying on a RocksDB-specific opaque value format and allows different compaction policies and TTL time semantics to share one storage representation.
Time semantics
When only table.kv.row.ttl is configured, TTL uses processing time. The lifetime of a row starts from the timestamp associated with its latest write.
Users may configure table.kv.row.ttl.time-column to use event time. The referenced column may use:
BIGINT, interpreted as epoch milliseconds; orTIMESTAMP_LTZ.
A null event-time value means that the row does not expire. The time column is persisted by stable column identity, so renaming it does not change the TTL association.
Updates, including partial updates, recompute the TTL value tag from the resulting complete row. Recovery preserves the original TTL semantics instead of extending row lifetime from the recovery time.
table.kv.row.ttl.changelog-mode defines the CDC behavior of expiration. Only none is supported in this version, so compaction-based expiration does not emit delete records.
Compatibility
TTL tables use a versioned Fluss KV value layout containing the additional value tag. All encoding and decoding of this layout is centralized in the value-layout abstraction.
Clients using raw KV operations must advertise a protocol version that understands this layout before accessing a TTL table. Older clients are rejected instead of being allowed to decode or write the table incorrectly.
The KV format version and stable identity of an event-time column are persisted in table metadata so that servers, clients, snapshots, and recovery paths derive consistent behavior.
Limitations
- TTL cleanup is asynchronous and best-effort.
- Compaction-based expiration does not produce delete records in CDC.
- Altering row TTL or its time column is not supported in this change.
- Dropping a column referenced by row TTL is not supported.
- Exact row-count pushdown is not available for TTL tables because compaction removes rows outside the normal write path.
Anything else?
No response
Willingness to contribute
Search before asking
Motivation
Primary-key tables often keep the latest value for each key, but many workloads only need those values within a bounded lifetime. Without table-level row TTL, users must either issue explicit deletes, maintain cleanup jobs, or tolerate unbounded KV state growth.
This is inefficient for common use cases such as session state, device heartbeat state, deduplicated event state, and temporary lookup dimensions. Fluss should provide a native row-level TTL mechanism so stale KV rows can be cleaned up automatically with clear, best-effort semantics.
Solution
Introduce row-level TTL for primary-key tables through
table.kv.row.ttl. Thetable.kv.row.ttlnamespace explicitly scopes row-level expiration to KV state and distinguishes it from log or partition retention. Row TTL is disabled by default. When enabled, expired rows are removed asynchronously during RocksDB compaction. Cleanup is best-effort: a row is not guaranteed to become inaccessible immediately after its expiration time.Fluss owns the versioned KV value layout used by compaction policies. The layout carries a generic value tag, which row TTL uses as its TTL reference timestamp. Writers, readers, recovery paths, and compaction filters interpret values through the same layout contract. This avoids relying on a RocksDB-specific opaque value format and allows different compaction policies and TTL time semantics to share one storage representation.
Time semantics
When only
table.kv.row.ttlis configured, TTL uses processing time. The lifetime of a row starts from the timestamp associated with its latest write.Users may configure
table.kv.row.ttl.time-columnto use event time. The referenced column may use:BIGINT, interpreted as epoch milliseconds; orTIMESTAMP_LTZ.A null event-time value means that the row does not expire. The time column is persisted by stable column identity, so renaming it does not change the TTL association.
Updates, including partial updates, recompute the TTL value tag from the resulting complete row. Recovery preserves the original TTL semantics instead of extending row lifetime from the recovery time.
table.kv.row.ttl.changelog-modedefines the CDC behavior of expiration. Onlynoneis supported in this version, so compaction-based expiration does not emit delete records.Compatibility
TTL tables use a versioned Fluss KV value layout containing the additional value tag. All encoding and decoding of this layout is centralized in the value-layout abstraction.
Clients using raw KV operations must advertise a protocol version that understands this layout before accessing a TTL table. Older clients are rejected instead of being allowed to decode or write the table incorrectly.
The KV format version and stable identity of an event-time column are persisted in table metadata so that servers, clients, snapshots, and recovery paths derive consistent behavior.
Limitations
Anything else?
No response
Willingness to contribute