KSQL-15026 | Address misc items for ksql - #11060
Open
Vedarth Sharma (VedarthConfluent) wants to merge 3 commits into
Open
KSQL-15026 | Address misc items for ksql#11060Vedarth Sharma (VedarthConfluent) wants to merge 3 commits into
Vedarth Sharma (VedarthConfluent) wants to merge 3 commits into
Conversation
…NERATE_SERIES output sizes to prevent overflow / OOM Both functions previously computed an output-element count via int arithmetic on user-supplied parameters and then preallocated an ArrayList of that size. Two failure modes were reachable: (1) Cube.createAllCombinations: `final int combinations = 1 << columns.size()` overflows once columns.size() >= 31. At size==31 the resulting Integer.MIN_VALUE trips new ArrayList(neg). At size==32 the shift wraps to 1 (silently returning a one-row cube). At size==30 the function attempts to allocate ~1B ArrayList entries and OOMs the server. (2) GenerateSeries: `int diff = end - start` overflows for ranges spanning the int domain; `1 + diff/step` then computes a negative or gigantic size and either throws IllegalArgumentException on new ArrayList(neg) or allocates multiple GB on the heap. The long overload additionally narrowed size to int unsafely, truncating the series silently for very large ranges. Fix: - Cube: cap input at MAX_COLUMNS=20 (~1M output rows, well below the shift-overflow boundary) and throw KsqlFunctionException above the cap with a descriptive message. - GenerateSeries: compute diff in long (use Math.subtractExact on the long overload to catch long-range overflow); compute size in long and validate against MAX_SERIES_SIZE=1_000_000 before narrowing to int. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
🎉 All Contributor License Agreements have been signed. Ready to merge. |
…E_EXPLODE and GENERATE_SERIES size bounds Pins the contract introduced by the size-cap fix in this PR: CubeTest: - shouldRejectInputLargerThanMaxColumns: inputs above MAX_COLUMNS throw KsqlFunctionException (previously: silent OOM at ~30 columns, IllegalArgumentException at 31, silent one-row cube at 32). - shouldAcceptInputAtTheMaxColumns: the cap is inclusive; exactly MAX_COLUMNS still produces 2^MAX_COLUMNS rows. GenerateSeriesTest: - shouldThrowIfIntRangeExceedsMaxSize: int overload rejects sizes above MAX_SERIES_SIZE (previously OOMed on GENERATE_SERIES(0, Integer.MAX_VALUE)). - shouldThrowIfLongRangeExceedsMaxSize: same for the long overload (previously narrowed to int silently). - shouldDetectLongRangeOverflow: Math.subtractExact on the long overload now surfaces overflow as a clear KsqlFunctionException instead of producing a wrapped diff and a wrong-shaped series. - shouldAcceptIntRangeAtMaxSize: cap is inclusive; exactly MAX_SERIES_SIZE elements still succeed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ve tests for Cube and GenerateSeries The previously-added shouldAcceptInputAtTheMaxColumns (Cube) and shouldAcceptIntRangeAtMaxSize (GenerateSeries) tests each materialised ~1M Integer-boxed entries (Cube: 2^MAX_COLUMNS rows of MAX_COLUMNS elements; GenerateSeries: a 1M-element List<Integer>). The combined allocation + GC pressure pushed ksqldb-engine's surefire run past the CI timeout, killing the Test pipeline after ~38 minutes with no clear failure signal. The reject tests (over-max columns / over-max int range / over-max long range / long-range overflow) already pin the cap behaviour and run in milliseconds; happy-path correctness at small inputs is well covered by the pre-existing tests. Drop the at-MAX positive tests and document why in a comment so they don't get reintroduced. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
What behavior do you want to change, why, how does your patch achieve the changes?
Address misc items for ksql
Testing done
Describe the testing strategy. Unit and integration tests are expected for any behavior changes.
Reviewer checklist