Fix #1323: multi-statement parser ignores delimiters inside quoted strings#1360
Open
cobyfrombrooklyn-bot wants to merge 1 commit intogolang-migrate:masterfrom
Open
Fix #1323: multi-statement parser ignores delimiters inside quoted strings#1360cobyfrombrooklyn-bot wants to merge 1 commit intogolang-migrate:masterfrom
cobyfrombrooklyn-bot wants to merge 1 commit intogolang-migrate:masterfrom
Conversation
The splitWithDelimiter function used a naive bytes.Index to find
delimiters, which would incorrectly split on semicolons inside
single-quoted or double-quoted strings (e.g. SETTINGS(format_csv_delimiter = ';')).
Added findDelimiterOutsideQuotes that tracks quote state and only
matches delimiters outside of strings. Also handles SQL-style
escaped quotes ('').
Fixes golang-migrate#1323
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
splitWithDelimiterfunction usedbytes.Indexto find delimiters, which incorrectly splits on semicolons inside single-quoted or double-quoted strings. This breaks migrations containing string literals with semicolons, like ClickHouseSETTINGS(format_csv_delimiter = ';').Fix
Added
findDelimiterOutsideQuotesthat tracks quote state (single and double quotes) and only matches delimiters outside of strings. Also handles SQL-style escaped quotes ('').Test
Added 4 test cases to
TestParse:Without the fix, the parser splits
SELECT 'hello;world'into two statements. With the fix, it correctly treats it as one.Full test suite passes on macOS ARM (Apple Silicon).
Fixes #1323