Skip to content

[native_iceberg_compat] Spark SQL core-2 "filter pushdown - StringPredicate" failure #1767

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andygrove opened this issue May 21, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@andygrove
Copy link
Member

andygrove commented May 21, 2025

Describe the bug

Test fails:

[info] - filter pushdown - StringPredicate *** FAILED *** (3 seconds, 333 milliseconds)
[info]   0 was not greater than 0 (ParquetFilterSuite.scala:242)

Debug logging shows the filter that should have been pushed down:

testStringPredicate filter=value like 'a%' shouldFilterOut=true

Comet plan shows that the filter is pushed down:

*(1) CometColumnarToRow
+- CometFilter [value#14], (isnotnull(value#14) AND StartsWith(value#14, a))
   +- CometScan parquet [value#14] Batched: true, DataFilters: [isnotnull(value#14), StartsWith(value#14, a)], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/Users/andy/git/apache/apache-spark/target/tmp/spark-e8f77490-5e9..., PartitionFilters: [], PushedFilters: [IsNotNull(value), StringStartsWith(value,a)], ReadSchema: struct<value:string>

However, this fails to filter out partitions during Spark execution.

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant