You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The native_datafusion Parquet scan does not support bucketed scan, and fails most of the tests in Spark's BucketedReadSuite without a fallback. With a bucketed scan, some partitions end up without a file to read so their PartitionedFile is empty.
Describe the potential solution
I don't think DataSourceExec will take no file at construction. We might need to replace that node with a different no-op node that generates an empty data set with the correct schema in the case of a bucketed scan when a partition has no corresponding file.
Additional context
No response
The text was updated successfully, but these errors were encountered:
Interestingly, native_datafusion currently passes the "bucketed table" Comet test. I suspect the Spark SQL tests are doing some bucket pruning, which is where we end up with empty file paths for some of the partitions.
What is the problem the feature request solves?
The native_datafusion Parquet scan does not support bucketed scan, and fails most of the tests in Spark's BucketedReadSuite without a fallback. With a bucketed scan, some partitions end up without a file to read so their
PartitionedFile
is empty.Describe the potential solution
I don't think DataSourceExec will take no file at construction. We might need to replace that node with a different no-op node that generates an empty data set with the correct schema in the case of a bucketed scan when a partition has no corresponding file.
Additional context
No response
The text was updated successfully, but these errors were encountered: