Question about partitioned tables

[RomanSibirtsev]

Hi, Velox Community!

I want to ask a question about the way Velox handles partitioned table.

I have external Hive table storing its data on HDFS using Parquet (or ORC) format. The data is partitioned by a specific column. And the values of this column are not stored in files, but in the folders names only.

I need to query this table using Velox. I did not find the way to access it through Hive, but only through HDFS directly.

I did not find the way to make Velox recognize this data as partitioned. So Velox just does not see the partitioning column.

Could I kindly ask you to explain the way I should use Velox with partitioned tables, please?

Kind regards,
Roman

[pedroerp]

Hi @RomanSibirtsev. Velox works on the physical layer, so it doesn't have metadata to understand the table structure. Velox only knows about files (through splits).

There is usually an engine on top of Velox that can understand the table structure, create the correct distributed plan, and instantiate the proper Velox tasks with give splits. If you need access to partition columns from Velox, there is a way to add a constant values as part of the split (check HiveConnectorSplit.partitionKeys).