[discussion] dealing with multiple pyarrow versions

[kevinjqliu]

Feature Request / Improvement

I've seen in multiple issues and across multiple PRs where we depend on a specific version of pyarrow.

• Some features are only available after a certain pyarrow version.
• Some bugs might only be resolved after a certain pyarrow version.

The easiest solution is to bump the minimum pyarrow version we depend on. But since this library is often used in diverse python environments, users might want to use many different versions of pyarrow.

For example, we bumped pyarrow minimum version to 17.0.0 in #1621 in order to support filtering pa.RecordBatch directly. The previous solution still works, albeit not efficiently, for pyarrow versions lower than 17.0.0.
I dont have a clear solution but maybe we can decorate certain functions to branch depending on the pyarrow version.

Would love to hear other's thoughts on this!

[dingo4dev]

I've go thought some popular repos which also depends on pyarrow. Using if else condition to check version might be the simple and easy way to branch pyarrow version. The downside is the function may have masses code regarding to different version.
Using decorate dispatch is good for readability but I not sure is it worth to add these which will increase the maintenance process

Dask
https://github.com/dask/dask/blob/3f6051ae242f894af0f43fd9de7ad7131fb3e70d/dask/dataframe/_compat.py#L27-L28

Pandas
https://github.com/pandas-dev/pandas/blob/bc6ad140daf230c470fef92bec598831d4f94a16/pandas/compat/pyarrow.py#L17

[gabeiglio]

Pyarrow versioning is a pain-point internally since it prevents us from integrating to existing services that depend on a specific (old) pyarrow version. I would be very interested to hear solutions for this. I do like the idea of decorators for branching versions impl.

(Funny I say this now and it was me that opened the PR for updating the pyarrow version)

Another thing would be on agreeing how many pyarrow versions would PyIceberg be willing to support at a time, so we dont accumulate as much maintenance burden through the years.

Thanks for bringing this up!

[kevinjqliu]

Thanks for the examples @dingo4dev. i agree we're balancing features availability and project maintainability

Another thing would be on agreeing how many pyarrow versions would PyIceberg be willing to support at a time, so we dont accumulate as much maintenance burden through the years.

we can have 2 separate conversations. one for minimal supported versions, another for how to support different versions from minimal to current

i also do want to split out the pyarrow specific code also so that we can possibly make it more modular