refactor(arrow2): daft-csv crate

[universalmind303]

Changes Made

Related Issues

[greptile-apps]

Greptile Summary

This PR refactors the daft-csv crate by migrating from arrow2 to arrow-rs. The changes replace daft-arrow (arrow2-based) with arrow-rs crates (arrow-schema, arrow-array) and switch from arrow2's CSV reader to the standard csv/csv-async crates.

Key improvements:

• Fixes previously reported time unit bugs in deserialize.rs - the code now properly matches on time_unit parameters and uses the correct Arrow primitive types (Time32SecondType, Time32MillisecondType, etc.) instead of hardcoding them
• Removes deprecated arrow2 migration warnings by eliminating #![allow(deprecated)] attributes
• Reorganizes utilities by moving coerce_to_daft_compatible_schema from daft-core/utils/arrow.rs to daft-parquet/src/utils.rs
• Rewrites test helper check_equal_local_arrow_rs with a more comprehensive reference implementation using the sync csv crate

Notable changes:

• New local_read_rows and async_read_rows functions replace arrow2's reader implementations
• Error handling updated from ArrowError to External errors in schema inference
• Field conversions now use arrow-rs types throughout the pipeline

Confidence Score: 5/5

• This PR is safe to merge - it's a well-executed refactor that fixes existing bugs and modernizes the codebase
• High confidence because this is a systematic migration with clear improvements: fixes previously identified time unit bugs, removes deprecated code, maintains test coverage, and follows a consistent pattern throughout. The changes are well-structured with proper error handling and type conversions.
• No files require special attention - all changes follow consistent migration patterns

Important Files Changed

Filename	Overview
src/daft-decoding/src/deserialize.rs	Migrated from arrow2 to arrow-rs, fixing previously reported time unit bugs by properly matching time_unit parameter with correct Arrow types
src/daft-decoding/Cargo.toml	Updated dependencies from daft-arrow (arrow2) to arrow-array and csv crates
src/daft-csv/Cargo.toml	Updated dependencies, added arrow-schema, csv, and daft-schema, removed daft-arrow
src/daft-csv/src/local.rs	Migrated from arrow2 CSV reader to csv crate, added local_read_rows function, updated field conversions
src/daft-csv/src/read.rs	Major refactor of CSV reading logic and test helper check_equal_local_arrow_rs with new reference implementation
src/daft-csv/src/metadata.rs	Migrated schema inference to arrow-schema, updated error handling from ArrowError to External
src/daft-parquet/src/utils.rs	Moved coerce_to_daft_compatible_schema utility from daft-core to daft-parquet

_{Last reviewed commit: 869da3c}

[greptile-apps]

_{22 files reviewed, 4 comments}

_{Edit Code Review Agent Settings | Greptile}

[universalmind303]

@greptileai

[codecov]

Codecov Report

❌ Patch coverage is 76.44110% with 94 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.32%. Comparing base (4b450a6) to head (cba377a).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
src/daft-decoding/src/deserialize.rs	51.78%	81 Missing ⚠️
src/daft-csv/src/read.rs	93.37%	10 Missing ⚠️
src/daft-csv/src/local.rs	94.11%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6298      +/-   ##
==========================================
- Coverage   74.37%   74.32%   -0.06%     
==========================================
  Files        1006     1006              
  Lines      134209   134292      +83     
==========================================
- Hits        99823    99810      -13     
- Misses      34386    34482      +96     

Files with missing lines	Coverage Δ
src/daft-csv/src/lib.rs	40.00% <ø> (ø)
src/daft-csv/src/local/pool.rs	81.63% <100.00%> (ø)
src/daft-csv/src/metadata.rs	90.37% <100.00%> (+0.17%)	⬆️
src/daft-csv/src/schema.rs	100.00% <100.00%> (ø)
src/daft-parquet/src/lib.rs	77.77% <100.00%> (ø)
src/daft-parquet/src/utils.rs	81.52% <ø> (ø)
src/daft-scan/src/hive.rs	98.23% <100.00%> (-0.03%)	⬇️
src/daft-csv/src/local.rs	86.39% <94.11%> (+0.25%)	⬆️
src/daft-csv/src/read.rs	94.12% <93.37%> (-0.53%)	⬇️
src/daft-decoding/src/deserialize.rs	57.30% <51.78%> (-7.15%)	⬇️

... and 12 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[universalmind303]

@greptileai

[desmondcheongzx]

Nice