Consider a default `max_statistics_truncate_length`

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
By default the arrow-rs parquet writer will save the entire actual min and max values for any column that has statistics enabled into the page metadata

For large binary/string columns (think JSON blobs), this means that two (a min and a max) potentially large values will be stored in both the file level metadata as well as in each page header

This can lead to pathalogical cases such as described in

• Files containing binary data with >=8_388_855 bytes per row written with arrow-rs can't be read with pyarrow #7489

It is possible to control the maximum size of the values using

1. WriterPropertiesBuilder::set_statistics_truncate_length
2. WriterPropertiesBuilder::set_column_index_truncate_length

However the values currently defaults to None (unlimited)

I also think it is unlikely that the actual min/max values for large string columns will add significantly better pruning.

Describe the solution you'd like
I propose we set the default statistics truncate length to a non None value to avoid pathalogical cases

Describe alternatives you've considered
I would propose picking a value like 128 that is long enough to capture all primitive data types and
"sort" strings.

We can (and should) also document the default better

Additional context

• related to Files containing binary data with >=8_388_855 bytes per row written with arrow-rs can't be read with pyarrow #7489
• [C++][Python][Parquet] Files with very large data page header can't be read with pyarrow arrow#46404
• Files containing binary data with >=8_388_855 bytes per row written with arro3 can't be read with pyarrow kylebarron/arro3#324

[etseidl]

I'm in favor of truncating by default now that the statistics have the exact flags. I know some who have in the past relied on the statistics using actual values. Looks like parquet-java uses 64 for the column index, but by default leaves binary values in Statistics alone. 128 there seems reasonable to me.

[etseidl]

I just noticed that the docs are wrong for set_column_index_truncate_length. They say the default is None, but it is in fact Some(64). statistics_truncate_length does default to None (no truncation). I'll submit a fix.

[alamb]

I'm in favor of truncating by default now that the statistics have the exact flags.

I agree

128 there seems reasonable to me.

Likewise this seems very reasonable to me

[alamb]

Here is a fun reproducer

use std::io::BufWriter;
use std::sync::Arc;
use arrow::array::{ArrayRef, Int64Array, RecordBatch, StringViewArray};
use parquet::arrow::arrow_reader::ParquetRecordBatchReader;
use parquet::arrow::arrow_writer::ArrowWriterOptions;
use parquet::arrow::ArrowWriter;
use parquet::file::properties::WriterProperties;

fn main() {

    let output= std::fs::File::create("output.parquet").unwrap();
    let mut output = BufWriter::new(output);

    let batch = make_batch('a');
    let props = WriterProperties::builder()
        .set_max_row_group_size(1)
        .build();

    let mut writer = ArrowWriter::try_new(&mut output, batch.schema(), Some(props)).unwrap();
    writer.write(&batch).unwrap();

    for char in ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'] {
        let batch = make_batch(char);
        writer.write(&batch).unwrap();
    }
    writer.close().unwrap();
    drop(output);

    
    println!("Hello, world!");
}

// Makes a batch with long string values for testing purposes.
fn make_batch(val: char) -> RecordBatch {
    let col = Arc::new(StringViewArray::from_iter_values(
        [val.to_string().repeat(100000)]
    )) as ArrayRef;
    RecordBatch::try_from_iter([("col", col)]).unwrap()
}

This results in a 3 MB parquet file of which 2.1MB is the statistics 😱

[etseidl]

Makes me wonder if we should rethink EnabledStatistics. The Parquet spec actually recommends not writing page level statistics if the page indexes are written. Perhaps we could add something like EnabledStatistics::ChunkAndIndex to write chunk level and offset/column indexes but no statistics in the page header.

[alamb]

Makes me wonder if we should rethink EnabledStatistics. The Parquet spec actually recommends not writing page level statistics if the page indexes are written. Perhaps we could add something like EnabledStatistics::ChunkAndIndex to write chunk level and offset/column indexes but no statistics in the page header.

After reviewing #7555 (see #7555 (review)) I %100 agree -- I didn't even realize there were statistics in the page headers.

I will file a ticket about this

Also, while testing I found that the column index truncate length was 64 bytes by which is probably a good size for the main statisics length too. I'll make a PR