Choosing sort on a column may complain about a url link

[hyparam-jbsmith]

Error: parquet sort error 400 Bad Request url not supported

https://huggingface.co/datasets/HuggingFaceFW/fineweb/resolve/main/data/CC-MAIN-2013-20/000_00000.parquet?download=true

[hyparam-jbsmith]

When attempting to sort on the column for the above referenced parquet file, on the language_score column, this issue presents.

[severo]

Did you notice this in the hightable demo?

https://hyparam.github.io/demos/hyparquet/?key=https%3A%2F%2Fhuggingface.co%2Fdatasets%2FHuggingFaceFW%2Ffineweb%2Fresolve%2Fmain%2Fdata%2FCC-MAIN-2013-20%2F000_00000.parquet%3Fdownload%3Dtrue

I could not reproduce this in the demo or in the hyperparam tool:

npm i -g hyperparam
hyp https://huggingface.co/datasets/HuggingFaceFW/fineweb/resolve/main/data/CC-MAIN-2013-20/000_00000.parquet?download=true
# or locally
hyp 000_00000.parquet

Maybe it's due to a temporary issue on the hosting platform.

[severo]

What I got, after several tries, is a 429 error (Too Many Requests). And effectively, nothing happens later: no automatic retry, and clicking again the same column again and again does not trigger refetch.

I guess we should be more robust to some errors in the >2k requests we send.

[severo]

But I didn't get this error: 400 Bad Request url not supported, which should not be retried in any case, I think.

[severo]

cc @platypii what do you think?

[severo]

Not an issue with hightable, transferring to https://github.com/hyparam/hyperparam-cli

[severo]

[optimization?]

Sorting on language_score sends hundreds of range requests of size ~7kB. When doing the same on the text column, the range requests are around 1-2MB, which seems to be a better balance between the number of requests and their size. Note that the group size in this parquet file (https://huggingface.co/datasets/HuggingFaceFW/fineweb/resolve/main/data/CC-MAIN-2013-20/000_00000.parquet) is set to 1,000 rows, we don't control this.

We could be smarter and avoid fetching multiple small ranges (ie: multiple row groups) at once when possible. Drawback: It would add some complexity to the code.

cc @platypii

---

• I opened Fetch several Parquet row groups when appropriate #234

[severo]

Maybe we should let the user pass fetch as a parameter of asyncBufferFromUrl in hyparquet (instead of requestInit). This way, in hyperparam-cli we could depend on https://www.npmjs.com/package/fetch-retry for example, and retry automatically when receiving a 429 Too Many Request error, for example, respecting the Retry-After header.

---

• I opened Handle error 429 and Retry-After header #235

[severo]

I opened hyparam/hyparquet#73 to pass fetch to hyparquet utils.

I think it's useful, even if we have other ideas to implement in hyperparam-cli, without touching hyparquet, to avoid triggering rate limits, such as reducing the number of requests:

1. by grouping small range requests into a bigger one (Fetch several Parquet row groups when appropriate #234)
2. by improving the cache logic

[severo]

re 429 on huggingface specifically, logging in (ie: passing a token with the requests) helps raise the rate-limiting threshold (source).

So, if we identify the source is a Hugging Face URL (move https://github.com/hyparam/space/blob/master/src/lib/huggingfaceSource.ts here?), we can tweak the error message in case the user is not logged in.

---

See #235