Question (potential issue?) related to datasets caching

[ioana-blue]

I thought I had disabled datasets caching in my code, as follows:

from datasets import set_caching_enabled
...
def main():

    # disable caching in datasets
    set_caching_enabled(False)

However, in my log files I see messages like the following:

04/07/2021 18:34:42 - WARNING - datasets.builder -   Using custom data configuration default-888a87931cbc5877
04/07/2021 18:34:42 - WARNING - datasets.builder -   Reusing dataset csv (xxxx/cache-transformers/datasets/csv/default-888a87931cbc5877/0.0.0/965b6429be0fc05f975b608ce64e1fa941cc8fb4f30629b523d2390f3c0e1a93

Can you please let me know what this reusing dataset csv means? I wouldn't expect any reusing with the datasets caching disabled. Thank you!

[ioana-blue]

An educated guess: does this refer to the fact that depending on the custom column names in the dataset files (csv in this case), there is a dataset loader being created? and this dataset loader - using the "custom data configuration" is used among all jobs running using this particular csv files? (thinking out loud here...)

If this is the case, it may be ok for my use case (have to think about it more), still a bit surprising given that datasets caching is disabled (or so I hope) by the lines I pasted above.

[lhoestq]

Hi ! Currently disabling the caching means that all the dataset transform like map, filter etc. ignore the cache: it doesn't write nor read processed cache files.
However load_dataset reuses datasets that have already been prepared: it does reload prepared dataset files.

Indeed from the documentation:

datasets.set_caching_enabled(boolean: bool)

When applying transforms on a dataset, the data are stored in cache files. The caching mechanism allows to reload an existing cache file if it’s already been computed.
Reloading a dataset is possible since the cache files are named using the dataset fingerprint, which is updated after each transform.
If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets. More precisely, if the caching is disabled:

• cache files are always recreated
• cache files are written to a temporary directory that is deleted when session closes
• cache files are named using a random hash instead of the dataset fingerprint - use datasets.Dataset.save_to_disk() to save a transformed dataset or it will be deleted when session closes
• caching doesn’t affect datasets.load_dataset(). If you want to regenerate a dataset from scratch you should use the download_mode parameter in datasets.load_dataset().

[ioana-blue]

Thank you for the clarification.

This is a bit confusing. On one hand, it says that cache files are always recreated and written to a temporary directory that is removed; on the other hand the last bullet point makes me think that since the default according to the docs for download_mode (Optional datasets.GenerateMode) – select the download/generate mode - Default to REUSE_DATASET_IF_EXISTS => it almost sounds that it could reload prepared dataset files. Where are these files stored? I guess not in the temporary directory that is removed...

I find this type of api design error-prone. When I see as a programmer datasets.set_caching_enabled(False) I expect no reuse of anything in the cache.

[ioana-blue]

It would be nice if the documentation elaborated on all the possible values for download_mode and/or a link to datasets.GenerateMode.
This info here:

    """`Enum` for how to treat pre-existing downloads and data.
    The default mode is `REUSE_DATASET_IF_EXISTS`, which will reuse both
    raw downloads and the prepared dataset if they exist.
    The generations modes:
    |                                    | Downloads | Dataset |
    | -----------------------------------|-----------|---------|
    | `REUSE_DATASET_IF_EXISTS` (default)| Reuse     | Reuse   |
    | `REUSE_CACHE_IF_EXISTS`            | Reuse     | Fresh   |
    | `FORCE_REDOWNLOAD`                 | Fresh     | Fresh   |

[ioana-blue]

I have another question. Assuming that I understood correctly and there is reuse of datasets files when caching is disabled (!), I'm guessing there is a directory that is created based on some information on the dataset file. I'm interested in the situation where I'm loading a (custom) dataset from local disk. What information is used to create the directory/filenames where the files are stored?

I'm concerned about the following scenario: if I have a file, let's say train.csv at path the_path, run once, the dataset is prepared, some models are run, etc. Now let's say there is an issue and I recreate train.csv at the same path the_path. Is there enough information in the temporary name/hash to not reload the old prepared dataset (e.g., timestamp of the file)? Or is it going to reload the old prepared file?

[lhoestq]

Thanks for the feedback, we'll work in improving this aspect of the documentation.

Where are these files stored? I guess not in the temporary directory that is removed...

We're using the Arrow file format to load datasets. Therefore each time you load a dataset, it is prepared as an arrow file on your disk. By default the file is located in the ~/.cache/huggingface/datasets/<dataset_name>/<config_id>/ directory.

What information is used to create the directory/filenames where the files are stored?

The config_id contains a hash that takes into account:

• the dataset loader used and its source code (e.g. the "csv" loader)
• the arguments passed to the loader (e.g. the csv delimiter)
• metadata of the local data files if any (e.g. their timestamps)

I'm concerned about the following scenario: if I have a file, let's say train.csv at path the_path, run once, the dataset is prepared, some models are run, etc. Now let's say there is an issue and I recreate train.csv at the same path the_path. Is there enough information in the temporary name/hash to not reload the old prepared dataset (e.g., timestamp of the file)? Or is it going to reload the old prepared file?

Yes the timestamp of the local csv file is taken into account. If you edit your csv file, the config_id will change and loading the dataset will create a new arrow file.

[ioana-blue]

Thank you for all your clarifications, really helpful!

If you have the bandwidth, please do revisit the api wrt cache disabling. Anywhere in the computer stack (hardware included) where you disable the cache, one assumes there is no caching that happens.

[lhoestq]

That makes total sense indeed !
I think we can do the change

[ioana-blue]

I have another question about caching, this time in the case where FORCE_REDOWNLOAD is used to load the dataset, the datasets cache is one directory as defined by HF_HOME and there are multiple concurrent jobs running in a cluster using the same local dataset (i.e., same local files in the cluster). Does anything in the naming convention and/or file access/locking that you're using prevent race conditions between the concurrent jobs on the caching of the local dataset they all use?

I noticed some errors (can provide more details if helpful) in load_dataset/prepare_split that lead to my question above.

Let me know if my question is clear, I can elaborate more if needed @lhoestq Thank you!

[ioana-blue]

I got another error that convinces me there is a race condition (one of the test files had zero samples at prediction time). I think it comes down to the fact that the config_id above (used in the naming for the cache) has no information on who's touching the data. If I have 2 concurrent jobs, both loading the same dataset and forcing redownload, they may step on each other foot/caching of the dataset.

[lhoestq]

We're using a locking mechanism to prevent two processes from writing at the same time. The locking is based on the filelock module.
Also directories that are being written use a suffix ".incomplete" so that reading is not possible on a dataset being written.

Do you think you could provide a simple code to reproduce the race condition you experienced ?

[ioana-blue]

I can provide details about the code I'm running (it's really-really close to some official samples from the huggingface transformers examples, I can point to the exact sample file, I kept a record of that). I can also describe in which conditions this race occurs (I'm convinced it has to do with forcing the redownloading of the dataset, I've been running hundreds of experiments before and didn't have a problem before I forced the redownload). I also can provide samples of the different stack errors I get and some details about the level of concurrency of jobs I was running. I can also try to imagine how the race manifests (I'm fairly sure that it's a combo of one job cleaning up and another job being in the middle of the run).

However, I have to cleanup all this to make sure I'm no spilling any info I shouldn't be spilling. I'll try to do it by the end of the week, if you think all this is helpful.

For now, I have a workaround. Don't use forcing redownloading. And to be ultra careful (although I don't think this is a problem), I run a series of jobs that will prepare the datasets and I know there is no concurrency wrt the dataset. Once that's done (and I believe even having multiple jobs loading the datasets at the same time doesn't create problems, as long as REUSE_DATASET_IF_EXISTS is the policy for loading the dataset, so the filelock mechanism you're using is working in that scenario), the prepared datasets will be reused, no race possible in any way.

Thanks for all the details you provided, it helped me understand the underlying implementation and coming up with workarounds when I ran into issues.

[piersbatch]

Hi! I have the same challenge with caching, where the .cache folder is required even though it isn't possible for me.

I'd like to run transformers in Snowflake, using Snowpark for Python, this would mean I could provide configurable transformers in real-time for business users without having data leave an environment (for security reasons). With no need for data transfer,n the compute is faster. It is a large use case - is it possible to entirely disable caching in certain scenarios?
@lhoestq ?