Loading Data From S3 Path in Sagemaker

[mahesh1amour]

In Sagemaker Im tring to load the data set from S3 path as follows

`train_path = 's3://xxxxxxxxxx/xxxxxxxxxx/train.csv'
valid_path = 's3://xxxxxxxxxx/xxxxxxxxxx/validation.csv'
test_path = 's3://xxxxxxxxxx/xxxxxxxxxx/test.csv'

data_files = {}
data_files["train"] = train_path
data_files["validation"] = valid_path
data_files["test"] = test_path
extension = train_path.split(".")[-1]
datasets = load_dataset(extension, data_files=data_files, s3_enabled=True)
print(datasets)`

I getting an error of

algo-1-7plil_1 | File "main.py", line 21, in <module> algo-1-7plil_1 | datasets = load_dataset(extension, data_files=data_files) algo-1-7plil_1 | File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 603, in load_dataset algo-1-7plil_1 | **config_kwargs, algo-1-7plil_1 | File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 155, in __init__ algo-1-7plil_1 | **config_kwargs, algo-1-7plil_1 | File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 305, in _create_builder_config algo-1-7plil_1 | m.update(str(os.path.getmtime(data_file))) algo-1-7plil_1 | File "/opt/conda/lib/python3.6/genericpath.py", line 55, in getmtime algo-1-7plil_1 | return os.stat(filename).st_mtime algo-1-7plil_1 | FileNotFoundError: [Errno 2] No such file or directory: 's3://lsmv-sagemaker/pubmedbert/test.csv

But when im trying with pandas , it is able to load from S3

Does the datasets library support S3 path to load

[julien-c]

This would be a neat feature

[mahesh1amour]

neat feature

I dint get these clearly, can you please elaborate like how to work on these

[thomwolf]

It could maybe work almost out of the box just by using cached_path in the text/csv/json scripts, no?

[mahesh1amour]

Thanks thomwolf and julien-c

I'm still confusion on what you guys said,

I have solved the problem as follows:

1. read the csv file using pandas from s3
2. Convert to dictionary key as column name and values as list column data
3. convert it to Dataset using
   from datasets import Dataset
train_dataset = Dataset.from_dict(train_dict)

[thomwolf]

We were brainstorming around your use-case.

Let's keep the issue open for now, I think this is an interesting question to think about.

[mahesh1amour]

We were brainstorming around your use-case.

Let's keep the issue open for now, I think this is an interesting question to think about.

Sure thomwolf, Thanks for your concern

[lhoestq]

I agree it would be cool to have that feature. Also that's good to know that pandas supports this.
For the moment I'd suggest to first download the files locally as thom suggested and then load the dataset by providing paths to the local files

[jman1973]

Don't get

[dorlavie]

Any updates on this issue?
I face a similar issue. I have many parquet files in S3 and I would like to train on them.
To be honest I even face issues with only getting the last layer embedding out of them.

[mahesh1amour]

Hi dorlavie,
You can find one solution that i have mentioned above, that can help you.
And there is one more solution also which is downloading files locally

[dorlavie]

Hi dorlavie,
You can find one solution that i have mentioned above, that can help you.
And there is one more solution also which is downloading files locally

mahesh1amour, thanks for the fast reply

Unfortunately, in my case I can not read with pandas. The dataset is too big (50GB).
In addition, due to security concerns I am not allowed to save the data locally