Datatset too big

[rosscleung]

I'm having the same problem and (very surprisingly) cannot find the right answer on google nor chatgbt nor in discussions. There was one discussion but it died off without an answer.

From my understanding, this is how a dataset looks like:

class CustomDataSet(Dataset):
    
    def __init__(self, file_path):
        self.file_path = file_path
        self.transforms = transforms
        self.data = self.read_data(self.file_path)

    def read_data(self):
        #Using pandas as an example
        # NOTE: this will cause out-of-memory issue if the file is too big, and this is where I need a solution.
        return pd.read_parquet(self.file_path)
        
    def __len__(self):
        return len(self.data)
    
    def __getitem__(self,idx):  
        return self.data.loc[idx]

The problem:
To instantiate this CustomDataSet(), the function read_data() runs and reads everything into memory, so that __len__() and __getitem__() works. However, what if the file is too big for memory? Dataloader doesn't address the problem. A Dataloader() (at least from my understanding) just sends a batch of indices from the CustomDataSet(), it still relies on the fact that a CustomDataSet() is already instantiated without OOM issues. If the file is too big, my code won't even get to the Dataloader() part.

So, one split one big dataset into several smaller files, so the CustomDataSet() can run its own read_data() and load everything into memory, but that means I need to keep calling the fit() function on multiple datasets in a loop?

[LogWell]

You can try h5py!