Chapter 4 - CPU Workers mac M1

[AnHumanLearner]

Hello, I have an issue when I transform my data into a data loader. It works only for num_workers=0. I'm here in the video : https://youtu.be/V_xro1bcAuA?feature=shared&t=79170

And here's the error:

There are 2 directories and 1 images in data/pizza_steak_sushi
There are 3 directories and 0 images in data/pizza_steak_sushi/test
There are 0 directories and 19 images in data/pizza_steak_sushi/test/steak
There are 0 directories and 31 images in data/pizza_steak_sushi/test/sushi
There are 0 directories and 25 images in data/pizza_steak_sushi/test/pizza
There are 3 directories and 1 images in data/pizza_steak_sushi/train
There are 0 directories and 75 images in data/pizza_steak_sushi/train/steak
There are 0 directories and 72 images in data/pizza_steak_sushi/train/sushi
There are 0 directories and 78 images in data/pizza_steak_sushi/train/pizza
data/pizza_steak_sushi directory already exists… skipping download
Downloading pizza, steak, sushi data…
Unzipping pizza, steak and sushi data…
'mps'
'2.7.1'

data/pizza_steak_sushi/test/pizza/2236338.jpg
pizza
Random image path : data/pizza_steak_sushi/test/pizza/2236338.jpg
Image class : pizza
Image height : 512
Image width : 512
(PosixPath('data/pizza_steak_sushi/train'),
 PosixPath('data/pizza_steak_sushi/test'))









(np.float64(-0.5), np.float64(511.5), np.float64(511.5), np.float64(-0.5))
torch.Size([3, 64, 64])
[PosixPath('data/pizza_steak_sushi/test/steak/296375.jpg'),
 PosixPath('data/pizza_steak_sushi/test/steak/673127.jpg'),
 PosixPath('data/pizza_steak_sushi/test/steak/894825.jpg'),
 PosixPath('data/pizza_steak_sushi/test/steak/1868005.jpg'),
 PosixPath('data/pizza_steak_sushi/test/steak/27415.jpg'),
 PosixPath('data/pizza_steak_sushi/test/steak/1016217.jpg'),
 PosixPath('data/pizza_steak_sushi/test/steak/1285886.jpg'),
 PosixPath('data/pizza_steak_sushi/test/steak/502076.jpg'),
 PosixPath('data/pizza_steak_sushi/test/steak/3424937.jpg'),
 PosixPath('data/pizza_steak_sushi/test/steak/690177.jpg'),
 PosixPath('data/pizza_steak_sushi/test/steak/354513.jpg'),
 PosixPath('data/pizza_steak_sushi/test/steak/2117351.jpg'),
 PosixPath('data/pizza_steak_sushi/test/steak/100274.jpg'),
 PosixPath('data/pizza_steak_sushi/test/steak/2144308.jpg'),
 PosixPath('data/pizza_steak_sushi/test/steak/966174.jpg'),
 PosixPath('data/pizza_steak_sushi/test/steak/1882831.jpg'),
 PosixPath('data/pizza_steak_sushi/test/steak/3873283.jpg'),
 PosixPath('data/pizza_steak_sushi/test/steak/1302563.jpg'),
 PosixPath('data/pizza_steak_sushi/test/steak/1627703.jpg'),
 PosixPath('data/pizza_steak_sushi/test/sushi/858157.jpg'),
 PosixPath('data/pizza_steak_sushi/test/sushi/988559.jpg'),
 PosixPath('data/pizza_steak_sushi/test/sushi/479711.jpg'),
 PosixPath('data/pizza_steak_sushi/test/sushi/1987407.jpg'),
 PosixPath('data/pizza_steak_sushi/test/sushi/1230335.jpg'),
 PosixPath('data/pizza_steak_sushi/test/sushi/1245193.jpg'),
...
 PosixPath('data/pizza_steak_sushi/train/pizza/68684.jpg'),
 PosixPath('data/pizza_steak_sushi/train/pizza/2451169.jpg'),
 PosixPath('data/pizza_steak_sushi/train/pizza/2285942.jpg'),
 PosixPath('data/pizza_steak_sushi/train/pizza/12301.jpg'),
 PosixPath('data/pizza_steak_sushi/train/pizza/1947572.jpg')]
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...
(Dataset ImageFolder
     Number of datapoints: 225
     Root location: data/pizza_steak_sushi/train
     StandardTransform
 Transform: Compose(
                Resize(size=(64, 64), interpolation=bilinear, max_size=None, antialias=True)
                RandomHorizontalFlip(p=0.5)
                ToTensor()
            ),
 Dataset ImageFolder
     Number of datapoints: 75
     Root location: data/pizza_steak_sushi/test
     StandardTransform
 Transform: Compose(
                Resize(size=(64, 64), interpolation=bilinear, max_size=None, antialias=True)
                RandomHorizontalFlip(p=0.5)
                ToTensor()
            ))
(PosixPath('data/pizza_steak_sushi/train'),
 PosixPath('data/pizza_steak_sushi/test'))
['pizza', 'steak', 'sushi']
{'pizza': 0, 'steak': 1, 'sushi': 2}
(225, 75)
('data/pizza_steak_sushi/train/pizza/1008844.jpg', 0)


Image tensor :
tensor([[[0.1176, 0.1216, 0.1255,  ..., 0.0980, 0.1020, 0.1137],
         [0.1294, 0.1294, 0.1294,  ..., 0.0980, 0.0980, 0.1059],
         [0.1333, 0.1333, 0.1333,  ..., 0.0941, 0.0980, 0.1020],
         ...,
         [0.1686, 0.1647, 0.1686,  ..., 0.1255, 0.1098, 0.1098],
         [0.1686, 0.1647, 0.1686,  ..., 0.1098, 0.0941, 0.0902],
         [0.1647, 0.1647, 0.1686,  ..., 0.0980, 0.0863, 0.0863]],

        [[0.0588, 0.0588, 0.0588,  ..., 0.0745, 0.0706, 0.0745],
         [0.0627, 0.0627, 0.0627,  ..., 0.0745, 0.0706, 0.0745],
         [0.0706, 0.0706, 0.0706,  ..., 0.0745, 0.0745, 0.0706],
         ...,
         [0.2392, 0.2392, 0.2510,  ..., 0.1373, 0.1333, 0.1255],
         [0.2314, 0.2392, 0.2510,  ..., 0.1255, 0.1176, 0.1098],
         [0.2275, 0.2353, 0.2431,  ..., 0.1137, 0.1059, 0.1020]],

        [[0.0196, 0.0196, 0.0157,  ..., 0.0902, 0.0902, 0.0941],
         [0.0196, 0.0157, 0.0196,  ..., 0.0902, 0.0863, 0.0902],
         [0.0196, 0.0157, 0.0157,  ..., 0.0902, 0.0902, 0.0902],
         ...,
         [0.1843, 0.1882, 0.1961,  ..., 0.1490, 0.1333, 0.1294],
         [0.1804, 0.1843, 0.1922,  ..., 0.1255, 0.1137, 0.1098],
...
Image label : 0 (its classname is pizza)
Label datatype : <class 'int'>


Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...
Original shape : torch.Size([3, 64, 64]) -> [color_channels, height, width]
Image permute : torch.Size([64, 64, 3]) -> [height, width, color_channels]
Text(0.5, 1.0, 'pizza')
Batch: 0, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 1, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 2, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 3, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 4, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 5, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 6, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 7, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 8, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 9, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 10, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 11, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 12, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 13, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 14, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 15, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 16, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 17, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 18, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 19, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 20, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 21, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 22, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 23, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 24, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
...
Batch: 221, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 222, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 223, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Batch: 224, (X : torch.Size([1, 3, 64, 64]), y: torch.Size([1]))
Output is truncated. View as a [scrollable element](command:cellOutput.enableScrolling?826143b3-270a-4c47-a8c7-401718b146f7) or open in a [text editor](command:workbench.action.openLargeOutput?826143b3-270a-4c47-a8c7-401718b146f7). Adjust cell output [settings](command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D)...
(['pizza', 'steak', 'sushi'], {'pizza': 0, 'steak': 1, 'sushi': 2})
Target dir: data/pizza_steak_sushi/train
[<DirEntry '.DS_Store'>,
 <DirEntry 'steak'>,
 <DirEntry 'sushi'>,
 <DirEntry 'pizza'>]
(['pizza', 'steak', 'sushi'], {'pizza': 0, 'steak': 1, 'sushi': 2})
(225, 225, 75, 75)
['pizza', 'steak', 'sushi']
{'pizza': 0, 'steak': 1, 'sushi': 2}
(True, True)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=95, pipe_handle=143)
                                                  ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.13.4/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/opt/homebrew/Cellar/[email protected]/3.13.4/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
AttributeError: Can't get attribute 'ImageFolderCustom' on <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>"

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File ~/.local/share/virtualenvs/PyTorch_-_fCC-xZTJLg_8/lib/python3.13/site-packages/torch/utils/data/dataloader.py:1284, in _MultiProcessingDataLoaderIter._try_get_data(self, timeout)
   1283 try:
-> [1284](https://file+.vscode-resource.vscode-cdn.net/Volumes/Development/Python/Learning/AI/PyTorch%20-%20fCC/Chapter%204/~/.local/share/virtualenvs/PyTorch_-_fCC-xZTJLg_8/lib/python3.13/site-packages/torch/utils/data/dataloader.py:1284)     data = self._data_queue.get(timeout=timeout)
   1285     return (True, data)

File /opt/homebrew/Cellar/[email protected]/3.13.4/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/queues.py:111, in Queue.get(self, block, timeout)
    110 timeout = deadline - time.monotonic()
--> [111](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Cellar/[email protected]/3.13.4/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/queues.py:111) if not self._poll(timeout):
    112     raise Empty

File /opt/homebrew/Cellar/[email protected]/3.13.4/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/connection.py:257, in _ConnectionBase.poll(self, timeout)
    256 self._check_readable()
--> [257](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Cellar/[email protected]/3.13.4/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/connection.py:257) return self._poll(timeout)

File /opt/homebrew/Cellar/[email protected]/3.13.4/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/connection.py:440, in Connection._poll(self, timeout)
    439 def _poll(self, timeout):
--> [440](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Cellar/[email protected]/3.13.4/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/connection.py:440)     r = wait([self], timeout)
    441     return bool(r)

File /opt/homebrew/Cellar/[email protected]/3.13.4/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/connection.py:1148, in wait(object_list, timeout)
   1147 while True:
-> [1148](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Cellar/[email protected]/3.13.4/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/connection.py:1148)     ready = selector.select(timeout)
   1149     if ready:
...
   1299     ) from e
   1300 if isinstance(e, queue.Empty):
   1301     return (False, None)

RuntimeError: DataLoader worker (pid(s) 9664) exited unexpectedly
Output is truncated. View as a [scrollable element](command:cellOutput.enableScrolling?8d9dd4c5-d9d2-4fde-b21f-622a928d79d0) or open in a [text editor](command:workbench.action.openLargeOutput?8d9dd4c5-d9d2-4fde-b21f-622a928d79d0). Adjust cell output [settings](command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D)...

[AnHumanLearner]

Finally I found how to resolve that, the custom image folder class has to be put into a python file because it need to be serialized