doesn't work with multi-process DataLoader

[vayzenb]

Hi,

I'm trying to speed up training by using a multi-processing in the DataLoader, however this throws an error. Any insight would be appreciated.

Here is a snippet of code

trainloader = torch.utils.data.DataLoader(train_dataset, batch_size=128, shuffle=True)
valloader = torch.utils.data.DataLoader(val_dataset, batch_size=128, shuffle=True, num_workers = 4, pin_memory=True)

learning_rule = KrotovsRule()
optimizer = Local(named_params=model.named_parameters(), lr=0.01)

evaluator = HebbianEvaluator(model=model, score_name='accuracy',
                                score_function=lambda engine: engine.state.metrics['accuracy'], epochs=1, supervised_from=-1)

trainer = HebbianTrainer(model=model, learning_rule=learning_rule, optimizer=optimizer, supervised_from=-1, device='cuda')

evaluator.attach(trainer.engine, Events.EPOCH_COMPLETED(every=1), trainloader, valloader)

trainer.run(train_loader=trainloader, epochs=1)

Here is the error:

Traceback (most recent call last):
  File "cornet_hebbian_training.py", line 84, in <module>
    trainer.run(train_loader=valloader, epochs=1)
  File "/home/vayzenbe/GitHub_Repos/GiNN/Models/pytorch_hebbian/trainers.py", line 31, in run
    self.engine.run(train_loader, max_epochs=epochs)
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/ignite/engine/engine.py", line 702, in run
    return self._internal_run()
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/ignite/engine/engine.py", line 775, in _internal_run
    self._handle_exception(e)
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/ignite/engine/engine.py", line 469, in _handle_exception
    raise e
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/ignite/engine/engine.py", line 745, in _internal_run
    time_taken = self._run_once_on_dataset()
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/ignite/engine/engine.py", line 850, in _run_once_on_dataset
    self._handle_exception(e)
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/ignite/engine/engine.py", line 469, in _handle_exception
    raise e
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/ignite/engine/engine.py", line 801, in _run_once_on_dataset
    self.state.batch = next(self._dataloader_iter)
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 517, in __next__
    data = self._next_data()
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1199, in _next_data
    return self._process_data(data)
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1225, in _process_data
    data.reraise()
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/torch/_utils.py", line 429, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/vayzenbe/GitHub_Repos/GiNN/Models/load_without_faces.py", line 62, in __getitem__
    tensor_image = self.transform(image)
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 60, in __call__
    img = t(img)
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 869, in forward
    i, j, h, w = self.get_params(img, self.scale, self.ratio)
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 831, in get_params
    log_ratio = torch.log(torch.tensor(ratio))
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/torch/cuda/__init__.py", line 161, in _lazy_init
    "Cannot re-initialize CUDA in forked subprocess. To use CUDA with "
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

[julestalloen]

Have you tried using the spawn start method, as suggested in the error?

torch.multiprocessing.set_start_method('spawn')

See https://pytorch.org/docs/stable/notes/multiprocessing.html#cuda-in-multiprocessing

[vayzenb]

Yes I did. The issue seems somewhat specific to pytorch-hebbian , since I don't have any issues parallelizing the dataloader when training other types of models.

I also get an error when I remove the num_workers argument and just use pin_memory, so the issue might not be specific to parallelization, but something more general about how inputs are being read. Again I don't normally have an issue with these same dataloaders with other models.

Overall, this project is super cool and useful, I'm just trying to speed up training where I can. Thanks!

Current run is terminating due to exception: cannot pin 'torch.cuda.LongTensor' only dense CPU tensors can be pinned
Engine run is terminating due to exception: cannot pin 'torch.cuda.LongTensor' only dense CPU tensors can be pinned
Traceback (most recent call last):
  File "cornet_hebbian_training.py", line 92, in <module>
    trainer.run(train_loader=trainloader, epochs=1)
  File "/home/vayzenbe/GitHub_Repos/GiNN/Models/pytorch_hebbian/trainers.py", line 31, in run
    self.engine.run(train_loader, max_epochs=epochs)
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/ignite/engine/engine.py", line 702, in run
    return self._internal_run()
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/ignite/engine/engine.py", line 775, in _internal_run
    self._handle_exception(e)
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/ignite/engine/engine.py", line 469, in _handle_exception
    raise e
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/ignite/engine/engine.py", line 745, in _internal_run
    time_taken = self._run_once_on_dataset()
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/ignite/engine/engine.py", line 850, in _run_once_on_dataset
    self._handle_exception(e)
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/ignite/engine/engine.py", line 469, in _handle_exception
    raise e
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/ignite/engine/engine.py", line 801, in _run_once_on_dataset
    self.state.batch = next(self._dataloader_iter)
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 517, in __next__
    data = self._next_data()
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 559, in _next_data
    data = _utils.pin_memory.pin_memory(data)
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/pin_memory.py", line 55, in pin_memory
    return [pin_memory(sample) for sample in data]
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/pin_memory.py", line 55, in <listcomp>
    return [pin_memory(sample) for sample in data]
  File "/home/vayzenbe/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/pin_memory.py", line 47, in pin_memory
    return data.pin_memory()
RuntimeError: cannot pin 'torch.cuda.LongTensor' only dense CPU tensors can be pinned

[julestalloen]

The issue might be related to this line. When the device is set to 'cuda', tensors will be cuda tensors by default. pin_memory=True should only be used for data loaded on CPU. Online I see users reporting similar issues regarding the combination of torch.set_default_tensor_type() and multiprocessing (https://discuss.pytorch.org/t/pin-memory-vs-sending-direct-to-gpu-from-dataset/33891/2).

A possible fix to circumvent the use of torch.set_default_tensor_type() is to pass the device parameter around and make sure all tensors are created with the correct device parameter. Feel free to submit a PR if you've found a solution for your use case.

I'm glad to hear you find the framework useful! It was built with my use case in mind but I look forward to contributions to expand it to other cases as well.