-
Notifications
You must be signed in to change notification settings - Fork 315
Open
Description
Running from pytorch main
python benchmarks/dynamo/torchbench.py --only opacus_cifar10 --accuracy --no-translation-validation --training --amp --backend inductor --device cuda --output out.csv
cuda train opacus_cifar10
Traceback (most recent call last):
File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2744, in validate_model
self.model_iter_fn(model, example_inputs)
File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 455, in forward_and_backward_pass
self.grad_scaler.scale(loss).backward()
File "/data/users/ivankobzarev/a/pytorch/torch/_tensor.py", line 648, in backward
torch.autograd.backward(
File "/data/users/ivankobzarev/a/pytorch/torch/autograd/__init__.py", line 353, in backward
_engine_run_backward(
File "/data/users/ivankobzarev/a/pytorch/torch/autograd/graph.py", line 815, in _engine_run_backward
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
File "/data/users/ivankobzarev/a/pytorch/torch/nn/modules/module.py", line 86, in __call__
return self.hook(module, *args, **kwargs)
File "/data/users/ivankobzarev/a/opacus/opacus/grad_sample/grad_sample_module.py", line 327, in capture_backprops_hook
activations, backprops = self.rearrange_grad_samples(
File "/data/users/ivankobzarev/a/opacus/opacus/grad_sample/grad_sample_module.py", line 383, in rearrange_grad_samples
raise ValueError(
ValueError: No activations detected for <class 'torch.nn.modules.linear.Linear'>, run forward after add_hooks(model)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 4857, in run
) = runner.load_model(
File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 372, in load_model
self.validate_model(model, example_inputs)
File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2746, in validate_model
raise RuntimeError("Eager run failed") from e
RuntimeError: Eager run failed
eager_fail_to_run
Metadata
Metadata
Assignees
Labels
No labels