ValueError: Attempting to unscale FP16 gradients.

[QQQ300kuai]

Thanks for the greate work. When I was training the LLM model with the run.sh, I meet the following problem. The LLM model was training with FP16,

/data/InspireMusic/inspiremusic/utils/train_utils.py", line 252, in update_parameter_and_lr
scaler.unscale_(optimizer) # Unscale gradients before clipping
File "/data/InspireMusic/anaconda3/envs/inspiremusic/lib/python3.8/site-packages/torch/amp/grad_scaler.py", line 337, in unscale_
optimizer_state["found_inf_per_device"] = self.unscale_grads(
File "/data/InspireMusic/anaconda3/envs/inspiremusic/lib/python3.8/site-packages/torch/amp/grad_scaler.py", line 259, in unscale_grads
raise ValueError("Attempting to unscale FP16 gradients.")
ValueError: Attempting to unscale FP16 gradients.

[iris2c]

Thank you for your comments. However, we cannot reproduce this error.

[dongwon00kim]

I also got same error.

 traceback : Traceback (most recent call last):
    File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
      return f(*args, **kwargs)
    File "/workspace/MusicGen/inspiremusic/bin/train.py", line 192, in main
      executor.train_one_epoch(model, optimizer, scheduler, train_data_loader, cv_data_loader, writer, info_dict, group_join, scaler=scaler)
    File "/workspace/MusicGen/inspiremusic/utils/executor.py", line 77, in train_one_epoch
      info_dict = update_parameter_and_lr(model, optimizer, scheduler, info_dict, scaler)
    File "/workspace/MusicGen/inspiremusic/utils/train_utils.py", line 338, in update_parameter_and_lr
      scaler.unscale_(optimizer)  # Unscale gradients before clipping
    File "/opt/conda/lib/python3.10/site-packages/torch/amp/grad_scaler.py", line 337, in unscale_
      optimizer_state["found_inf_per_device"] = self._unscale_grads_(
    File "/opt/conda/lib/python3.10/site-packages/torch/amp/grad_scaler.py", line 259, in _unscale_grads_
      raise ValueError("Attempting to unscale FP16 gradients.")
  ValueError: Attempting to unscale FP16 gradients.

I built docker image using Dockerfile(https://github.com/FunAudioLLM/InspireMusic/blob/main/Dockerfile).

I changed antlr4-python3-runtime version to 4.9.3 in requirement.txt file due to dependency issue with hydra-core version 1.3.2.

@iris2c could you share Dockerfile or docker image?

pip list
Package                   Version
------------------------- ------------
absl-py                   2.2.2
accelerate                1.6.0
aiofiles                  24.1.0
aiohappyeyeballs          2.6.1
aiohttp                   3.11.16
aiosignal                 1.3.2
anaconda-anon-usage       0.4.4
annotated-types           0.7.0
antlr4-python3-runtime    4.9.3
anyio                     4.9.0
archspec                  0.2.3
asttokens                 2.0.5
astunparse                1.6.3
async-timeout             5.0.1
attrs                     23.1.0
audioread                 3.0.1
beautifulsoup4            4.12.2
boltons                   23.0.0
Brotli                    1.0.9
certifi                   2024.2.2
cffi                      1.16.0
chardet                   4.0.0
charset-normalizer        2.0.4
click                     8.1.7
coloredlogs               15.0.1
conda                     23.5.2
conda-build               24.3.0
conda-content-trust       0.2.0
conda_index               0.4.0
conda-libmamba-solver     23.7.0
conda-package-handling    2.2.0
conda_package_streaming   0.9.0
conformer                 0.3.2
contourpy                 1.3.1
cryptography              42.0.5
cycler                    0.12.1
decorator                 5.1.1
deepspeed                 0.16.5
diffusers                 0.33.1
distro                    1.8.0
dnspython                 2.6.1
docker-pycreds            0.4.0
einops                    0.8.1
exceptiongroup            1.2.0
executing                 0.8.3
expecttest                0.2.1
fastapi                   0.115.12
fastapi-cli               0.0.7
ffmpy                     0.5.0
filelock                  3.13.1
flash_attn                2.7.4.post1
flatbuffers               25.2.10
fonttools                 4.57.0
frozenlist                1.5.0
fsspec                    2024.3.1
gdown                     5.2.0
gitdb                     4.0.12
GitPython                 3.1.44
gmpy2                     2.1.2
gradio                    5.25.0
gradio_client             1.8.0
groovy                    0.1.2
grpcio                    1.71.0
grpcio-tools              1.71.0
h11                       0.14.0
hjson                     3.1.0
httpcore                  1.0.8
httptools                 0.6.4
httpx                     0.28.1
huggingface-hub           0.30.2
humanfriendly             10.0
hydra-core                1.3.2
HyperPyYAML               1.2.2
hypothesis                6.100.1
idna                      3.4
importlib_metadata        8.6.1
importlib_resources       6.5.2
inflect                   7.5.0
ipython                   8.20.0
jedi                      0.18.1
Jinja2                    3.1.3
joblib                    1.4.2
jsonpatch                 1.33
jsonpointer               2.1
jsonschema                4.19.2
jsonschema-specifications 2023.7.1
kiwisolver                1.4.8
lark                      1.1.9
lazy_loader               0.4
libarchive-c              2.9
libmambapy                1.5.8
librosa                   0.11.0
lightning                 2.5.1
lightning-utilities       0.14.3
llvmlite                  0.44.0
Markdown                  3.8
markdown-it-py            3.0.0
MarkupSafe                2.1.3
matplotlib                3.10.1
matplotlib-inline         0.1.6
mdurl                     0.1.2
menuinst                  2.0.2
mkl-fft                   1.3.8
mkl-random                1.2.4
mkl-service               2.4.0
modelscope                1.25.0
more-itertools            10.1.0
mpmath                    1.3.0
msgpack                   1.1.0
multidict                 6.4.3
networkx                  3.1
ninja                     1.11.1.4
numba                     0.61.2
numpy                     1.26.4
omegaconf                 2.3.0
onnx                      1.17.0
onnxruntime               1.21.0
onnxruntime-gpu           1.21.0
optree                    0.11.0
orjson                    3.10.16
packaging                 23.2
pandas                    2.2.3
parso                     0.8.3
peft                      0.13.2
pexpect                   4.8.0
pillow                    10.2.0
pip                       23.3.1
pkginfo                   1.9.6
platformdirs              3.10.0
pluggy                    1.0.0
pooch                     1.8.2
prompt-toolkit            3.0.43
propcache                 0.3.1
protobuf                  5.29.4
psutil                    5.9.0
ptyprocess                0.7.0
pure-eval                 0.2.2
py-cpuinfo                9.0.0
pyarrow                   19.0.1
pycosat                   0.6.6
pycparser                 2.21
pydantic                  2.11.3
pydantic_core             2.33.1
pydub                     0.25.1
Pygments                  2.15.1
pynini                    2.1.6
pyOpenSSL                 24.0.0
pyparsing                 3.2.3
PySocks                   1.7.1
pystoi                    0.4.1
python-dateutil           2.9.0.post0
python-dotenv             1.1.0
python-etcd               0.4.5
python-multipart          0.0.20
pytorch-lightning         2.5.1
pytz                      2023.3.post1
PyYAML                    6.0.1
referencing               0.30.2
regex                     2024.11.6
requests                  2.31.0
rich                      14.0.0
rich-toolkit              0.14.1
rpds-py                   0.10.6
ruamel.yaml               0.18.10
ruamel.yaml.clib          0.2.12
ruff                      0.11.5
safehttpx                 0.1.6
safetensors               0.5.3
scikit-learn              1.6.1
scipy                     1.15.2
semantic-version          2.10.0
sentry-sdk                2.25.1
setproctitle              1.3.5
setuptools                68.2.2
shellingham               1.5.4
six                       1.16.0
smmap                     5.0.2
sniffio                   1.3.1
sortedcontainers          2.4.0
soundfile                 0.13.1
soupsieve                 2.5
soxr                      0.5.0.post1
stack-data                0.2.0
starlette                 0.46.2
sympy                     1.12
tensorboard               2.19.0
tensorboard-data-server   0.7.2
threadpoolctl             3.6.0
tokenizers                0.20.3
tomli                     2.0.1
tomlkit                   0.13.2
toolz                     0.12.0
torch                     2.3.0
torchaudio                2.3.0
torchelastic              0.2.2
torchmetrics              1.7.1
torchvision               0.18.0
tqdm                      4.65.0
traitlets                 5.7.1
transformers              4.46.3
triton                    2.3.0
truststore                0.8.0
typeguard                 4.4.2
typer                     0.15.2
types-dataclasses         0.6.6
typing_extensions         4.13.2
typing-inspection         0.4.0
tzdata                    2025.2
urllib3                   2.1.0
uvicorn                   0.34.1
uvloop                    0.21.0
wandb                     0.19.9
watchfiles                1.0.5
wcwidth                   0.2.5
websockets                15.0.1
Werkzeug                  3.1.3
WeTextProcessing          1.0.4.1
wget                      3.2
wheel                     0.41.2
yarl                      1.19.0
zipp                      3.21.0
zstandard                 0.19.0

[dongwon00kim]

This issue appears to be identical to issue #18.

[dongwon00kim]

Dear @iris2c, @QQQ300kuai

I modified https://github.com/FunAudioLLM/InspireMusic/blob/main/examples/music_generation/conf/inspiremusic_base.yaml file as below.

I changed the dtype of qwen encoder(llm) to fp32. Please check if this is correct.

llm: !new:inspiremusic.llm.llm.LLM
    text_encoder_input_size: !ref <text_encoder_input_size>
    llm_input_size: !ref <llm_input_size>
    llm_output_size: !ref <llm_output_size>
    audio_token_size: 4096
    length_normalized_loss: True
    lsm_weight: 0
    text_encoder_conf:
        name: "none"
    llm: !new:inspiremusic.transformer.qwen_encoder.QwenEmbeddingEncoder
        input_size: !ref <text_encoder_input_size>
        pretrain_path: !ref <basemodel_path>
        dtype: "fp32"

[iris2c]

I also got same error.

 traceback : Traceback (most recent call last):
    File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
      return f(*args, **kwargs)
    File "/workspace/MusicGen/inspiremusic/bin/train.py", line 192, in main
      executor.train_one_epoch(model, optimizer, scheduler, train_data_loader, cv_data_loader, writer, info_dict, group_join, scaler=scaler)
    File "/workspace/MusicGen/inspiremusic/utils/executor.py", line 77, in train_one_epoch
      info_dict = update_parameter_and_lr(model, optimizer, scheduler, info_dict, scaler)
    File "/workspace/MusicGen/inspiremusic/utils/train_utils.py", line 338, in update_parameter_and_lr
      scaler.unscale_(optimizer)  # Unscale gradients before clipping
    File "/opt/conda/lib/python3.10/site-packages/torch/amp/grad_scaler.py", line 337, in unscale_
      optimizer_state["found_inf_per_device"] = self._unscale_grads_(
    File "/opt/conda/lib/python3.10/site-packages/torch/amp/grad_scaler.py", line 259, in _unscale_grads_
      raise ValueError("Attempting to unscale FP16 gradients.")
  ValueError: Attempting to unscale FP16 gradients.

I built docker image using Dockerfile(https://github.com/FunAudioLLM/InspireMusic/blob/main/Dockerfile).

I changed antlr4-python3-runtime version to 4.9.3 in requirement.txt file due to dependency issue with hydra-core version 1.3.2.

@iris2c could you share Dockerfile or docker image?

pip list
Package                   Version
------------------------- ------------
absl-py                   2.2.2
accelerate                1.6.0
aiofiles                  24.1.0
aiohappyeyeballs          2.6.1
aiohttp                   3.11.16
aiosignal                 1.3.2
anaconda-anon-usage       0.4.4
annotated-types           0.7.0
antlr4-python3-runtime    4.9.3
anyio                     4.9.0
archspec                  0.2.3
asttokens                 2.0.5
astunparse                1.6.3
async-timeout             5.0.1
attrs                     23.1.0
audioread                 3.0.1
beautifulsoup4            4.12.2
boltons                   23.0.0
Brotli                    1.0.9
certifi                   2024.2.2
cffi                      1.16.0
chardet                   4.0.0
charset-normalizer        2.0.4
click                     8.1.7
coloredlogs               15.0.1
conda                     23.5.2
conda-build               24.3.0
conda-content-trust       0.2.0
conda_index               0.4.0
conda-libmamba-solver     23.7.0
conda-package-handling    2.2.0
conda_package_streaming   0.9.0
conformer                 0.3.2
contourpy                 1.3.1
cryptography              42.0.5
cycler                    0.12.1
decorator                 5.1.1
deepspeed                 0.16.5
diffusers                 0.33.1
distro                    1.8.0
dnspython                 2.6.1
docker-pycreds            0.4.0
einops                    0.8.1
exceptiongroup            1.2.0
executing                 0.8.3
expecttest                0.2.1
fastapi                   0.115.12
fastapi-cli               0.0.7
ffmpy                     0.5.0
filelock                  3.13.1
flash_attn                2.7.4.post1
flatbuffers               25.2.10
fonttools                 4.57.0
frozenlist                1.5.0
fsspec                    2024.3.1
gdown                     5.2.0
gitdb                     4.0.12
GitPython                 3.1.44
gmpy2                     2.1.2
gradio                    5.25.0
gradio_client             1.8.0
groovy                    0.1.2
grpcio                    1.71.0
grpcio-tools              1.71.0
h11                       0.14.0
hjson                     3.1.0
httpcore                  1.0.8
httptools                 0.6.4
httpx                     0.28.1
huggingface-hub           0.30.2
humanfriendly             10.0
hydra-core                1.3.2
HyperPyYAML               1.2.2
hypothesis                6.100.1
idna                      3.4
importlib_metadata        8.6.1
importlib_resources       6.5.2
inflect                   7.5.0
ipython                   8.20.0
jedi                      0.18.1
Jinja2                    3.1.3
joblib                    1.4.2
jsonpatch                 1.33
jsonpointer               2.1
jsonschema                4.19.2
jsonschema-specifications 2023.7.1
kiwisolver                1.4.8
lark                      1.1.9
lazy_loader               0.4
libarchive-c              2.9
libmambapy                1.5.8
librosa                   0.11.0
lightning                 2.5.1
lightning-utilities       0.14.3
llvmlite                  0.44.0
Markdown                  3.8
markdown-it-py            3.0.0
MarkupSafe                2.1.3
matplotlib                3.10.1
matplotlib-inline         0.1.6
mdurl                     0.1.2
menuinst                  2.0.2
mkl-fft                   1.3.8
mkl-random                1.2.4
mkl-service               2.4.0
modelscope                1.25.0
more-itertools            10.1.0
mpmath                    1.3.0
msgpack                   1.1.0
multidict                 6.4.3
networkx                  3.1
ninja                     1.11.1.4
numba                     0.61.2
numpy                     1.26.4
omegaconf                 2.3.0
onnx                      1.17.0
onnxruntime               1.21.0
onnxruntime-gpu           1.21.0
optree                    0.11.0
orjson                    3.10.16
packaging                 23.2
pandas                    2.2.3
parso                     0.8.3
peft                      0.13.2
pexpect                   4.8.0
pillow                    10.2.0
pip                       23.3.1
pkginfo                   1.9.6
platformdirs              3.10.0
pluggy                    1.0.0
pooch                     1.8.2
prompt-toolkit            3.0.43
propcache                 0.3.1
protobuf                  5.29.4
psutil                    5.9.0
ptyprocess                0.7.0
pure-eval                 0.2.2
py-cpuinfo                9.0.0
pyarrow                   19.0.1
pycosat                   0.6.6
pycparser                 2.21
pydantic                  2.11.3
pydantic_core             2.33.1
pydub                     0.25.1
Pygments                  2.15.1
pynini                    2.1.6
pyOpenSSL                 24.0.0
pyparsing                 3.2.3
PySocks                   1.7.1
pystoi                    0.4.1
python-dateutil           2.9.0.post0
python-dotenv             1.1.0
python-etcd               0.4.5
python-multipart          0.0.20
pytorch-lightning         2.5.1
pytz                      2023.3.post1
PyYAML                    6.0.1
referencing               0.30.2
regex                     2024.11.6
requests                  2.31.0
rich                      14.0.0
rich-toolkit              0.14.1
rpds-py                   0.10.6
ruamel.yaml               0.18.10
ruamel.yaml.clib          0.2.12
ruff                      0.11.5
safehttpx                 0.1.6
safetensors               0.5.3
scikit-learn              1.6.1
scipy                     1.15.2
semantic-version          2.10.0
sentry-sdk                2.25.1
setproctitle              1.3.5
setuptools                68.2.2
shellingham               1.5.4
six                       1.16.0
smmap                     5.0.2
sniffio                   1.3.1
sortedcontainers          2.4.0
soundfile                 0.13.1
soupsieve                 2.5
soxr                      0.5.0.post1
stack-data                0.2.0
starlette                 0.46.2
sympy                     1.12
tensorboard               2.19.0
tensorboard-data-server   0.7.2
threadpoolctl             3.6.0
tokenizers                0.20.3
tomli                     2.0.1
tomlkit                   0.13.2
toolz                     0.12.0
torch                     2.3.0
torchaudio                2.3.0
torchelastic              0.2.2
torchmetrics              1.7.1
torchvision               0.18.0
tqdm                      4.65.0
traitlets                 5.7.1
transformers              4.46.3
triton                    2.3.0
truststore                0.8.0
typeguard                 4.4.2
typer                     0.15.2
types-dataclasses         0.6.6
typing_extensions         4.13.2
typing-inspection         0.4.0
tzdata                    2025.2
urllib3                   2.1.0
uvicorn                   0.34.1
uvloop                    0.21.0
wandb                     0.19.9
watchfiles                1.0.5
wcwidth                   0.2.5
websockets                15.0.1
Werkzeug                  3.1.3
WeTextProcessing          1.0.4.1
wget                      3.2
wheel                     0.41.2
yarl                      1.19.0
zipp                      3.21.0
zstandard                 0.19.0

Thank you for your comments.
To install the environment, you may refer to the conda environment file (https://github.com/FunAudioLLM/InspireMusic/blob/main/envs/inspiremusic.yaml).

conda env create -f inspiremusic.yml