"Specify the reduction dim with the 'dim' argument" Runtime error while training VITS model

[freezerain]

Hello. I switched from Glow-TTS to VITS but no matter what I was not able to start training it localy.
I got "RuntimeError: min(): Expected reduction dim to be specified for input.numel() == 0. Specify the reduction dim with the 'dim' argument." on first iteration.

Does anybody can point where to start? I will try to reproduce exception with english dataset in notebook later on.
model:

import os

from trainer import Trainer, TrainerArgs

from TTS.config import BaseAudioConfig
from TTS.tts.configs.shared_configs import BaseDatasetConfig, CharactersConfig
from TTS.tts.configs.vits_config import VitsConfig
from TTS.tts.datasets import load_tts_samples
from TTS.tts.models.vits import Vits, VitsAudioConfig
from TTS.tts.utils.text.tokenizer import TTSTokenizer
from TTS.utils.audio import AudioProcessor

output_path = os.path.dirname(os.path.abspath(__file__))

dataset_config = BaseDatasetConfig(
    name="ruslan", meta_file_train="metadata_RUSLAN_22200.csv",
    path=os.path.join(output_path, "ruslan_dataset")
)

audio_config = VitsAudioConfig(
    sample_rate=22050, win_length=1024, hop_length=256, num_mels=80, mel_fmin=0, mel_fmax=None,
)

config = VitsConfig(
    audio=audio_config,
    run_name="vits_1",
    batch_size=16,
    run_eval=True,
    test_delay_epochs=-1,
    epochs=100,
    text_cleaner="phoneme_cleaners",
    use_phonemes=False,
    phoneme_language="ru-ru",
    phoneme_cache_path=os.path.join(output_path, "phoneme_cache"),
    compute_input_seq_cache=True,
    print_step=25,
    print_eval=False,
    mixed_precision=True,
    output_path=output_path,
    datasets=[dataset_config],
    cudnn_benchmark=False,
    characters=CharactersConfig(
        characters_class="TTS.tts.utils.text.characters.IPAPhonemes",
        pad="<PAD>",
        eos="<EOS>",
        bos="<BOS>",
        blank="<BLNK>",
        characters="\u0430\u0431\u0432\u0433\u0434\u0435\u0436\u0437\u0438\u0439\u043a\u043b\u043c\u043d\u043e\u043f\u0440\u0441\u0442\u0443\u0444\u0445\u0446\u0447\u0448\u0449\u044a\u044b\u044c\u044d\u044e\u044f\u0451abcdefghijklmnopqrstuvwxyz",
        punctuations="!(),-.:;?-\u2013\u2014 \"'\u2019",
        is_unique=False,
        phonemes=None,
    ),
)

ap = AudioProcessor.init_from_config(config)

tokenizer, config = TTSTokenizer.init_from_config(config)

train_samples, eval_samples = load_tts_samples(
    dataset_config,
    eval_split=True,
    eval_split_max_size=config.eval_split_max_size,
    eval_split_size=config.eval_split_size,
)

model = Vits(config, ap, tokenizer, speaker_manager=None)

trainer = Trainer(
    TrainerArgs(), config, output_path,
    model=model, ntrain_samples=train_samples, eval_samples=eval_samples,
)
trainer.fit()

Stacktrace:

D:\Soft\Projects\TTS\venv\lib\site-packages\torch\functional.py:606: UserWarning: stft will soon require the return_complex parameter be given for real inputs, and will further require that return_complex=True in a future PyTorch release. (Triggered internally at  C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:803.)
  return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
Traceback (most recent call last):
  File "D:\Soft\Projects\TTS\venv\lib\site-packages\trainer\trainer.py", line 1533, in fit
    self._fit()
  File "D:\Soft\Projects\TTS\venv\lib\site-packages\trainer\trainer.py", line 1517, in _fit
    self.train_epoch()
  File "D:\Soft\Projects\TTS\venv\lib\site-packages\trainer\trainer.py", line 1282, in train_epoch
    _, _ = self.train_step(batch, batch_num_steps, cur_step, loader_start_time)
  File "D:\Soft\Projects\TTS\venv\lib\site-packages\trainer\trainer.py", line 1135, in train_step
    outputs, loss_dict_new, step_time = self._optimize(
  File "D:\Soft\Projects\TTS\venv\lib\site-packages\trainer\trainer.py", line 996, in _optimize
    outputs, loss_dict = self._model_train_step(batch, model, criterion, optimizer_idx=optimizer_idx)
  File "D:\Soft\Projects\TTS\venv\lib\site-packages\trainer\trainer.py", line 954, in _model_train_step
    return model.train_step(*input_args)
  File "D:\Soft\Projects\TTS\TTS\tts\models\vits.py", line 1247, in train_step
    outputs = self.forward(
  File "D:\Soft\Projects\TTS\TTS\tts\models\vits.py", line 1016, in forward
    outputs, attn = self.forward_mas(outputs, z_p, m_p, logs_p, x, x_mask, y_mask, g=g, lang_emb=lang_emb)
  File "D:\Soft\Projects\TTS\TTS\tts\models\vits.py", line 913, in forward_mas
    loss_duration = self.duration_predictor(
  File "D:\Soft\Projects\TTS\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\Soft\Projects\TTS\TTS\tts\layers\vits\stochastic_duration_predictor.py", line 255, in forward
    z_q, logdet_q = flow(z_q, x_mask, g=(x + h))
  File "D:\Soft\Projects\TTS\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\Soft\Projects\TTS\TTS\tts\layers\vits\stochastic_duration_predictor.py", line 133, in forward
    x1, logabsdet = piecewise_rational_quadratic_transform(
  File "D:\Soft\Projects\TTS\TTS\tts\layers\vits\transforms.py", line 32, in piecewise_rational_quadratic_transform
    outputs, logabsdet = spline_fn(
  File "D:\Soft\Projects\TTS\TTS\tts\layers\vits\transforms.py", line 80, in unconstrained_rational_quadratic_spline
    outputs[inside_interval_mask], logabsdet[inside_interval_mask] = rational_quadratic_spline(
  File "D:\Soft\Projects\TTS\TTS\tts\layers\vits\transforms.py", line 112, in rational_quadratic_spline
    if torch.min(inputs) < left or torch.max(inputs) > right:
RuntimeError: min(): Expected reduction dim to be specified for input.numel() == 0. Specify the reduction dim with the 'dim' argument.

[erogol]

What is your torch version? It might a bug that comes with the latest torch release.

[freezerain]

I was having a bad time reinstalling torch before switchg to VITS, no matter what it was installing cpu version without cuda even if command from pytorch website is used, I finally found a way to force 1.21.1+cu113 version, however looks like it leads to this exception.

I downgraded to this version:

pip install torch==1.8.0+cu101 torchvision==0.9.0+cu101 torchaudio===0.8.0 -f https://download.pytorch.org/whl/torch_stable.html

and the problem was fixed. thank you

[tieincred]

How did you decide to downgrade to this torch and did you change code in anyway to make the entire thing work?
@freezerain

[freezerain]

I was googling until found specific, relatively new combination of versions that were working together for author. For the time that was a minor downgrade and no code were modified, only dependency

[Enucatl]

In my case I had saved the files with the wrong frame rate, and it didn't match the audio_config anymore. Fixing that also fixed this error.

[offside609]

@freezerain I downgraded to your torch version with combinations. I am getting a new error now.

[Thomcle]

I got the same error when I computed the conditioning latents of a short audio (the length is less than 1 second, I guess).

You can either pass short audios when training or add silence.