"use_speaker_embeddings" and "use_d_vector" options

[MatthewRomanishin]

I want to train a model with the same parameters as your YourTTS model you use for voice conversion in colab notebook. I looked at the "use_speaker_embeddings" and "use_d_vector" options. Both of these options are True in the config.json file. I calculated the embeddings for my dataset using compute_embeddings.py. I tried to run model training with "use_speaker_embeddings"=true and "use_d_vector"=true, got the error "ValueError: [!] Speaker embedding layer already initialized before d_vector settings.". I have read the discussion .

Do I understand correctly that the model can only be trained with one of the parameters equal to True? How did you train the model with both parameters equal to true? If I set only one parameter to true, how will it affect the result of the inference?

[vince62s]

read this #1058

[MatthewRomanishin]

Thank you @vince62s. Sorry, but I did not find the answer to the last two questions there.

[vince62s]

either you pick use_speaker_embeddings (set to true) to train embeddings with the model, OR you use d_vectors and feed your model with externally computed embeddings.

[MatthewRomanishin]

@vince62s Thanks a lot!

[MatthewRomanishin]

@vince62s Hi! I ran into a problem while training a Vits model using "d_vector_file". I got an error: "AttributeError: 'WN' object has no attribute 'cond_layer' ".
Have you encountered such a problem?
Do you have any ideas?

Thanks.

[catalinnega]

You should specify "d_vector_dim" to VitsArgs(). By default it is set to 0, thats why the "cond_layer" within the PositionalEncoder (which uses WaveNet (WN) layers) is not initialized.

[MatthewRomanishin]

Thank you very much! I solved this problem shortly before you answered.
I have one last question. When I use "d_vector_file", in my config.json "num_speakers" is set to 0. Is this normal behavior?
Just for voice conversion it is necessary that num_speakers be > 0

[catalinnega]

"num_speakers" is used to initialize speaker_embedding layers. For d-vectors, these layers are not created, d-vectors are used only for layer conditioning. When you are using use_speaker_embedding=False, the "num_speakers" setting will not affect your architecture.
Seems that the voice_conversion() function expects both "use_speaker_embedding" and "use_d_vector_file" configs to be True when using d_vectors, I have no idea why..
To save the number of speakers in the config.json export, you have to manually add it before declaring the model as in

TTS/recipes/vctk/vits/train_vits.py

Line 76 in c63bb48

config.model_args.num_speakers = speaker_manager.num_speakers

[OmiWakode]

Hey, I got the same error before passing d_vector_dim. But when I try to pass d_vector_dim in VitsArgs and VitsConfig, I get another error. . Do you have any idea about this?

[vince62s]

if you use the vits recipe you should not face this error that deals with WaveNet when vits does not use WN but Hifigan.