Can't find trained model in my output_path

[adueck]

Hi everyone, I'm new to TTS and I'm trying to train a model to read single words in a new language. To do this I have 16k great, clear audio samples of dictionary words. I created a dataset that seems to load and train fine, and I get nice results on my tensorboard panel, but for the life of me I cannot find the model files like checkpoint_x.pth anywhere after training!

I tried to follow these steps, running it on the docker image ghcr.io/coqui-ai/tts-cpu

Here's my script I'm using for training:

import os

# Trainer: Where the ✨️ happens.
# TrainingArgs: Defines the set of arguments of the Trainer.
from trainer import Trainer, TrainerArgs

# GlowTTSConfig: all model related values for training, validating and testing.
from TTS.tts.configs.glow_tts_config import GlowTTSConfig

# BaseDatasetConfig: defines name, formatter and path of the dataset.
from TTS.tts.configs.shared_configs import BaseDatasetConfig
from TTS.tts.datasets import load_tts_samples
from TTS.tts.models.glow_tts import GlowTTS
from TTS.tts.utils.text.tokenizer import TTSTokenizer
from TTS.utils.audio import AudioProcessor

output_path = "models/out"

dataset_config = BaseDatasetConfig(
    formatter="ljspeech", language="en-us",  meta_file_train="metadata.txt", path="dataset/f-dataset/")
)

config = GlowTTSConfig(
    batch_size=128,
    eval_batch_size=65,
    num_loader_workers=4,
    num_eval_loader_workers=4,
    run_eval=True,
    test_delay_epochs=-1,
    epochs=8,
    text_cleaner="phoneme_cleaners",
    use_phonemes=True,
    phoneme_language="en-us",
    phoneme_cache_path=os.path.join(output_path, "phoneme_cache"),
    print_step=25,
    print_eval=False,
    mixed_precision=True,
    output_path="models/out",
    datasets=[dataset_config],
)

# INITIALIZE THE AUDIO PROCESSOR
# Audio processor is used for feature extraction and audio I/O.
# It mainly serves to the dataloader and the training loggers.
ap = AudioProcessor.init_from_config(config)

# INITIALIZE THE TOKENIZER
# Tokenizer is used to convert text to sequences of token IDs.
# If characters are not defined in the config, default characters are passed to the config
tokenizer, config = TTSTokenizer.init_from_config(config)

train_samples, eval_samples = load_tts_samples(
    dataset_config,
    eval_split=True,
    eval_split_max_size=config.eval_split_max_size,
    eval_split_size=config.eval_split_size,
)

# INITIALIZE THE MODEL
# Models take a config object and a speaker manager as input
# Config defines the details of the model like the number of layers, the size of the embedding, etc.
# Speaker manager is used by multi-speaker models.
model = GlowTTS(config, ap, tokenizer, speaker_manager=None)

# INITIALIZE THE TRAINER
# Trainer provides a generic API to train all the 🐸TTS models with all its perks like mixed-precision training,
# distributed training, etc.
trainer = Trainer(
    TrainerArgs(), config, output_path, model=model, train_samples=train_samples, eval_samples=eval_samples
)

trainer.fit()

All the training seems to go well, and I can run the tensorboard in the output directory just fine, it shows all the graphs and evaluation of the models.

$ tensorboard --logdir=<path to your training directory>

But in the output directory all I see is this:

$ ~/models/out$ ls
run-December-25-2024_01+54AM-0000000
$ ~/models/out$ cd run-December-25-2024_01+54AM-0000000
$ ls -lh
total 184K
-rw-r--r-- 1 root root 6.0K Dec 25 01:54 config.json
-rw-r--r-- 1 root root 166K Dec 25 02:23 events.out.tfevents.1735091655.3e5cc5bb266b.95.0
-rw-r--r-- 1 root root 2.5K Dec 25 01:54 train.py
-rw-r--r-- 1 root root 2.7K Dec 25 02:23 trainer_0_log.txt
config.json  events.out.tfevents.1735091655.3e5cc5bb266b.95.0  train.py  trainer_0_log.txt

Step 7. says to "Use your best model for inference." Where are these models? How do I choose them? There's nothing else I can find anywhere. All there is are the little config and log files in the output directory. Am I missing a step? How and where do I find the actual model files I can use with like this?

$ tts --text "Text for TTS" \
      --model_path path/to/checkpoint_x.pth \
      --config_path path/to/config.json \
      --out_path folder/to/save/output.wav

Thanks so much!

[eginhard]

By default, the model is saved only every 10k steps, so you haven't trained long enough yet. You can set a different value for save_step in your GlowTTSConfig.

[adueck]

Thanks so much. That makes sense. To confirm, that mean steps (like appears in the "global steps" of the progress) not epochs right?

[eginhard]

Yes, the 1048 in your screenshot.