Kokoro usage improvements by michalkulakowski · Pull Request #4357 · openvinotoolkit/model_server

michalkulakowski · 2026-07-03T13:56:19Z

🛠 Summary

JIRA/Issue if applicable.
Describe the changes.

🧪 Checklist

Unit tests added.
The documentation updated.
Change follows security best practices.
``

Copilot

Pull request overview

This PR updates Kokoro text-to-speech (TTS) integration by shifting voice embedding discovery to runtime (from the model directory) and moving espeak-ng from an OVMS binary dependency to a separately built artifact in Docker builds.

Changes:

Load Kokoro voice embeddings from <models_path>/voices/*.bin when the graph doesn’t explicitly specify voices.
Remove Bazel --//:espeak=on/off flag plumbing and build espeak-ng as an optional standalone Docker step (ARG ESPEAK=1/0).
Adjust tests and export tooling to align with the new Kokoro usage expectations.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
windows_build.bat	Removes Bazel espeak flag wiring from Windows build invocation.
third_party/BUILD	Makes espeak-ng aliases unconditional (no build-flag select).
src/test/graph_export_test.cpp	Removes Kokoro voices list expectations and voices/ directory precreation.
src/test/audio/text2speech_test.cpp	Adds config validation test for missing `voices` in graph.
src/graph_export/graph_export.cpp	Stops enumerating Kokoro `voices/*.bin` into generated graph templates.
src/BUILD	Removes espeak-ng deps from the OVMS binary build.
src/audio/text_to_speech/t2s_servable.cpp	Implements fallback loading of embeddings from `<models_path>/voices` when graph voices are omitted.
src/audio/text_to_speech/t2s_calculator.cc	Minor comment cleanup in voice-selection error handling.
Makefile	Removes Bazel espeak flag usage; passes ESPEAK as a Docker build arg.
Dockerfile.ubuntu	Adds optional standalone Bazel build step for espeak-ng targets controlled by `ARG ESPEAK`.
Dockerfile.redhat	Same as Ubuntu Dockerfile: optional standalone espeak-ng build step.
distro.bzl	Removes the Bazel espeak build flag definition/config settings.
demos/common/export_models/export_model.py	Changes TTS exporter behavior, including default `--model_type`.
demos/audio/README.md	Adds documentation for ASR leaderboard-based transcription evaluation.
common_settings.bzl	Stops loading/creating the removed espeak flag config settings.

 #include <fstream>
 #include <sstream>
 #include <limits>
+#include <vector>


+static std::vector<std::filesystem::path> getVoiceEmbeddingPaths(const std::filesystem::path& voicesDir) {
+    std::vector<std::filesystem::path> voicePaths;
+    std::error_code ec;
+    for (const auto& entry : std::filesystem::directory_iterator(voicesDir, ec)) {
+        if (ec) {
+            throw std::runtime_error("Failed to iterate voices directory: " + voicesDir.string());
+        }
+        if (!entry.is_regular_file(ec) || ec) {
+            ec.clear();
+            continue;
+        }
+        if (entry.path().extension() == ".bin") {
+            voicePaths.emplace_back(entry.path());
+        }
+    }
+    std::sort(voicePaths.begin(), voicePaths.end());
+    return voicePaths;
+}


 add_common_arguments(parser_text2speech)
 parser_text2speech.add_argument('--num_streams', default=0, type=int, help='The number of parallel execution streams to use for the models in the pipeline.', dest='num_streams')
-parser_text2speech.add_argument('--model_type', default='speecht5', choices=['speecht5', 'kokoro'], help='Type of the source TTS model. speecht5 uses optimum-cli; kokoro uses a dedicated PyTorch->OpenVINO conversion path.', dest='model_type')
+parser_text2speech.add_argument('--model_type', default='kokoro', choices=['speecht5', 'kokoro'], help='Type of the source TTS model. speecht5 uses optimum-cli; kokoro uses a dedicated PyTorch->OpenVINO conversion path.', dest='model_type')


+        [type.googleapis.com / mediapipe.T2sCalculatorOptions]: {
+            models_path: "/ovms/models_audio/Kokoro-82M"
+            plugin_config: '{"NUM_STREAMS": "1" }',
+            target_device: "CPU"
+        }
+        }


Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

 add_common_arguments(parser_text2speech)
 parser_text2speech.add_argument('--num_streams', default=0, type=int, help='The number of parallel execution streams to use for the models in the pipeline.', dest='num_streams')
-parser_text2speech.add_argument('--model_type', default='speecht5', choices=['speecht5', 'kokoro'], help='Type of the source TTS model. speecht5 uses optimum-cli; kokoro uses a dedicated PyTorch->OpenVINO conversion path.', dest='model_type')
+parser_text2speech.add_argument('--model_type', default='kokoro', choices=['speecht5', 'kokoro'], help='Type of the source TTS model. speecht5 uses optimum-cli; kokoro uses a dedicated PyTorch->OpenVINO conversion path.', dest='model_type')


dtrawins · 2026-07-08T10:01:31Z

+Kokoro is the primary example in this demo, but SpeechT5 remains supported for existing deployments.
 ### Model preparation
-Supported models should use the topology of [microsoft/speecht5_tts](https://huggingface.co/microsoft/speecht5_tts) which needs to be converted to IR format before using in OVMS.
+Supported models should use the topology of [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) and be converted to IR format before using in OVMS.


you can use https://huggingface.co/luis-castillo/Kokoro-82M-OpenVINO-FP16-OVMS/ till there is official model published

dtrawins · 2026-07-08T21:44:15Z

-> **Note:** Exporting `microsoft/speecht5_tts` model requires Python 3.10
-
-**CPU**
+Create the model directory:


this step is not needed - duplicated

dtrawins · 2026-07-08T21:45:27Z

 **Deploying on Bare Metal**

 ```bat
 mkdir models


on windows it's better to use c:\models

dtrawins · 2026-07-08T21:46:02Z

 ```bash
-mkdir -p models
-docker run -d -u $(id -u):$(id -g) --rm -p 8000:8000 -v $(pwd)/models:/models:rw openvino/model_server:latest --rest_port 8000 --model_path /models/microsoft/speecht5_tts --model_name microsoft/speecht5_tts
+docker run -d -u $(id -u):$(id -g) --rm -p 8000:8000 -v $(pwd)/models:/models:rw openvino/model_server:latest --rest_port 8000 --source_model luis-castillo/Kokoro-82M-OpenVINO-FP16-OVMS --model_repository_path /models --model_name Kokoro-82M-OpenVINO-FP16-OVMS


Suggested change

docker run -d -u $(id -u):$(id -g) --rm -p 8000:8000 -v $(pwd)/models:/models:rw openvino/model_server:latest --rest_port 8000 --source_model luis-castillo/Kokoro-82M-OpenVINO-FP16-OVMS --model_repository_path /models --model_name Kokoro-82M-OpenVINO-FP16-OVMS

mkdir models

docker run -d -u $(id -u):$(id -g) --rm -p 8000:8000 -v $(pwd)/models:/models:rw openvino/model_server:latest --rest_port 8000 --source_model luis-castillo/Kokoro-82M-OpenVINO-FP16-OVMS --model_repository_path /models --model_name Kokoro-82M-OpenVINO-FP16-OVMS

dtrawins · 2026-07-08T21:48:45Z

-The default configuration should work in most cases but the parameters can be tuned via `export_model.py` script arguments. Run the script with `--help` argument to check available parameters and see the [T2s calculator documentation](../../docs/speech_generation/reference.md) to learn more about configuration options and limitations.
+See the [T2s calculator documentation](../../docs/speech_generation/reference.md) to learn more about configuration options and limitations.

 ### Deployment


there should be included option for deployment on GPU and NPU

dtrawins · 2026-07-08T21:53:36Z

 if [ -n "$ESPEAK_DATA_SRC" ] && [ -d "$ESPEAK_DATA_SRC" ] ; then
    mkdir -p /ovms_release/share
    cp -rL "$ESPEAK_DATA_SRC" /ovms_release/share/ ;
+    chmod -R u+w /ovms_release/share/espeak-ng-data 2>/dev/null || true ;


add also a note in building from src doc with info that espeek is optional and can be delete from the package if not needed. explain the consequences - which language is supported without it

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.

 add_common_arguments(parser_text2speech)
 parser_text2speech.add_argument('--num_streams', default=0, type=int, help='The number of parallel execution streams to use for the models in the pipeline.', dest='num_streams')
-parser_text2speech.add_argument('--model_type', default='speecht5', choices=['speecht5', 'kokoro'], help='Type of the source TTS model. speecht5 uses optimum-cli; kokoro uses a dedicated PyTorch->OpenVINO conversion path.', dest='model_type')
+parser_text2speech.add_argument('--model_type', default='kokoro', choices=['speecht5', 'kokoro'], help='Type of the source TTS model. speecht5 uses optimum-cli; kokoro uses a dedicated PyTorch->OpenVINO conversion path.', dest='model_type')


+                    output.success = True
+                    output.latency = time.perf_counter() - st
+                    try:
+                        output.audio_duration = soundfile.info(io.BytesIO(audio_bytes)).duration
+                    except Exception:
+                        output.error = "Could not decode WAV response to compute audio duration"


+pip install -r https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/mkulakow/kokoro_improvements/demos/benchmark/v3/requirements.txt
+curl https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/mkulakow/kokoro_improvements/demos/benchmark/v3/benchmark.py -o benchmark.py
+python benchmark.py --api_url http://localhost:8000/v3/audio/speech --model Kokoro-82M-OpenVINO-FP16-OVMS --batch_size 1 --limit 100 --request_rate inf --backend text2speech --dataset edinburghcstr/ami --hf-subset ihm --voice af_alloy
+Number of documents: 1000


[test_doc_files_linux=demos/audio/README.md]

[test_doc_files_linux=demos/audio/README.md] [test_doc_files_windows=demos/audio/README.md]

Copilot AI review requested due to automatic review settings July 3, 2026 13:56

Copilot started reviewing on behalf of michalkulakowski July 3, 2026 13:56 View session

michalkulakowski force-pushed the mkulakow/kokoro_improvements branch from 666f708 to 80f3156 Compare July 3, 2026 13:59

Copilot AI reviewed Jul 3, 2026

View reviewed changes

michalkulakowski requested a review from Copilot July 6, 2026 11:28

Copilot started reviewing on behalf of michalkulakowski July 6, 2026 11:29 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

dtrawins reviewed Jul 8, 2026

View reviewed changes

michalkulakowski force-pushed the mkulakow/kokoro_improvements branch from e118219 to 9dc3500 Compare July 8, 2026 14:30

dtrawins reviewed Jul 8, 2026

View reviewed changes

michalkulakowski requested a review from Copilot July 9, 2026 12:09

Copilot started reviewing on behalf of michalkulakowski July 9, 2026 12:11 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

michalkulakowski added 10 commits July 9, 2026 14:42

Kokoro usage improvements

05aaa8e

adjust uts

31d1840

update docs

2ba854d

fix

26997dc

fix

9f72de1

Docs fixes

62a67a0

[test_doc_files_linux=demos/audio/README.md]

fixes

01ea5b4

fix

6a3dfe4

TMP

683c50d

[test_doc_files_linux=demos/audio/README.md] [test_doc_files_windows=demos/audio/README.md]

Use Kokoro int8 TTS artifacts in UT model prep

8ed0835

[test_doc_files_linux=demos/audio/README.md] [test_doc_files_windows=demos/audio/README.md]

michalkulakowski force-pushed the mkulakow/kokoro_improvements branch from 14180b8 to 8ed0835 Compare July 9, 2026 12:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Kokoro usage improvements#4357

Kokoro usage improvements#4357
michalkulakowski wants to merge 10 commits into
mainfrom
mkulakow/kokoro_improvements

michalkulakowski commented Jul 3, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

dtrawins Jul 8, 2026

Uh oh!

dtrawins Jul 8, 2026

Uh oh!

dtrawins Jul 8, 2026

Uh oh!

dtrawins Jul 8, 2026

Uh oh!

dtrawins Jul 8, 2026

Uh oh!

dtrawins Jul 8, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	docker run -d -u $(id -u):$(id -g) --rm -p 8000:8000 -v $(pwd)/models:/models:rw openvino/model_server:latest --rest_port 8000 --source_model luis-castillo/Kokoro-82M-OpenVINO-FP16-OVMS --model_repository_path /models --model_name Kokoro-82M-OpenVINO-FP16-OVMS
	mkdir models
	docker run -d -u $(id -u):$(id -g) --rm -p 8000:8000 -v $(pwd)/models:/models:rw openvino/model_server:latest --rest_port 8000 --source_model luis-castillo/Kokoro-82M-OpenVINO-FP16-OVMS --model_repository_path /models --model_name Kokoro-82M-OpenVINO-FP16-OVMS

Uh oh!

Conversation

michalkulakowski commented Jul 3, 2026

🛠 Summary

🧪 Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

dtrawins Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

dtrawins Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

dtrawins Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

dtrawins Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

dtrawins Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

dtrawins Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants