Skip to content

feat(asr): add rtzr streaming extension - #2320

Open
kimdwkimdw wants to merge 4 commits into
TEN-framework:mainfrom
kimdwkimdw:feature/rtzr
Open

kimdwkimdw wants to merge 4 commits into
TEN-framework:mainfrom
kimdwkimdw:feature/rtzr

Conversation

@kimdwkimdw

Copy link
Copy Markdown

Summary

Add rtzr_asr_python to make RTZR streaming speech recognition available through TEN's standard ASR interface. The extension follows AsyncASRBaseExtension and uses aiohttp directly, without an additional vendor SDK dependency.

  • Support sommers_ko, sommers_ja, and sommers_en, mapping interim/final transcripts, timestamps, and word timing to TEN results.
  • Implement authentication and token reuse, same-connection Finalize, EOS shutdown, bounded audio buffering and retries, connection status, metrics, and optional audio dumping.
  • Configure credentials and endpoints through RTZR_CLIENT_ID, RTZR_CLIENT_SECRET, RTZR_API_BASE, and RTZR_WEBSOCKET_URL. Default endpoints are https://openapi.vito.ai and wss://openapi.vito.ai.
  • Add standalone tests, opt-in live tests, test configurations, README setup/graph examples, and registration for the common ASR guarder's connection-status tests.

Type of Change

  • Bug fix
  • New feature
  • Documentation update
  • Performance improvement
  • Code refactoring

Testing

  • Tests added/updated
  • All repository tests pass — the full repository suite was not run
  • Manual testing completed against an existing RTZR development service
Check Result
Final standalone offline suite, Python 3.10 / pytest 8 39 passed
Final common ASR guarder 11 passed, 2 deselected
Final Japanese live recognition with repeated Finalize Passed with 3 CSJ utterances
Three-model continuous streaming 300 seconds per model, no errors; run before the final malformed-response validation change
Direct API vs. TEN transcripts Normalized transcripts matched for the Korean, Japanese, and English samples, including the CSJ follow-up
Ruff / Black / Pylint Passed; Pylint 10.00/10
TEN manifest/property schemas and package creation Passed

The common guarder's Chinese-specific multilingual case was replaced by model-specific Korean, Japanese, and English live tests. Its long-duration case was replaced by the 300-second-per-model tests. These live checks use small samples and do not establish corpus-wide recognition quality.

Live tests are opt-in and require credentials and an external audio manifest. Credentials, evaluation audio, reference transcripts, and local experiment artifacts are not included in this change. No upstream CI result is claimed.

Documentation

  • Documentation updated
  • Examples provided if needed

The extension README covers dependencies, environment/property configuration, graph wiring, lifecycle behavior, and test commands. The shared .env.example includes the RTZR variables.

Breaking Changes

None. This is an opt-in extension and does not change existing ASR implementations or application graphs.

Protocol limitations

  • RTZR does not send a separate Finalize acknowledgement for silence following an automatic final. When no hypothesis remains pending, the extension reports local completion after a bounded wait with a finalize_without_result metric; it does not fabricate a vendor final result.
  • Audio already sent before a disconnect is not replayed because the streaming protocol provides no per-frame acknowledgement/resume mechanism.

@diyuyi-agora diyuyi-agora left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the connection, buffering, finalize, protocol-mapping, packaging, and test paths. The offline suite passes (39 tests), and Black/Pylint are clean. I found three issues that should be addressed before merge.

Comment thread ai_agents/agents/ten_packages/extension/rtzr_asr_python/extension.py Outdated
Comment thread ai_agents/agents/ten_packages/extension/rtzr_asr_python/config.py Outdated
kimdwkimdw added a commit to kimdwkimdw/ten-framework that referenced this pull request Sep 14, 2026
kimdwkimdw added a commit to kimdwkimdw/ten-framework that referenced this pull request Sep 14, 2026
@kimdwkimdw

Copy link
Copy Markdown
Author

Addressed the three review findings:

  1. Finalize boundary — Fixed in b94f230. Completion now checks the final result's end timestamp and pending hypothesis state instead of treating all bytes sent as finalized. Added regression coverage for delayed automatic finals followed by trailing audio.

  2. WebSocket 401 recovery — Fixed in 431594f. A WebSocket handshake 401 now invalidates the token and retries once with a freshly issued token. Authentication-endpoint failures and repeated handshake 401s remain terminal. Added tests for revoked-token recovery and bounded retries.

  3. English model — Fixed in 431594f. Removed the previous English model from the implementation, test configurations, and documentation. English now uses model_name=whisper with language=en, and reports en-US in TEN results.

diyuyi-agora
diyuyi-agora previously approved these changes Sep 15, 2026
@kimdwkimdw

kimdwkimdw commented Sep 16, 2026

Copy link
Copy Markdown
Author

@kimdwkimdw

Copy link
Copy Markdown
Author

rebased from latest main branch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants