Skip to content

[Rust] Add multiplexed stream builder - #818

Open
danilonajkov-db wants to merge 3 commits into
mainfrom
effort/mux-streams-streambuilder
Open

danilonajkov-db wants to merge 3 commits into
mainfrom
effort/mux-streams-streambuilder

Conversation

@danilonajkov-db

@danilonajkov-db danilonajkov-db commented Sep 1, 2026

Copy link
Copy Markdown
Member

Summary

Builds on the lifecycle hardening merged in #826.

  • Add StreamBuilder::multiplexed(n) (Beta) for JSON, compiled protobuf, and dynamic protobuf, available with default features.
  • Open 1–64 homogeneous lanes concurrently with stable indices and bounded startup jitter (none for one lane). Divide the configured in-flight budget across lanes.
  • Acquire each lane's gRPC client through the SDK helper, so connection_per_stream(true) creates dedicated transports and false retains shared-channel behavior.
  • Preserve ordinary ack_callback(...) and provide a separate multiplexed_ack_callback(...) using MessageId. Validate callback/format mismatches before opening connections.
  • Preserve lane-specific waits, all-lane flush, and close semantics from [Rust] Harden multiplexed stream lifecycle #826. Keep the existing testing-feature MultiplexedStream::new constructor.
  • Guard cancelled construction and abort owned sender/receiver tasks when their supervisor is dropped. Ordinary Avro and Arrow telemetry support remains intact; mux formats are limited to JSON/protobuf.
  • Document usage, ordering, capacity, lifecycle, and callback behavior, with a complete compiled-protobuf example.

Use multiplexing when one gRPC stream limits throughput and global ordering is unnecessary.

Validation

  • make test — workspace tests, all-feature/default-feature SDK tests, doctests, and all-feature JNI tests
  • make lint — full repository Clippy matrix with warnings denied
  • cargo check --offline --locked -p rust-examples-proto --example proto_compiled_multiplexed
  • cargo fmt --all --check and git diff --check
  • Locked dependency resolution for Python and TypeScript; the builder reuses the repository's existing rand 0.9 resolution

Focused regressions cover callback-mode validation, lane-count/capacity validation, Avro rejection, cancelled construction, and dedicated versus shared mux connections.

@danilonajkov-db
danilonajkov-db force-pushed the effort/mux-streams-streambuilder branch from c901de4 to d121baa Compare September 2, 2026 13:23
@danilonajkov-db
danilonajkov-db changed the base branch from main to effort/mux-stream-lifecycle-hardening September 17, 2026 10:31
Base automatically changed from effort/mux-stream-lifecycle-hardening to main September 17, 2026 12:30
@danilonajkov-db
danilonajkov-db force-pushed the effort/mux-streams-streambuilder branch 2 times, most recently from f8b6aef to 04ae050 Compare September 17, 2026 14:56
@danilonajkov-db
danilonajkov-db marked this pull request as ready for review September 17, 2026 14:56
Signed-off-by: danilo-najkov-db <danilo.najkov@databricks.com>
@danilonajkov-db
danilonajkov-db force-pushed the effort/mux-streams-streambuilder branch 2 times, most recently from b3a21ad to ee00d55 Compare September 17, 2026 16:30
Signed-off-by: danilo-najkov-db <danilo.najkov@databricks.com>
@danilonajkov-db
danilonajkov-db force-pushed the effort/mux-streams-streambuilder branch from ee00d55 to bf0ea41 Compare September 17, 2026 16:37
@danilonajkov-db
danilonajkov-db requested a review from a team September 17, 2026 16:38
Signed-off-by: danilo-najkov-db <danilo.najkov@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant