feat(compiler): add dart gRPC codegen by yash-agarwa-l · Pull Request #3723 · apache/fory

yash-agarwa-l · 2026-05-31T13:44:52Z

Why?

Dart users need gRPC support from FDL/proto/fbs schemas

What does this PR do?

Adds DartServiceGeneratorMixin so foryc --grpc --dart_out=… emits a /_grpc.dart next to the messages file. Includes:

Client over package:grpc's Client with one method per RPC, covering unary and all three streaming modes (server-stream, client-stream, bidi).
abstract ServiceBase over Service with $addMethod registrations and a _Pre shim per method.
Top-level _serialize / _deserialize helpers routing through the schema module's getFory(), with reference tracking on so payloads round-trip with the Java/Python peers.
Works from FDL, Protobuf, and FlatBuffers IDL service definitions.
Class- and method-name collision detection.
Opt-in dart analyze + dart format smoke test on the emitted file.
A Java↔Dart gRPC interop test exercising all four modes in both directions.

Related issues

Part of #3266
Part of #3279
Closes #3266

AI Contribution Checklist

Substantial AI assistance was used in this PR: yes / no
If yes, I included a completed AI Contribution Checklist in this PR description and the required AI Usage Disclosure.
If yes, my PR description includes the required ai_review summary and screenshot evidence of the final clean AI review results from both fresh reviewers on the current PR diff or current HEAD after the latest code changes.

Does this PR introduce any user-facing change?

Does this PR introduce any public API change?
Does this PR introduce any binary protocol compatibility change?
New _grpc.dart companion when --grpc --dart_out is used on a schema with services.

Benchmark

Wire DartServiceGeneratorMixin into DartGenerator via MRO matching JavaGenerator and PythonGenerator. The mixin currently exposes only the streaming-rejection check, which raises ValueError listing every offending method when an FDL service contains stream RPCs. File emission and Client/ServiceBase generation land in follow-up commits.

Add generate_grpc_module to DartServiceGeneratorMixin: emits a <stem>_grpc.dart companion next to the messages file with Apache header, dart:async/dart:typed_data/package:grpc imports, a sibling import of the messages module as _models, and the two top-level codec helpers (_serialize<T> and _deserialize<T>) that route through the existing ForyRegistration.getFory() static. _deserialize includes the is-Uint8List guard for zero-copy on the common HTTP/2 path.

Emit per-service Client with ClientMethod constants and $createUnaryCall-backed unary methods. Constructor uses super-parameter syntax confirmed by grpc 4.3.1's in-tree TestClient(super.channel). Method names are normalized through safe_identifier(to_camel_case(method.name)).

Emit <Service>ServiceBase: registers each unary RPC via $addMethod in the constructor and exposes an abstract handler per method. Uses a _Pre shim that awaits the request Future before dispatching, since grpc-dart's ServiceMethod always hands the handler a Future<Q>

Add check_dart_grpc_service_collisions to the mixin. For each local service, compute its emitted Client and ServiceBase class names and check against the set of top-level names DartGenerator would already emit for messages/enums/unions/registration. Also detect service<->service collisions when two services in the same schema would emit the same Client/ServiceBase name. Raises ValueError naming the offending class; the CLI's existing ValueError wrap at cli.py:686-690 prints it and exits non-zero.

Add check_dart_grpc_method_collisions. For each service, normalize every method name through safe_identifier(to_camel_case(name)) and raise ValueError if two distinct RpcMethod names collapse to the same Dart identifier (e.g. SayHello and say_hello both produce sayHello).

Mirror Python's test_proto_and_fbs_grpc_service_codegen: one test exercises a .proto-sourced service through ProtoTranslator and a .fbs-sourced service through FbsTranslator, then asserts both end up in the Dart gRPC emitter with the right method paths and Client classes. Confirms no Dart-specific translator wiring was needed.

Register DartGenerator in GENERATOR_CLASSES and add it to the gRPC-supported skip in the unsupported-generators test.

Emit _grpc.dart under the same package directory as the messages file. Add opt-in dart analyze gate covering the emitted file.

dart format reflowed the ClientMethod constant and $addMethod call; align emitter strings with format output and add a dart format gate to the smoke test.

chaokunyang · 2026-06-02T03:29:11Z

Could you add java-dart grpc integration tests? You can take #3692 as reference

# Conflicts: # compiler/fory_compiler/tests/test_service_codegen.py

Generate server-stream, client-stream, and bidi methods alongside unary, and serialize with ref tracking so payloads interop with the Java and Python peers.

Add a Dart interop peer (integration_tests/grpc_tests/dart) mirroring the Python peer's FDL transforms with client and server subcommands, driven both directions by DartGrpcInteropTest over the shared GrpcTestBase via new dartCommand/runDart helpers. Generate Dart stubs in generate_grpc.py, build the peer (dart pub get + build_runner) in run_tests.sh, and add a grpc_java_dart_tests CI job.

yash-agarwa-l · 2026-06-15T04:57:04Z

Thanks for the review, I have add all the remaining rpc methods with the integration tests

chaokunyang · 2026-06-15T05:29:06Z

Please take #3761 as reference to add comprehensive docs

yash-agarwa-l · 2026-06-15T15:25:13Z

Thanks, I have updated the docs!

chaokunyang · 2026-06-16T06:10:43Z

+            "no separate registration helper needed."
+        )
+        lines.append("")
+        fory = f"_models.{self.module_type_name()}.getFory()"


This makes the generated gRPC path depend on callers having already run <Module>ForyModule.install(Fory()); otherwise getFory() throws before the first RPC. Generated gRPC services should not require manual generated-type registration, and the other language companions hide this behind a module-owned ready runtime. Please make the Dart companion construct or otherwise obtain a ready Fory instance for generated gRPC, keeping custom runtime injection optional rather than required.

chaokunyang · 2026-06-16T06:10:45Z

+        lines.append(f"class {service.name}Client extends Client {{")
+        for method in service.methods:
+            method_const = f"_${self.dart_grpc_method_name(method)}"
+            req_t = f"_models.{method.request_type.name}"


This raw AST name only works for top-level local message types. For an imported payload such as demo.common.SharedReq, the companion emits _models.demo.common.SharedReq; for a nested payload such as Envelope.Request, it emits _models.Envelope.Request, while Dart model code emits/imports flattened or alias-qualified symbols like Envelope_Request. Please resolve RPC payloads through the Dart generator's emitted type-name path (resolve_type/ref_name) and add analyzer coverage for imported and nested RPC payloads.

chaokunyang · 2026-06-16T06:10:46Z

+| `GreeterServiceBase` in `demo_greeter_grpc.dart` | Base class for server implementations                |
+| `GreeterClient` in `demo_greeter_grpc.dart`      | Client stub for gRPC calls                           |
+
+## Register Types


This section documents a setup step that generated gRPC users should not need. The gRPC companion should own or obtain a ready Fory instance and register generated service payload types automatically, consistent with the other language gRPC generators. Once the generator handles that, please remove this manual registration section and the install(Fory()) calls from the server/client examples below.

chaokunyang · 2026-06-16T06:10:48Z

+
+A single-response client method returns `ResponseFuture<R>` (client-streaming
+adapts the streaming call with `.single`); a streaming-response method returns
+`ResponseStream<R>`. On the server, single requests arrive as `Future<Q>` and


This describes the internal _Pre shim, not the public service override. The generated abstract methods receive a concrete Q for single-request RPCs and Stream<Q> for client-streaming RPCs. Please adjust this sentence so users do not implement server methods with Future<Q> request parameters.

chaokunyang · 2026-06-16T06:31:35Z

+
+| File                                             | Purpose                                              |
+| ------------------------------------------------ | ---------------------------------------------------- |
+| `demo/greeter/demo_greeter.dart`                 | Fory model types and the schema module               |


These generated names do not match the current Dart generator for the command above. With source service.fdl and package demo.greeter, _module_file_name_for_schema uses the package leaf and emits demo/greeter/greeter.dart, greeter.fory.dart, greeter_grpc.dart, and GreeterForyModule, not demo_greeter.* / DemoGreeterForyModule. Please either update the docs/examples to match the current output or change the generator if source-stem-owned module names are the intended design.

chaokunyang · 2026-06-16T06:31:44Z

+          python-version: 3.11
+          cache: "pip"
+      - name: Set up Dart
+        uses: dart-lang/setup-dart@v1


This adds an unpinned third-party action to ASF CI. The existing Dart release workflow pins dart-lang/setup-dart to a commit SHA (e51d8... # v1.7.1), so this job should use a pinned SHA or the same approved setup path instead of @v1 to avoid mutable action resolution and ASF action-policy issues.

chaokunyang · 2026-06-16T06:32:14Z

+    assert "import 'demo_greeter.dart' as _models;" in content
+
+    assert (
+        "_models.DemoGreeterForyModule.getFory().serialize(value, trackRef: true)"


These assertions pin the same manual DemoGreeterForyModule.getFory() ownership path that the generated gRPC companion should avoid. Once the generator owns or obtains a ready Fory runtime automatically, this test needs to assert that new generated path rather than preserving the startup-only install(Fory()) dependency.

chaokunyang · 2026-06-16T07:01:39Z

+# specific language governing permissions and limitations
+# under the License.
+
+"""dart analyze smoke test for emitted Dart gRPC files.


This test should not live under compiler pytest. It shells out to dart pub get, dart format, and dart analyze, and it validates the generated Dart gRPC companion rather than compiler-only behavior. The owner should be the Dart gRPC integration test path under integration_tests/grpc_tests/dart (or the Java-driven gRPC integration harness that already installs Dart), so CI exercises it with the Dart toolchain instead of adding a compiler test that skips when Dart is absent. Please move this validation there and remove this compiler-level test file.

chaokunyang · 2026-06-16T07:02:42Z

+
+  @Test
+  public void testJavaServerDartClient() throws Exception {
+    Server server = startJavaFdlServer();


This only exercises the FDL service path, but the PR documents and tests Dart gRPC as supporting Fory IDL, protobuf IDL, and FlatBuffers IDL. Building the generated proto/FBS Dart files is not enough for that feature surface; the Dart client/server interop needs to exercise all claimed schema inputs, like the Python/Rust/Kotlin gRPC tests do with exerciseAllSchemas, or the public docs/tests need to narrow the claim. As it stands, Dart gRPC support is only partially implemented and leaves the protobuf/FBS runtime paths unproven.

chaokunyang · 2026-06-16T07:04:08Z

 - Dart xlang or runtime ownership changes need local Dart package tests plus the Java-driven `DartXlangTest`; package-only smoke tests are not enough.
 - When claiming non-VM Dart support, prove a relevant non-VM compile path such as `dart compile js` against active runtime or example code.
+- Generated Dart gRPC service companions (`<stem>_grpc.dart`) are compiler-owned files that depend on the application-provided `grpc` package, not `dart/packages/fory`. Keep gRPC dependencies out of the Fory Dart runtime package.
+- Dart generated schema modules (`<Stem>ForyModule`) are the source-file owners. Service companions must resolve their `Fory` through that module's `getFory()` and must not introduce package-derived aliases or duplicate serializer registration paths.


This guidance encodes the same owner model that the implementation should move away from. If generated Dart gRPC is meant to be complete, the companion should own or obtain a ready Fory runtime for service payloads; putting getFory() as a required rule in .agents will keep future changes preserving the manual-install dependency. Please update or remove this rule along with the generator/docs so the design source does not stay stale.

chaokunyang · 2026-06-16T07:04:08Z

+}
+
+Future<void> main() async {
+  DemoGreeterForyModule.install(Fory());


These server/client snippets call Fory() but do not import package:fory/fory.dart, so they do not compile as written while the manual install remains. The preferred fix is still to remove the manual install requirement; if any Fory() construction remains in the docs, the examples need complete imports.

chaokunyang · 2026-06-16T07:04:16Z

+**Fix**: Add `grpc` to your `pubspec.yaml` (and the `build_runner` dev
+dependency), then run `dart pub get`. See [gRPC Support](grpc-support.md).
+
+## `getFory()` throws before the first gRPC call


This troubleshooting entry turns the current startup failure into documented behavior. A first gRPC call should not fail because the user did not call <Schema>ForyModule.install(fory); generated service code should be ready by construction or through explicit custom injection. Please remove or rewrite this entry when the generator ownership is fixed.

chaokunyang · 2026-06-16T07:04:17Z

+When a schema contains services and the compiler is run with `--grpc`, Dart
+generation emits one `<module>_grpc.dart` file per schema next to the model
+types. It targets `package:grpc` and serializes each request and response with
+the schema module's `getFory()` instance (for example


This compiler-level docs section repeats the manual getFory() runtime ownership model. Once Dart gRPC owns or obtains a ready runtime automatically, this should describe that public workflow instead of telling users service serialization goes through a preinstalled schema module. Otherwise the generated-code docs will remain inconsistent with the service design.

chaokunyang · 2026-06-16T07:04:18Z

+                    )
+                service_names.add(emitted)
+
+    def check_dart_grpc_method_collisions(self, services: List[Service]) -> None:


This only checks method-vs-method collisions after camelCasing. The generated members also sit on classes extending Client and Service, and every Dart object inherits Object, so valid RPC names like ToString or HashCode generate invalid overrides or conflicting members (toString(...), hashCode(...)). Please reserve those inherited/member surfaces or fail codegen with a clear diagnostic, and add analyzer coverage for those names.

chaokunyang · 2026-06-17T06:04:36Z

@yash-agarwa-l Please merge apache/main first, we updated the docs and ci tests

# Conflicts: # docs/compiler/compiler-guide.md # docs/compiler/flatbuffers-idl.md # docs/compiler/index.md # docs/compiler/protobuf-idl.md # docs/compiler/schema-idl.md # integration_tests/grpc_tests/run_tests.sh

…lution and reserved-name guard

…y runtime

…est, pin CI dart action, move analyze coverage to integration

yash-agarwa-l added 11 commits May 30, 2026 19:13

test(compiler): include dart in parametric service-codegen tests

93b2e5e

Register DartGenerator in GENERATOR_CLASSES and add it to the gRPC-supported skip in the unsupported-generators test.

test(compiler): add opt-in dart analyze smoke test

71c3d9e

Emit _grpc.dart under the same package directory as the messages file. Add opt-in dart analyze gate covering the emitted file.

fix(compiler): emit dart-format-clean dart grpc output

34fd564

dart format reflowed the ClientMethod constant and $addMethod call; align emitter strings with format output and add a dart format gate to the smoke test.

style(compiler): ruff format dart grpc files

2f7d69e

chaokunyang self-requested a review June 2, 2026 03:29

yash-agarwa-l added 4 commits June 15, 2026 09:50

Merge remote-tracking branch 'upstream/main' into grpc-dart

2defb2a

# Conflicts: # compiler/fory_compiler/tests/test_service_codegen.py

feat(compiler): emit dart grpc streaming methods

755bd6e

Generate server-stream, client-stream, and bidi methods alongside unary, and serialize with ref tracking so payloads interop with the Java and Python peers.

style(compiler): ruff format test service file

e75e61f

yash-agarwa-l requested review from PragmaTwice and theweipeng as code owners June 15, 2026 04:40

yash-agarwa-l changed the title ~~feat(compiler): add dart unary gRPC codegen~~ feat(compiler): add dart gRPC codegen Jun 15, 2026

docs: document dart grpc support

f6880f4

chaokunyang reviewed Jun 16, 2026

View reviewed changes

yash-agarwa-l added 4 commits June 17, 2026 23:09

Merge remote-tracking branch 'upstream/main' into grpc-dart

d7a591b

# Conflicts: # docs/compiler/compiler-guide.md # docs/compiler/flatbuffers-idl.md # docs/compiler/index.md # docs/compiler/protobuf-idl.md # docs/compiler/schema-idl.md # integration_tests/grpc_tests/run_tests.sh

feat(compiler): make dart grpc self-sufficient with payload type reso…

d33b55d

…lution and reserved-name guard

docs: align dart grpc docs and agent guidance with the auto-ready For…

0025c4a

…y runtime

test(grpc): exercise all schemas in dart interop, rename to DartGrpcT…

b602021

…est, pin CI dart action, move analyze coverage to integration

yash-agarwa-l force-pushed the grpc-dart branch from 77f7e4d to b602021 Compare June 18, 2026 06:24

Conversation

yash-agarwa-l commented May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why?

What does this PR do?

Related issues

AI Contribution Checklist

Does this PR introduce any user-facing change?

Benchmark

Uh oh!

chaokunyang commented Jun 2, 2026

Uh oh!

yash-agarwa-l commented Jun 15, 2026

Uh oh!

chaokunyang commented Jun 15, 2026

Uh oh!

yash-agarwa-l commented Jun 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chaokunyang commented Jun 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yash-agarwa-l commented May 31, 2026 •

edited

Loading