Skip to content

[ISSUE #5231#5232#5233] Harden ConnectorRuntime against data loss; split connector-api SPI module; add plugin unit tests (#4642) - #5328

Merged
qqeasonchen merged 2 commits into
apache:developfrom
qqeasonchen:connector/hardening-and-tests
Sep 3, 2026
Merged

[ISSUE #5231#5232#5233] Harden ConnectorRuntime against data loss; split connector-api SPI module; add plugin unit tests (#4642)#5328
qqeasonchen merged 2 commits into
apache:developfrom
qqeasonchen:connector/hardening-and-tests

Conversation

@qqeasonchen

Copy link
Copy Markdown
Contributor

Motivation

Connector plane hardening batch — three P0/P1 items from the connector review (#5296 follow-up):

  1. P0 data-loss hardening (follow-up to [Bug] Using the HTTP connector, data cannot be saved to RocketMQ after being pushed. #5231 / [Bug] Using http connector in common protocol,data cant be saved in the topic #5232 / [Bug] http connector in common protocol has a json deserialization error #5233): those three bugs crashed the old openconnect framework (already removed by the uni-arch redesign); the crash classes (SourceWorker, RecordOffsetManagement, CommonProtocol) no longer exist. But the failure mode — one bad event killing the loop and silently stranding the rest of the batch — could still happen in the new ConnectorRuntime. This PR closes that class of bugs:

    • runSourceOnce(): per-event isolation — a publish that throws (bad CloudEvent, broken URL) is logged + skipped + counted; the rest of the batch still publishes; null events in a poll() batch are skipped, not NPE.
    • runSinkOnce(): per-delivery ACK isolation — one ACK failure no longer loses the remaining ACKs/offset; EventMesh re-delivers.
    • offsetStore.put() failures no longer abort the publish/commit path (checkpoint lost is logged, event flow continues).
    • getSourcePublishFailures() counter exposes skipped events to the admin API.
  2. P1 connector-api module split (plan doc included at docs/contrib/connector-api-split-plan.md): the 7 SPI interfaces (SourceConnector, SinkConnector, EventMeshEndpoint, HttpCaller, ConnectorOffsetStore, CloudEventSerializer, PollEntry) move from eventmesh-connector-runtime into a new :eventmesh-connector-api module (deps: cloudevents-core only). The package name stays org.apache.eventmesh.connector, so all 23 plugins keep their imports — only their build.gradle dependency swaps. Enforcement is two-layer:

  3. P0 plugin unit tests ([Unit Test] Add unit test for eventmesh-connector-file module #4642 follow-up): first tests ever for connector-file (5+5), connector-kafka (1, Kafka MockProducer), connector-pulsar (2, smoke), connector-rocketmq (3, smoke). File connectors gain close*Quietly() so tests release file handles on Windows.

Also: eventmesh.properties gains a commented Connector Runtime section; dist-connector packaging ships the api jar in apps/ next to the runtime jar.

Modifications

  • eventmesh-connector-api/ — new module: 7 SPI interfaces + package-info
  • eventmesh-connector-runtime/ — SPI files removed; api project(':eventmesh-connector-api') dependency; ConnectorRuntime hardening + 4 new tests
  • 23 × eventmesh-connector-plugin/*/build.gradle — dep swap runtime → api
  • eventmesh-architecture-guard/ — new rule + canary test (11/11 green)
  • build.gradle (root) — dist-connector ships api jar
  • eventmesh-runtime/conf/eventmesh.properties — connector section
  • eventmesh-connector-plugin/eventmesh-connector-{file,kafka,pulsar,rocketmq}/src/test/ — new tests

Documentation

  • docs/contrib/connector-api-split-plan.md — split rationale, migration steps (M1/M2/M3), risks, acceptance criteria

Verifying this patch

Local (JDK 21, Windows): :eventmesh-connector-api:build, :eventmesh-connector-runtime:build, :connector-{file,kafka,pulsar,rocketmq}:build (incl. checkstyle), :eventmesh-runtime:compileJava, :eventmesh-architecture-guard:test 11/11 — all green. 33 new/updated unit tests total (17 runtime + 16 plugin).

…ss, split connector-api SPI module, add plugin unit tests

P0 — data-loss hardening (follow-up to apache#5231/apache#5232/apache#5233):
- runSourceOnce(): per-event isolation — one bad event (publish throws) no
  longer kills the source loop or strands the rest of the batch; null events
  in a poll() batch are skipped and counted
- runSinkOnce(): per-delivery ACK isolation — one ACK failure no longer loses
  the offsets/ACKs of the remaining deliveries; EventMesh re-delivers
- offsetStore.put failures no longer abort the publish/commit path
- new getSourcePublishFailures() counter surfaces skipped events to the admin API
- 4 new unit tests covering all three hardening scenarios

P1 — connector-api module split (plan: docs/contrib/connector-api-split-plan.md):
- new :eventmesh-connector-api module with the 7 SPI interfaces
  (SourceConnector, SinkConnector, EventMeshEndpoint, HttpCaller,
  ConnectorOffsetStore, CloudEventSerializer, PollEntry) — package name kept
  as org.apache.eventmesh.connector so the 23 plugins need zero code changes
- all 23 plugin build.gradles now depend on :eventmesh-connector-api instead
  of :eventmesh-connector-runtime; compile-time enforcement in addition to
  the ArchUnit guard
- new ArchUnit rule ruleConnectorPluginsDependOnlyOnSpi (FAIL mode) + canary
  negative test proving the rule actually fires
- dist-connector packaging ships the api jar in apps/ next to the runtime jar

P1 — configuration:
- eventmesh.properties gains a commented Connector Runtime section

P0 — plugin unit tests (issue apache#4642 follow-up):
- file: FileSourceConnectorTest (5) + FileSinkConnectorTest (5)
- kafka: KafkaSinkConnectorTest (1, MockProducer)
- pulsar: PulsarSinkConnectorTest (2, smoke)
- rocketmq: RocketmqSinkConnectorTest (3, smoke)
- file connectors gain close()Quietly() to release file handles (Windows)

Verified: :eventmesh-connector-api:build, :eventmesh-connector-runtime:build,
:connector-{file,kafka,pulsar,rocketmq}:build, :eventmesh-runtime:compileJava,
:eventmesh-architecture-guard:test (11/11) all green locally.
@qqeasonchen
qqeasonchen merged commit 34a97bf into apache:develop Sep 3, 2026
8 checks passed
qqeasonchen added a commit that referenced this pull request Sep 3, 2026
… doc with #5328 (#5329)

The capability status table marked Connector Runtime as Beta, but only 4 of
23 plugins (file/kafka/pulsar/rocketmq) carry unit tests; the rest are
template implementations. Downgrade to Experimental and point at what has
actually landed: data-loss hardening + SPI split into eventmesh-connector-api
(#5328).

- README.md / README.zh-CN.md: status row updated
- docs/eventmesh-features.md §8: status paragraph + SPI module path fixed
  (top-level eventmesh-connector-api/, not under connector-plugin)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant