[ISSUE #5231#5232#5233] Harden ConnectorRuntime against data loss; split connector-api SPI module; add plugin unit tests (#4642) - #5328
Merged
qqeasonchen merged 2 commits intoSep 3, 2026
Conversation
…ss, split connector-api SPI module, add plugin unit tests P0 — data-loss hardening (follow-up to apache#5231/apache#5232/apache#5233): - runSourceOnce(): per-event isolation — one bad event (publish throws) no longer kills the source loop or strands the rest of the batch; null events in a poll() batch are skipped and counted - runSinkOnce(): per-delivery ACK isolation — one ACK failure no longer loses the offsets/ACKs of the remaining deliveries; EventMesh re-delivers - offsetStore.put failures no longer abort the publish/commit path - new getSourcePublishFailures() counter surfaces skipped events to the admin API - 4 new unit tests covering all three hardening scenarios P1 — connector-api module split (plan: docs/contrib/connector-api-split-plan.md): - new :eventmesh-connector-api module with the 7 SPI interfaces (SourceConnector, SinkConnector, EventMeshEndpoint, HttpCaller, ConnectorOffsetStore, CloudEventSerializer, PollEntry) — package name kept as org.apache.eventmesh.connector so the 23 plugins need zero code changes - all 23 plugin build.gradles now depend on :eventmesh-connector-api instead of :eventmesh-connector-runtime; compile-time enforcement in addition to the ArchUnit guard - new ArchUnit rule ruleConnectorPluginsDependOnlyOnSpi (FAIL mode) + canary negative test proving the rule actually fires - dist-connector packaging ships the api jar in apps/ next to the runtime jar P1 — configuration: - eventmesh.properties gains a commented Connector Runtime section P0 — plugin unit tests (issue apache#4642 follow-up): - file: FileSourceConnectorTest (5) + FileSinkConnectorTest (5) - kafka: KafkaSinkConnectorTest (1, MockProducer) - pulsar: PulsarSinkConnectorTest (2, smoke) - rocketmq: RocketmqSinkConnectorTest (3, smoke) - file connectors gain close()Quietly() to release file handles (Windows) Verified: :eventmesh-connector-api:build, :eventmesh-connector-runtime:build, :connector-{file,kafka,pulsar,rocketmq}:build, :eventmesh-runtime:compileJava, :eventmesh-architecture-guard:test (11/11) all green locally.
This was referenced Sep 3, 2026
…yle + skywalking-eyes)
qqeasonchen
added a commit
that referenced
this pull request
Sep 3, 2026
… doc with #5328 (#5329) The capability status table marked Connector Runtime as Beta, but only 4 of 23 plugins (file/kafka/pulsar/rocketmq) carry unit tests; the rest are template implementations. Downgrade to Experimental and point at what has actually landed: data-loss hardening + SPI split into eventmesh-connector-api (#5328). - README.md / README.zh-CN.md: status row updated - docs/eventmesh-features.md §8: status paragraph + SPI module path fixed (top-level eventmesh-connector-api/, not under connector-plugin)
This was referenced Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Connector plane hardening batch — three P0/P1 items from the connector review (#5296 follow-up):
P0 data-loss hardening (follow-up to [Bug] Using the HTTP connector, data cannot be saved to RocketMQ after being pushed. #5231 / [Bug] Using http connector in common protocol,data cant be saved in the topic #5232 / [Bug] http connector in common protocol has a json deserialization error #5233): those three bugs crashed the old
openconnectframework (already removed by the uni-arch redesign); the crash classes (SourceWorker,RecordOffsetManagement,CommonProtocol) no longer exist. But the failure mode — one bad event killing the loop and silently stranding the rest of the batch — could still happen in the newConnectorRuntime. This PR closes that class of bugs:runSourceOnce(): per-event isolation — a publish that throws (bad CloudEvent, broken URL) is logged + skipped + counted; the rest of the batch still publishes;nullevents in apoll()batch are skipped, not NPE.runSinkOnce(): per-delivery ACK isolation — one ACK failure no longer loses the remaining ACKs/offset; EventMesh re-delivers.offsetStore.put()failures no longer abort the publish/commit path (checkpoint lost is logged, event flow continues).getSourcePublishFailures()counter exposes skipped events to the admin API.P1 connector-api module split (plan doc included at
docs/contrib/connector-api-split-plan.md): the 7 SPI interfaces (SourceConnector,SinkConnector,EventMeshEndpoint,HttpCaller,ConnectorOffsetStore,CloudEventSerializer,PollEntry) move fromeventmesh-connector-runtimeinto a new:eventmesh-connector-apimodule (deps: cloudevents-core only). The package name staysorg.apache.eventmesh.connector, so all 23 plugins keep their imports — only theirbuild.gradledependency swaps. Enforcement is two-layer:ruleConnectorPluginsDependOnlyOnSpi(FAIL mode, extends [Architecture Review][P2] Gradle dependency guardrails (architectureCheck) #5305 guard) + a canary negative test proving the rule fires.P0 plugin unit tests ([Unit Test] Add unit test for eventmesh-connector-file module #4642 follow-up): first tests ever for
connector-file(5+5),connector-kafka(1, KafkaMockProducer),connector-pulsar(2, smoke),connector-rocketmq(3, smoke). File connectors gainclose*Quietly()so tests release file handles on Windows.Also:
eventmesh.propertiesgains a commented Connector Runtime section;dist-connectorpackaging ships the api jar inapps/next to the runtime jar.Modifications
eventmesh-connector-api/— new module: 7 SPI interfaces + package-infoeventmesh-connector-runtime/— SPI files removed;api project(':eventmesh-connector-api')dependency;ConnectorRuntimehardening + 4 new testseventmesh-connector-plugin/*/build.gradle— dep swap runtime → apieventmesh-architecture-guard/— new rule + canary test (11/11 green)build.gradle(root) — dist-connector ships api jareventmesh-runtime/conf/eventmesh.properties— connector sectioneventmesh-connector-plugin/eventmesh-connector-{file,kafka,pulsar,rocketmq}/src/test/— new testsDocumentation
docs/contrib/connector-api-split-plan.md— split rationale, migration steps (M1/M2/M3), risks, acceptance criteriaVerifying this patch
Local (JDK 21, Windows):
:eventmesh-connector-api:build,:eventmesh-connector-runtime:build,:connector-{file,kafka,pulsar,rocketmq}:build(incl. checkstyle),:eventmesh-runtime:compileJava,:eventmesh-architecture-guard:test11/11 — all green. 33 new/updated unit tests total (17 runtime + 16 plugin).