[Fix-18624][Master] Make WorkflowSerialCoordinator restartable after HA failover

[yaodongen]

Was this PR generated or assisted by AI?

YES — the diagnosis, the patch and this description were written with AI assistance (Claude). The root cause was confirmed against a live 3.4.2 cluster: the stack trace, the starting.../started... mismatch and the recovery behaviour below are all taken from that cluster's logs, not inferred.

Purpose of the pull request

Closes #18624.

WorkflowSerialCoordinator is single-use, but the master restarts it on every HA transition. After an ACTIVE -> STAND_BY -> ACTIVE flip within the same JVM, start() throws and serial dispatch dies silently — SERIAL_WAIT / SERIAL_DISCARD / SERIAL_PRIORITY workflows then queue in t_ds_serial_command forever.

close() leaves internalThread non-null:

@Override
public void close() {
    flag = false;
}

while start() refuses to run while it is set:

if (internalThread != null) {
    throw new IllegalStateException("InternalThread is already started");
}

and MasterCoordinator.MasterCoordinatorListener calls both against the same instance:

public void changeToActive()  { taskGroupCoordinator.start(); workflowSerialCoordinator.start(); ... }
public void changeToStandBy() { taskGroupCoordinator.close(); workflowSerialCoordinator.close(); ... }

The sibling TaskGroupCoordinator.close(), invoked from that same listener, already interrupts the thread and nulls the field — which is why task groups survive a failover and serial dispatch does not. This PR makes WorkflowSerialCoordinator.close() match it.

Two details make this silent rather than loud, and are the reason it is worth fixing rather than documenting:

• AbstractHAServer.statusChange assigns serverStatus before notifying listeners and swallows listener exceptions. The master therefore ends up reporting ACTIVE with a dead coordinator while still holding /nodes/master-coordinator, so no peer takes over and nothing self-heals.
• taskGroupCoordinator.start() runs first and succeeds, so the master keeps looking healthy.

The only external symptom is a WorkflowSerialCoordinator starting... log line with no matching started....

Observed on 3.4.2 (2 masters, JDBC registry): workflow instances stuck in SERIAL_WAIT for 6h+, the oldest serial command waiting since 04:00 UTC, starting... logged twice against started... once:

2026-09-08 03:24:57.607 INFO  o.a.d.s.m.e.w.s.WorkflowSerialCoordinator - WorkflowSerialCoordinator starting...
2026-09-08 03:24:57.607 ERROR o.a.d.r.a.h.AbstractHAServer - Trigger ServerStatusChangeListener from STAND_BY -> ACTIVE error
java.lang.IllegalStateException: InternalThread is already started
	at org.apache.dolphinscheduler.server.master.engine.workflow.serial.WorkflowSerialCoordinator.start(WorkflowSerialCoordinator.java:83)
	at org.apache.dolphinscheduler.server.master.engine.MasterCoordinator$MasterCoordinatorListener.changeToActive(MasterCoordinator.java:98)
	at org.apache.dolphinscheduler.registry.api.ha.AbstractServerStatusChangeListener.change(AbstractServerStatusChangeListener.java:33)

Affects every 3.4.x — WorkflowSerialCoordinator was added in 3.4.0 by #17531 (DSIP-92) and close() has not changed since; still present in 3.4.3 and on dev. Single-master deployments cannot hit it, because AbstractHAServer only returns to ACTIVE from a REMOVE event carrying another server's identify, which cannot occur when one master is the sole writer of that path. That is presumably why it has gone unreported.

Brief change log

• WorkflowSerialCoordinator.close() now interrupts internalThread and sets it to null, returns early when already closed, and is synchronized to pair with the already-synchronized start() (internalThread is not volatile). Line-for-line mirror of TaskGroupCoordinator.close().
• Add WorkflowSerialCoordinatorTest covering start -> close -> start, plus start while started and a repeated close. TaskGroupCoordinatorTest.start() covers only start -> close, which is why this asymmetry went unnoticed.

Verify this pull request

Manually verified on a live 3.4.2 cluster:

• Reproduced — restarting both masters together produced the same-JVM ACTIVE -> STAND_BY -> ACTIVE flip and the exception above; serial dispatch stopped while the master still reported ACTIVE.
• Confirmed the fix's premise — bringing the coordinator up with internalThread == null (the state this patch restores on close) starts it cleanly and immediately drains the backlog:

WorkflowSerialCoordinator starting...
WorkflowSerialCoordinator started...
Launched SerialCommand: id=59, workflowInstanceId=69   # queued since 04:00 UTC

Built and tested locally on JDK 11 (one of the two CI targets; pom.xml targets source/target 1.8).

Without the fix the new test reproduces the exact production failure — same exception, same line:

Tests run: 3, Failures: 1, Errors: 0, Skipped: 0
[ERROR] WorkflowSerialCoordinatorTest.startAfterCloseShouldNotThrow <<< FAILURE!
org.opentest4j.AssertionFailedError: Unexpected exception thrown:
    java.lang.IllegalStateException: InternalThread is already started
Caused by: java.lang.IllegalStateException: InternalThread is already started
    at ...WorkflowSerialCoordinator.start(WorkflowSerialCoordinator.java:83)

With the fix:

./mvnw -pl dolphinscheduler-master -am test -Dtest=WorkflowSerialCoordinatorTest
...
[INFO] --- spotless-maven-plugin:2.27.2:check (default) @ dolphinscheduler-master ---
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

spotless:check is bound to the compile phase and was not skipped, so formatting is covered too.

This change is backwards compatible and needs no entry in docs/docs/en/guide/upgrade/incompatible.md.

Pull Request Notice

Pull Request Notice

[boring-cyborg]

Thanks for opening this pull request! Please check out our contributing guidelines. (https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md)

[ruanwenjun]

lgtm

[SbloodyS]

+1

[boring-cyborg]

Awesome work, congrats on your first merged pull request!