[Improvement-18611][Task] Support configurable TCP keepalive on the H… - #18612
[Improvement-18611][Task] Support configurable TCP keepalive on the H…#18612Admaing wants to merge 9 commits into
Conversation
…TTP task socket The HTTP task (OkHttp based on dev) opens sockets with SO_KEEPALIVE disabled. Behind a stateful firewall / security group (conntrack) that evicts idle TCP sessions, a long-running request whose response is delayed for longer than the idle-eviction window gets its session silently dropped, so the task fails with Read timed out. curl succeeds because libcurl enables TCP keepalive by default and keeps refreshing the conntrack entry. Add a per-task boolean 'socketKeepAlive' (default false, preserving current behaviour) to the HTTP task parameters, expose it in the UI, and wire it through the shared OkHttpUtils via dedicated overloads that select a socket factory enabling TCP keepalive when requested. Existing OkHttpUtils call sites (alert, oauth) are unchanged. Note: the keepalive probe interval itself is governed by the worker host OS sysctls (net.ipv4.tcp_keepalive_time etc), not by this JVM setting; documented next to the field. Tests: HttpParametersTest covers serialization/default, HttpTaskTest runs GET/POST/DELETE with keepalive enabled against a MockWebServer. Signed-off-by: yongfu.gao <2642474295@qq.com>
|
Thanks for opening this pull request! Please check out our contributing guidelines. (https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md) |
SbloodyS
left a comment
There was a problem hiding this comment.
You are using AI to create PR and did not follow the pull request template.
Sorry, I didn't follow the PR template. Updated it now. |
SbloodyS
left a comment
There was a problem hiding this comment.
** Separate connection pools for enabled and disabled TCP keepalive**
OkHttpUtils.java:315–320 derives both client variants from CLIENT.newBuilder(), which shares the original connection pool. In OkHttp 4.12.0, connection eligibility uses Address.equalsNonHost(), which does not compare the plain socketFactory.
If a request with keepalive disabled leaves a reusable connection in the pool, a subsequent keepalive-enabled request to the same origin can reuse that socket without invoking KEEP_ALIVE_SOCKET_FACTORY. TCP keepalive remains disabled, so the new option does not reliably address the idle-connection failure. The reverse also occurs: disabled requests can reuse enabled sockets.
Please maintain separate reusable pools for the two settings. Add a regression test that sends consecutive requests to the same origin with different settings and verifies the actual socket keepalive state; successful responses alone do not verify this behavior.
… reflecting OkHttp internals
Confirmed — Address.equalsNonHost$okhttp ignores socketFactory, so the shared pool made the option unreliable in both directions. KEEP_ALIVE_CLIENT is now built from OkHttpUtilsTest sends consecutive requests to the same origin with different settings, and asserts the other setting's pooled connection count stays 0, that the keepalive client's socket factory produces a socket with getKeepAlive() == true, and (recorded on the test server) that the two requests did not share one TCP connection. It fails if I revert to a shared pool. getHttpClient(...) is now package-private + @VisibleForTesting so the test stays on public OkHttp APIs instead of reflecting into okhttp3.internal.*. |
Was this PR generated or assisted by AI?
Yes. The OkHttpUtils keepalive overloads and part of the unit tests were drafted with AI help. I reviewed the whole diff myself, ran the tests and spotless locally, and confirmed the default stays
false.Ref: #18611
Purpose of the pull request
Add an opt-in TCP keepalive (SO_KEEPALIVE) switch to the HTTP task.
Some HTTP tasks fail with
Read timed outwhen the worker sits behind a NAT or firewall that drops idle TCP sessions. curl works in the same setup because it enables keepalive by default, while OkHttp does not. The option defaults tofalse, so existing tasks are unaffected.Brief change log
HttpParameters: newsocketKeepAlivefield, defaultfalse.OkHttpUtils:get/post/put/deleteoverloads with a keepalive flag; the oldoverloads are unchanged.
HttpTask: pass the flag through for GET/POST/PUT/DELETE.Verify this pull request
This change added tests and can be verified as follows:
HttpParametersTest: default value and serialization round-trip.HttpTaskTest: GET/POST/DELETE against a MockWebServer with keepalive on../mvnw -pl dolphinscheduler-task-plugin/dolphinscheduler-task-http -am test -Dtest=HttpTaskTest,HttpParametersTest -Dsurefire.failIfNoSpecifiedTests=false→ Tests run: 16, Failures: 0, Errors: 0.
./mvnw -pl dolphinscheduler-common,dolphinscheduler-task-plugin/dolphinscheduler-task-http spotless:checkpasses.
Pull Request Notice
Pull Request Notice
No incompatible change, so
incompatible.mdis unchanged.