[fix](cloud) prevent warm-up job scheduling starvation

[bobhan1]

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

When cloud warm-up jobs outnumber the configured active slots, the scheduler can over-submit tasks because a job is recorded as active only after its worker starts. The direct-handoff thread pool previously discarded rejected tasks, and unordered map traversal could repeatedly favor the same jobs, leaving other runnable jobs pending indefinitely.

This PR keeps the direct-handoff pool and makes scheduling bounded and retryable:

• Scan runnable jobs once and keep only the jobs needed by the current slots in a bounded priority heap.
• Reserve the active slot before submission and roll it back when submission is rejected.
• Schedule the least-recently-run jobs first, with never-scheduled one-time jobs taking the first turn.
• Surface pool rejection so the scheduler can retry the job in the next cycle.
• Apply mutable scheduler concurrency and interval configuration without requiring an FE restart.

Release note

Fix cloud warm-up jobs remaining pending when the scheduler thread pool is saturated.

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes. Runnable warm-up jobs are scheduled fairly, and rejected submissions remain eligible for retry.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[bobhan1]

/review

[github-actions]

Review status: complete. The third and final review round converged with no new valuable findings; five distinct issues remain and are attached inline.

Summary of findings:

1. Sequence-0 arrivals can indefinitely starve an already-RUNNING multi-step job, including its lease renewal and timeout checks.
2. A mutable scheduler interval of 0 hot-spins JobDaemon, while a negative value terminates it with an uncaught IllegalArgumentException.
3. The bounded heap eagerly allocates the full unbounded concurrency limit every cycle, even when almost no jobs are runnable.
4. The tests synthesize rejection with a mock and do not protect the production throwing direct-handoff pool wiring required for rollback.
5. Sorted submissions do not enforce ONCE-first destination admission because later worker threads can acquire the per-destination registration first.

Critical checkpoint conclusions:

• Goal and proof: Scheduler-side reservation fixes the original over-submission race, and a fixed finite job set rotates as intended. The PR does not fully achieve starvation-free/priority-correct scheduling because of findings 1 and 5. The unit tests cover bounded fixed-set selection, active reservation, synthesized retry, and positive resize, but not the production admission/rejection paths or the adverse schedules above.
• Scope: The implementation is localized to the manager plus one focused test file. The mutable-interval behavior is part of the stated PR scope.
• Concurrency: JobDaemon is the sole production scheduler/inserter and workers remove exact reservations in finally; no separate over-admission, leak, lock-order, or deadlock defect survived review. The remaining concurrency defect is worker-side destination admission racing after sorted handoff (finding 5).
• Lifecycle: PENDING, RUNNING, periodic wait/resume, event-driven bypass, cancellation, cleanup, replay, and master failover were traced. Cleanup and replay are otherwise consistent, but in-progress work can be denied all later state-machine turns by new arrivals (finding 1).
• Configuration: Positive pool growth/shrink and pause/resume are safe for the core-size-zero cached executor. Non-positive interval values are accepted without validation (finding 2), and an unbounded maximum drives eager per-cycle allocation (finding 3).
• Compatibility and parallel paths: No persisted field, journal opcode, RPC/thrift value, storage format, FE-BE variable, or rolling-upgrade contract changes. ONCE, PERIODIC, EVENT_DRIVEN, table, cancellation, and replay paths were checked; the supported ONCE/PERIODIC same-destination combination exposes finding 5.
• Tests and results: The added assertions are deterministic and restore modified globals. They miss continuous arrivals, invalid intervals, high-capacity allocation, real SynchronousQueue rejection, and delayed-first-worker destination admission. CheckStyle is green on the reviewed head; this review-only runner did not run builds or tests.
• Error handling and observability: Throwing rejection is rolled back and retried, accepted tasks release reservations in finally, and job IDs, destination-lock logs, rejection counts, and pool metrics remain available. The test gap in finding 4 leaves the critical production handler wiring unprotected.
• Persistence, transactions, and data writes: Scheduling history is intentionally in-memory; existing job edit-log/state transitions remain unchanged. No transaction, data-write, visibility, or delete-bitmap path is modified.
• Performance: Apart from finding 3, selection is bounded to O(N log slots) plus sorting retained slots; no additional CPU, memory, or redundant-work issue survived the final sweep.
• User focus: No additional user-provided focus was present, so the complete two-file PR was reviewed without a narrower focus.

Please address the five inline findings and add production-faithful concurrency/configuration coverage before merge.

[bobhan1]

run buildall

[hello-stephen]

TPC-H: Total hot run time: 16709 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6de8e580eb05d9511c8f0e604017d5b9e7676452, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17595	3017	3015	3015
q2	2059	249	220	220
q3	10272	905	505	505
q4	4671	250	201	201
q5	7679	597	384	384
q6	138	112	93	93
q7	546	486	387	387
q8	9235	834	896	834
q9	3460	2391	2385	2385
q10	6502	853	699	699
q11	402	201	180	180
q12	613	258	200	200
q13	18140	1520	1164	1164
q14	161	150	143	143
q15	q16	434	392	361	361
q17	1342	843	786	786
q18	3057	2251	2267	2251
q19	1253	910	761	761
q20	368	281	203	203
q21	5593	1707	1857	1707
q22	331	268	230	230
Total cold run time: 93851 ms
Total hot run time: 16709 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3383	3306	3283	3283
q2	512	393	401	393
q3	2237	2409	2139	2139
q4	1185	1165	897	897
q5	2210	2137	2147	2137
q6	170	119	87	87
q7	1023	934	874	874
q8	1584	1387	1394	1387
q9	3146	3102	3091	3091
q10	1872	1803	1625	1625
q11	350	265	249	249
q12	450	424	338	338
q13	1463	1517	1179	1179
q14	172	165	173	165
q15	q16	386	394	360	360
q17	3610	3281	3184	3184
q18	4795	4432	4755	4432
q19	828	821	924	821
q20	998	960	819	819
q21	3826	3090	3235	3090
q22	385	358	328	328
Total cold run time: 34585 ms
Total hot run time: 30878 ms

[hello-stephen]

TPC-DS: Total hot run time: 81970 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6de8e580eb05d9511c8f0e604017d5b9e7676452, data reload: false

query5	4248	402	335	335
query6	375	134	122	122
query7	4973	431	230	230
query8	290	129	118	118
query9	8691	2851	2866	2851
query10	427	215	175	175
query11	5393	1034	905	905
query12	133	71	71	71
query13	1180	450	325	325
query14	6131	2207	2076	2076
query14_1	1946	1957	1947	1947
query15	171	125	117	117
query16	917	392	360	360
query17	795	476	377	377
query18	2338	338	244	244
query19	167	152	114	114
query20	71	70	71	70
query21	199	100	88	88
query22	5419	5296	5280	5280
query23	6846	6336	6142	6142
query23_1	6099	6199	6074	6074
query24	7252	1121	774	774
query24_1	794	784	808	784
query25	438	325	261	261
query26	1258	263	126	126
query27	2751	413	253	253
query28	4713	1511	1485	1485
query29	906	418	341	341
query30	246	160	131	131
query31	812	395	324	324
query32	126	74	81	74
query33	452	206	171	171
query34	990	835	481	481
query35	403	395	343	343
query36	576	535	531	531
query37	114	86	73	73
query38	1001	837	826	826
query39	511	493	498	493
query39_1	461	460	450	450
query40	199	89	76	76
query41	54	51	54	51
query42	73	71	70	70
query43	238	238	214	214
query44	987	547	541	541
query45	108	103	102	102
query46	794	837	542	542
query47	759	769	708	708
query48	320	303	213	213
query49	543	241	188	188
query50	770	263	198	198
query51	8141	7963	8002	7963
query52	67	67	62	62
query53	195	200	146	146
query54	202	169	176	169
query55	70	59	56	56
query56	264	185	178	178
query57	700	693	667	667
query58	197	167	160	160
query59	1191	1219	1108	1108
query60	233	179	176	176
query61	118	117	112	112
query62	349	207	172	172
query63	169	138	135	135
query64	2839	720	663	663
query65	1594	1619	1612	1612
query66	1861	279	238	238
query67	10261	9820	9546	9546
query68	3038	1149	739	739
query69	341	229	195	195
query70	656	612	596	596
query71	247	189	166	166
query72	2337	1729	1543	1543
query73	649	592	335	335
query74	1975	1220	1116	1116
query75	1189	1110	972	972
query76	2398	740	561	561
query77	252	262	221	221
query78	3917	3677	3259	3259
query79	2792	876	587	587
query80	1602	317	278	278
query81	497	157	130	130
query82	624	127	94	94
query83	283	210	194	194
query84	300	115	95	95
query85	821	353	293	293
query86	390	203	165	165
query87	1005	975	886	886
query88	2883	2108	2098	2098
query89	286	195	173	173
query90	1955	128	126	126
query91	131	123	103	103
query92	76	70	71	70
query93	1915	1145	656	656
query94	615	261	222	222
query95	524	323	232	232
query96	779	585	273	273
query97	1045	1046	962	962
query98	165	140	133	133
query99	415	351	306	306
Total cold run time: 179447 ms
Total hot run time: 81970 ms

[hello-stephen]

ClickBench: Total hot run time: 14.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6de8e580eb05d9511c8f0e604017d5b9e7676452, data reload: false

query1	0.01	0.00	0.01
query2	0.08	0.04	0.04
query3	0.24	0.11	0.11
query4	1.61	0.10	0.11
query5	0.17	0.16	0.16
query6	1.27	0.68	0.68
query7	0.03	0.00	0.00
query8	0.05	0.03	0.03
query9	0.28	0.21	0.22
query10	0.34	0.35	0.35
query11	0.17	0.11	0.12
query12	0.14	0.12	0.12
query13	0.31	0.31	0.30
query14	0.45	0.45	0.43
query15	0.36	0.35	0.35
query16	0.22	0.23	0.24
query17	0.66	0.72	0.64
query18	0.17	0.16	0.17
query19	1.24	1.17	1.13
query20	0.02	0.01	0.01
query21	15.46	0.16	0.11
query22	5.06	0.05	0.04
query23	16.18	0.26	0.10
query24	2.98	0.36	0.27
query25	0.11	0.03	0.04
query26	0.72	0.16	0.11
query27	0.04	0.03	0.02
query28	3.60	0.52	0.28
query29	12.50	3.12	2.59
query30	0.24	0.12	0.12
query31	2.76	0.38	0.17
query32	3.55	0.30	0.23
query33	1.35	1.41	1.54
query34	15.39	2.17	1.77
query35	1.74	1.74	1.71
query36	0.46	0.30	0.29
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.03	0.03	0.02
query40	0.11	0.08	0.08
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 90.35 s
Total hot run time: 14.65 s