Skip to content

[fix](load) Preserve routine load CSV header formats - #67513

Open
0AyanamiRei wants to merge 1 commit into
apache:masterfrom
0AyanamiRei:fix/doris-25187-routine-load-csv-header-format
Open

[fix](load) Preserve routine load CSV header formats#67513
0AyanamiRei wants to merge 1 commit into
apache:masterfrom
0AyanamiRei:fix/doris-25187-routine-load-csv-header-format

Conversation

@0AyanamiRei

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

Nereids routine load normalized csv_with_names and csv_with_names_and_types to plain CSV while creating the job, then omitted the header type during task planning. Header rows were consequently parsed as data and counted as errors.

This change preserves the validated CSV header format in job properties and forwards it through the Nereids task to the data description.

Release note

Routine Load now honors csv_with_names and csv_with_names_and_types without treating header rows as data.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
      • Extended KafkaRoutineLoadJobTest coverage for plain and header-aware CSV formats.
      • Not run per request.
    • Manual test
    • No need to test or manual test.
  • Behavior changed:

    • No.
    • Yes. Routine Load skips CSV header rows according to the configured format.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Nereids routine load normalized csv_with_names and csv_with_names_and_types to plain CSV while creating the job, then omitted the header type during task planning. Header rows were consequently parsed as data and counted as errors. Preserve the validated CSV header format in job properties and forward it to the Nereids data description.

### Release note

Routine Load now honors csv_with_names and csv_with_names_and_types without treating header rows as data.

### Check List (For Author)

- Test: Unit Test
    - Extended KafkaRoutineLoadJobTest coverage for plain and header-aware CSV formats; not run per request.
- Behavior changed: Yes. Routine Load skips CSV header rows according to the configured format.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@0AyanamiRei

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 17049 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8a6165186abaa45d709f58c76020f6c6f1ea605f, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17562	3119	3122	3119
q2	2117	270	238	238
q3	10206	896	531	531
q4	4668	250	210	210
q5	7671	558	383	383
q6	137	110	93	93
q7	530	599	383	383
q8	9241	893	950	893
q9	3469	2413	2402	2402
q10	6544	867	717	717
q11	396	202	187	187
q12	614	259	202	202
q13	18179	1528	1159	1159
q14	165	150	138	138
q15	q16	431	394	372	372
q17	1330	882	790	790
q18	3091	2285	2247	2247
q19	1262	938	778	778
q20	405	290	196	196
q21	5609	1785	1900	1785
q22	330	268	226	226
Total cold run time: 93957 ms
Total hot run time: 17049 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3459	3389	3395	3389
q2	519	403	382	382
q3	2222	2333	2200	2200
q4	1185	1176	902	902
q5	2187	2137	2115	2115
q6	176	119	86	86
q7	1047	973	904	904
q8	1619	1430	1446	1430
q9	3184	3163	3145	3145
q10	1861	1791	1609	1609
q11	370	277	255	255
q12	469	430	353	353
q13	1509	1528	1175	1175
q14	178	172	169	169
q15	q16	395	395	358	358
q17	3615	3279	3272	3272
q18	4843	4442	4815	4442
q19	992	857	876	857
q20	1014	943	809	809
q21	3972	3233	3205	3205
q22	404	352	322	322
Total cold run time: 35220 ms
Total hot run time: 31379 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 82740 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8a6165186abaa45d709f58c76020f6c6f1ea605f, data reload: false

query5	4245	417	331	331
query6	373	138	124	124
query7	4968	393	240	240
query8	302	125	115	115
query9	8729	2969	2984	2969
query10	376	223	195	195
query11	5380	1039	920	920
query12	124	71	70	70
query13	1197	414	340	340
query14	6079	2231	2122	2122
query14_1	1985	1994	2033	1994
query15	170	121	110	110
query16	924	373	352	352
query17	798	452	364	364
query18	2340	349	246	246
query19	183	132	106	106
query20	68	70	68	68
query21	199	100	88	88
query22	5354	5496	5410	5410
query23	6763	6277	6009	6009
query23_1	6249	6090	6082	6082
query24	7294	1121	787	787
query24_1	781	791	782	782
query25	438	273	231	231
query26	1229	226	132	132
query27	2796	415	258	258
query28	4694	1506	1504	1504
query29	914	421	342	342
query30	252	156	131	131
query31	817	395	338	338
query32	123	69	72	69
query33	445	232	187	187
query34	978	831	491	491
query35	427	402	343	343
query36	572	557	565	557
query37	119	81	69	69
query38	1007	845	803	803
query39	491	483	460	460
query39_1	464	486	461	461
query40	202	96	77	77
query41	52	52	53	52
query42	72	71	69	69
query43	243	242	213	213
query44	1008	553	566	553
query45	116	113	103	103
query46	775	857	515	515
query47	775	764	690	690
query48	326	307	221	221
query49	532	239	194	194
query50	723	268	195	195
query51	8267	8170	8193	8170
query52	70	69	58	58
query53	191	193	146	146
query54	207	170	264	170
query55	76	60	54	54
query56	191	178	172	172
query57	712	692	658	658
query58	195	164	162	162
query59	1237	1247	1118	1118
query60	229	182	166	166
query61	114	119	117	117
query62	345	195	179	179
query63	172	137	142	137
query64	2685	793	630	630
query65	1615	1649	1644	1644
query66	1905	294	220	220
query67	9946	9911	9759	9759
query68	3002	1263	770	770
query69	360	222	199	199
query70	696	629	597	597
query71	253	181	167	167
query72	2397	1767	1575	1575
query73	654	570	330	330
query74	2004	1238	1123	1123
query75	1192	1107	975	975
query76	2396	733	537	537
query77	262	260	217	217
query78	4022	3651	3180	3180
query79	1161	809	585	585
query80	1185	348	267	267
query81	494	156	131	131
query82	625	130	98	98
query83	323	212	197	197
query84	296	114	91	91
query85	795	344	310	310
query86	387	205	173	173
query87	1037	972	905	905
query88	2798	2161	2120	2120
query89	291	197	176	176
query90	1926	136	134	134
query91	136	119	99	99
query92	79	71	71	71
query93	1391	1138	690	690
query94	667	262	236	236
query95	515	262	229	229
query96	831	572	273	273
query97	1045	1050	1013	1013
query98	145	133	138	133
query99	419	355	319	319
Total cold run time: 176852 ms
Total hot run time: 82740 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.79 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8a6165186abaa45d709f58c76020f6c6f1ea605f, data reload: false

query1	0.01	0.00	0.01
query2	0.08	0.04	0.03
query3	0.24	0.11	0.11
query4	1.60	0.10	0.10
query5	0.17	0.16	0.17
query6	1.24	0.70	0.69
query7	0.03	0.01	0.01
query8	0.04	0.03	0.03
query9	0.29	0.22	0.21
query10	0.35	0.33	0.36
query11	0.17	0.12	0.12
query12	0.16	0.12	0.12
query13	0.32	0.31	0.31
query14	0.47	0.45	0.46
query15	0.38	0.36	0.36
query16	0.23	0.21	0.21
query17	0.68	0.74	0.66
query18	0.19	0.17	0.17
query19	1.18	1.15	1.23
query20	0.02	0.01	0.01
query21	15.43	0.15	0.13
query22	5.07	0.04	0.04
query23	16.18	0.25	0.12
query24	3.01	0.33	0.26
query25	0.09	0.05	0.03
query26	0.75	0.16	0.13
query27	0.03	0.04	0.02
query28	3.68	0.56	0.26
query29	12.45	3.22	2.63
query30	0.25	0.12	0.13
query31	2.76	0.38	0.17
query32	3.52	0.32	0.24
query33	1.47	1.37	1.44
query34	15.40	2.22	1.77
query35	1.80	1.77	1.74
query36	0.46	0.30	0.30
query37	0.06	0.04	0.04
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.12	0.07	0.07
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 90.59 s
Total hot run time: 14.79 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants