Skip to content

cloudstorage: make all DateSeparator fields use the enum instead of string - #6078

Merged
ti-chi-bot[bot] merged 2 commits into
pingcap:masterfrom
kennytm:fix-6077
Sep 2, 2026
Merged

cloudstorage: make all DateSeparator fields use the enum instead of string#6078
ti-chi-bot[bot] merged 2 commits into
pingcap:masterfrom
kennytm:fix-6077

Conversation

@kennytm

@kennytm kennytm commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #6077

What is changed and how it works?

Changes all places that uses the DateSeparator expecting a string to the common enum config.DateSeparator so case-sensitivity no longer appears.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Questions

Will it cause performance regression or break compatibility?

No

Do you need to update user documentation, design documentation or monitoring documentation?

No

Release note

Previously sink.date-seperator accepted non-lowercased values like "DAY" but ignored them. Now they are accepted and treated the same as the lowercased versions.

Summary by CodeRabbit

  • Improvements
    • Standardized date-separator configuration using supported typed values across storage sinks.
    • Added consistent JSON and TOML serialization, including case normalization and validation of unsupported values.
    • Improved date-separator handling when parsing storage paths and generating dated files.
    • Cloud-storage cleanup now reports configured separators more clearly.
    • Sensitive configuration masking now also covers large-message storage credentials.
  • Bug Fixes
    • Invalid date-separator settings are now rejected with clear configuration errors.
    • Storage consumers consistently apply the configured separator during scans.

@ti-chi-bot ti-chi-bot Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/needs-triage-completed labels Aug 24, 2026
@coderabbitai

coderabbitai Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: c579b2eb-41cd-4639-b943-a175454ef586

📥 Commits

Reviewing files that changed from the base of the PR and between b8e93ff and e4c59f3.

📒 Files selected for processing (3)
  • api/v2/model.go
  • api/v2/model_test.go
  • pkg/config/sink.go

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The change replaces string date separators with the typed config.DateSeparator enum. It adds text serialization and validation, updates cloud storage path handling and cleanup, caches the separator in the storage consumer, and expands sensitive-data masking.

Changes

Typed date separator configuration

Layer / File(s) Summary
Configuration contract and serialization
pkg/config/sink.go, pkg/config/replica_config.go, api/v2/model.go, pkg/config/sink_test.go, api/v2/model_test.go
DateSeparator fields now use the enum type. JSON and TOML conversion accepts case variants, emits normalized values, and rejects unsupported values.
Sensitive-data masking validation
pkg/config/sink.go, api/v2/model_test.go
KafkaConfig.MaskSensitiveData masks non-empty sensitive fields and the claim-check storage URI. Tests verify masked output and preservation of the original configuration.

Cloud storage typed path handling

Layer / File(s) Summary
Path parsing, date generation, and cleanup
pkg/cloudstorage/config.go, pkg/cloudstorage/path_key.go, pkg/cloudstorage/generator.go, pkg/cloudstorage/*_test.go
Cloud storage APIs and comparisons now use config.DateSeparator values directly. Date path generation and expiration behavior remain unchanged.

Consumer and sink integration

Layer / File(s) Summary
Cached consumer configuration
cmd/storage-consumer/consumer.go
The consumer reads the configured separator during construction and reuses it when parsing DML index paths.
Typed sink cleanup and tests
downstreamadapter/sink/cloudstorage/sink.go, downstreamadapter/sink/cloudstorage/*_test.go
Cleanup compares the typed day separator and logs separator values with zap.Stringer. Cloud storage tests use enum values directly.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to e4c59

This change standardizes DateSeparator handling so equivalent casing is accepted consistently; no actionable merge-blocking risk remains after normal checks and review.

Suggested reviewers: 3aceshowhand, wk989898

Poem

A rabbit checks the separator’s way,
Typed enums guide each folder day.
JSON accepts the proper case,
Paths and cleanup keep their place.
Secrets hide beneath the hay.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Out of Scope Changes check ⚠️ Warning Most changes support the DateSeparator objective, but KafkaConfig.MaskSensitiveData changes and the related CloneWithMaskedSensitiveData test add sensitive-data masking behavior unrelated to issue #60 Remove the unrelated sensitive-data masking changes and their test from this PR, or link a separate issue and explain why that scope belongs here.
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 3 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: replacing string-based DateSeparator fields with the common enum.
Description check ✅ Passed The description includes the required issue reference, change summary, test selections, compatibility and documentation answers, and release note. It is sufficiently complete, although the checked man…
Linked Issues check ✅ Passed The PR updates DateSeparator fields, parsing, serialization, and comparisons to use the typed enum. Case-insensitive unmarshaling and the provided tests address issue #6077 by making values such as "D…
Full details: Description check

Explanation

The description includes the required issue reference, change summary, test selections, compatibility and documentation answers, and release note. It is sufficiently complete, although the checked manual-test item does not include detailed steps in the description.

Full details: Linked Issues check

Explanation

The PR updates DateSeparator fields, parsing, serialization, and comparisons to use the typed enum. Case-insensitive unmarshaling and the provided tests address issue #6077 by making values such as "DAY" work as date-separated configuration.

Full details: Out of Scope Changes check

Explanation

Most changes support the DateSeparator objective, but KafkaConfig.MaskSensitiveData changes and the related CloneWithMaskedSensitiveData test add sensitive-data masking behavior unrelated to issue #6077.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ti-chi-bot ti-chi-bot Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Aug 24, 2026
@wk989898

Copy link
Copy Markdown
Collaborator

Please add a manual test 🙏🏻

@kennytm

kennytm commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

Manual test (not integration test yes?):

make cdc

tiup playground-ng v8.5.7 --db 1 --kv 1 --pd 1 --tiflash 0 --ticdc 1 --ticdc.binpath ./bin/cdc --ticdc.config ./ticdc-newarch.toml --without-monitor

./bin/cdc cli changefeed create -c test5854 --sink-uri 's3://test5854/?endpoint=http://127.0.0.1:9000&protocol=canal-json&access-key=minioadmin&secret-access-key=minioadmin' --config ./test5854.toml

mysql -u root -h 127.0.0.1 -P 4000 test
# create table a (a bigint primary key);
# insert into a values (4);

The file is created at s3://test5854/test/a/468602811870085128/2026-08-24/CDC00000000000000000001.json yes.

@ti-chi-bot ti-chi-bot Bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Aug 24, 2026
@kennytm

kennytm commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

/check-issue-triage-complete

@ti-chi-bot ti-chi-bot Bot added needs-cherry-pick-release-nextgen-202603 Should cherry pick this PR to release-nextgen-202603 branch. and removed do-not-merge/needs-triage-completed labels Aug 25, 2026
@ti-chi-bot ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Aug 27, 2026
@ti-chi-bot

ti-chi-bot Bot commented Aug 27, 2026

Copy link
Copy Markdown

[LGTM Timeline notifier]

Timeline:

  • 2026-08-24 13:36:14.801980486 +0000 UTC m=+508809.973074627: ☑️ agreed by wk989898.
  • 2026-08-27 03:01:05.996477993 +0000 UTC m=+729901.167572152: ☑️ agreed by 3AceShowHand.

@kennytm

kennytm commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

🤔 Needs sig-critical-approvers approval

@wk989898 wk989898 added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Sep 2, 2026
@ti-chi-bot

ti-chi-bot Bot commented Sep 2, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3AceShowHand, lidezhu, wk989898

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added the approved label Sep 2, 2026
@ti-chi-bot
ti-chi-bot Bot merged commit 0697ba0 into pingcap:master Sep 2, 2026
39 of 40 checks passed
@ti-chi-bot

Copy link
Copy Markdown
Member

In response to a cherrypick label: new pull request created to branch release-8.5: #6148.
But this PR has conflicts, please resolve them!

@ti-chi-bot

Copy link
Copy Markdown
Member

In response to a cherrypick label: new pull request created to branch release-nextgen-202603: #6149.
But this PR has conflicts, please resolve them!

@ti-chi-bot

ti-chi-bot Bot commented Sep 2, 2026

Copy link
Copy Markdown

@kennytm: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-cdc-mysql-integration-heavy-next-gen-legacy-safepoint e4c59f3 link unknown /test pull-cdc-mysql-integration-heavy-next-gen-legacy-safepoint

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@kennytm
kennytm deleted the fix-6077 branch September 2, 2026 09:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. needs-cherry-pick-release-nextgen-202603 Should cherry pick this PR to release-nextgen-202603 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Wrong-cased date-separator is accepted but ignored

5 participants