Skip to content

[fix](routineload) Preserve paused job metadata during replay - #67514

Open
0AyanamiRei wants to merge 1 commit into
apache:masterfrom
0AyanamiRei:fix/doris-25341-routine-load-replay-metadata
Open

[fix](routineload) Preserve paused job metadata during replay#67514
0AyanamiRei wants to merge 1 commit into
apache:masterfrom
0AyanamiRei:fix/doris-25341-routine-load-replay-metadata

Conversation

@0AyanamiRei

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: In cloud mode, paused Kafka routine load jobs could retain their state and pause reason during FE replay while losing statistics, progress, lag, and current partition metadata. Pause edit logs only carried the state transition, so checkpoint replay could not reconstruct operator-visible metadata that lived outside the FE transaction journal. This change persists a metadata snapshot with Kafka pause operations, restores it during replay, and retains the Kafka fields in subsequent checkpoint images. Legacy pause operations without a snapshot keep the existing replay behavior.

Release note

Paused Kafka routine load jobs preserve their statistics, progress, lag, and current partitions after FE replay.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test (coverage added; not run per request)
    • Manual test
    • No need to test or manual test
  • Behavior changed:

    • No.
    • Yes. Paused Kafka routine load metadata is retained across FE replay.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: In cloud mode, paused Kafka routine load jobs could retain their state and pause reason during FE replay while losing statistics, progress, lag, and current partition metadata. Pause edit logs only carried the state transition, so checkpoint replay could not reconstruct operator-visible metadata that lived outside the FE transaction journal. Persist the metadata snapshot with Kafka pause operations, restore it during replay, and retain the Kafka fields in subsequent checkpoint images.

### Release note

Paused Kafka routine load jobs preserve their statistics, progress, lag, and current partitions after FE replay.

### Check List (For Author)

- Test: Unit Test added; not run at user request
- Behavior changed: Yes (paused Kafka routine load metadata is retained across FE replay)
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants