Skip to content

OCPBUGS-115111: Tolerate HyperShift startup pod alerts - #31633

Open
vsolanki12 wants to merge 1 commit into
openshift:mainfrom
vsolanki12:fix-OCPBUGS-115111-main
Open

vsolanki12 wants to merge 1 commit into
openshift:mainfrom
vsolanki12:fix-OCPBUGS-115111-main

Conversation

@vsolanki12

@vsolanki12 vsolanki12 commented Sep 11, 2026

Copy link
Copy Markdown

What does this PR do?

Tolerates bounded, expected KubePodNotReady alerts during HyperShift guest-cluster startup.

The exception applies only to external control-plane topology and the affected
openshift-dns, openshift-insights, and openshift-ingress-canary
namespaces.

Why is this change needed?

HyperShift guest clusters can temporarily report these alerts while initial
platform pods converge. The monitor previously evaluated the startup intervals
with normal thresholds, causing false failures.

Intervals that continue beyond the five-minute startup grace period remain
subject to the existing checks.

How was this tested?

  • go test ./pkg/monitortests/testframework/legacytestframeworkmonitortests
  • go test ./pkg/monitortests/testframework/... ./pkg/cmd/openshift-tests/dev
  • go test ./pkg/monitortests/...

Jira

https://issues.redhat.com/browse/OCPBUGS-115111

Summary by CodeRabbit

  • Bug Fixes
    • Improved alert evaluation for external-topology environments by ignoring selected startup-related pod readiness alerts during the first five minutes.
    • Correctly trims alerts that overlap the startup grace period while preserving unaffected alerts.
    • Added coverage for startup filtering across supported namespaces, topologies, and collection start times.

OCPBUGS-115111

Guest clusters with external control planes can report KubePodNotReady while initial platform pods converge. Limit the exception to the startup window and affected namespaces so later readiness regressions remain visible.

Signed-off-by: Vimal Solanki <vsolanki@redhat.com>
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 11, 2026
@openshift-ci

openshift-ci Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Sep 11, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@vsolanki12: This pull request references Jira Issue OCPBUGS-115111, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.1.0) matches configured target version for branch (5.1.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What does this PR do?

Tolerates bounded, expected KubePodNotReady alerts during HyperShift guest-cluster startup.

The exception applies only to external control-plane topology and the affected
openshift-dns, openshift-insights, and openshift-ingress-canary
namespaces.

Why is this change needed?

HyperShift guest clusters can temporarily report these alerts while initial
platform pods converge. The monitor previously evaluated the startup intervals
with normal thresholds, causing false failures.

Intervals that continue beyond the five-minute startup grace period remain
subject to the existing checks.

How was this tested?

  • go test ./pkg/monitortests/testframework/legacytestframeworkmonitortests
  • go test ./pkg/monitortests/testframework/... ./pkg/cmd/openshift-tests/dev
  • go test ./pkg/monitortests/...

Jira

https://issues.redhat.com/browse/OCPBUGS-115111

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Sep 11, 2026
@openshift-ci

openshift-ci Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: vsolanki12
Once this PR has been reviewed and has the lgtm label, please assign miyadav for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Sep 11, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Advanced

Run ID: e96ed27e-5247-486a-a611-ff0ea39bf059

📥 Commits

Reviewing files that changed from the base of the PR and between f55ef01 and 3195db2.

📒 Files selected for processing (3)
  • pkg/monitortests/testframework/legacytestframeworkmonitortests/alerts.go
  • pkg/monitortests/testframework/legacytestframeworkmonitortests/alerts_monitortest.go
  • pkg/monitortests/testframework/legacytestframeworkmonitortests/alerts_test.go

Included review availability: Your plan provides up to 8 included reviews per hour; 6 remain after this review.


Walkthrough

Alert evaluation now filters selected startup KubePodNotReady intervals for external-topology jobs. The monitor test framework records and passes the collection start time, with tests covering filtering, clipping, preservation, and bypass behavior.

Changes

External topology alert filtering

Layer / File(s) Summary
Startup alert interval filtering
pkg/monitortests/testframework/legacytestframeworkmonitortests/alerts.go, pkg/monitortests/testframework/legacytestframeworkmonitortests/alerts_test.go
External-topology alert evaluation applies a five-minute grace period to selected KubePodNotReady intervals. Tests cover removal, clipping, preservation, and bypass cases.
Collection start propagation
pkg/monitortests/testframework/legacytestframeworkmonitortests/alerts_monitortest.go
The monitor test state stores the collection start time and passes it to upgrade and conformance alert evaluation.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Suggested reviewers: dgoodwin

Merge Risk: ⚪ Minimal · up to 3195d

The startup grace-period filtering covers the intended boundary without leaving an alert at exactly five minutes eligible for failure.

🚥 Pre-merge checks | ✅ 13 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 3 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Structure And Quality ⚠️ Warning The added table test uses assert.Equal(t, tt.expected, actual) without a failure message at alerts_test.go:236. This violates the explicit assertion-message requirement. The test otherwise uses is… Add a meaningful message argument, for example: assert.Equal(t, tt.expected, actual, "unexpected filtered intervals for %q", tt.name).
✅ Passed checks (13 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The pull request adds only static Go test names. TestFilterExternalTopologyStartupAlertIntervals and its t.Run names contain descriptive fixed text. They do not include pod names, dates, UUIDs, no…
Microshift Test Compatibility ✅ Passed The custom check applies to new Ginkgo e2e tests. The pull request adds TestFilterExternalTopologyStartupAlertIntervals, an ordinary Go testing.T unit test using testify/assert; it adds no It,…
Single Node Openshift (Sno) Test Compatibility ✅ Passed The pull request adds a standard Go unit test, TestFilterExternalTopologyStartupAlertIntervals, not a Ginkgo e2e test. The changed test uses testing.T and testify, and no changed file contains `…
Topology-Aware Scheduling Compatibility ✅ Passed PASS: The pull request changes only alert-monitoring logic and unit tests in three Go files. The diff adds startup alert interval filtering and stores the collection start time. It adds no deployment …
Ote Binary Stdout Contract ✅ Passed PASS: The pull request adds only interval filtering, state propagation, and unit-test logic. The authoritative diff contains no fmt.Print*, log.Print*, klog, os.Stdout, Ginkgo suite setup, or other ne…
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The pull request adds a standard Go unit test, TestFilterExternalTopologyStartupAlertIntervals, not a Ginkgo e2e test. The changed files contain no hardcoded IPv4 addresses, IPv4-only parsing, IPv4 …
No-Weak-Crypto ✅ Passed PASS: The authoritative diff contains only three Go files for alert filtering and monitor timing. The added lines introduce time, namespace, and alert-name checks only. The changed files import no cry…
Container-Privileges ✅ Passed The pull request changes only three Go files. It adds no container or Kubernetes manifests, and the added or removed lines contain none of the checked privilege settings: privileged, hostPID, hostNetw…
No-Sensitive-Data-In-Logs ✅ Passed No sensitive-data logging was introduced. The authoritative diff adds interval filtering, time/state handling, and unit tests only; it adds zero logging or output calls and no passwords, tokens, API k…
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the main change: tolerating bounded HyperShift startup pod alerts. This matches the external-topology startup alert filtering implemented in the pull request.
Full details: Test Structure And Quality

Explanation

The added table test uses assert.Equal(t, tt.expected, actual) without a failure message at alerts_test.go:236. This violates the explicit assertion-message requirement. The test otherwise uses isolated subtests, creates no cluster resources, and performs no cluster waits, so setup, cleanup, and timeout requirements do not apply.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Sep 11, 2026
@vsolanki12
vsolanki12 marked this pull request as ready for review September 16, 2026 14:01
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 16, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6

@openshift-ci

openshift-ci Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

@vsolanki12: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants