Skip to content

osp_verification: stabilize LB tests in periodic CI jobs - #41

Open
tusharjadhav3302 wants to merge 1 commit into
mainfrom
osp-verification-lb-ci-stability
Open

osp_verification: stabilize LB tests in periodic CI jobs#41
tusharjadhav3302 wants to merge 1 commit into
mainfrom
osp-verification-lb-ci-stability

Conversation

@tusharjadhav3302

@tusharjadhav3302 tusharjadhav3302 commented Sep 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Stabilize LB-related failures in osp_verification* periodic jobs:

  • Run lb_tests before openstack_test in job definitions and playbooks/ocp_testing.yaml.
  • Block [lb] specs from the default openstack_test stage; LB coverage stays in lb_tests.
  • Serial OTE (run-test per case) with Amphora/OVN allowlists (no stdin batch parallelization).
  • Three Amphora phases in lb_tests:
    1. Isolation-sensitive cases (UDP create + TCP/UDP ETP:Local+monitors)
    2. Remaining Amphora cases (isolated + late sourceRanges patterns blocklisted so they are not double-run)
    3. Late UDP loadBalancerSourceRanges under Amphora (before OVN), so OCCM is less contended when clearing allowed_cidrs
  • IngressController LB test blocklisted in lb_tests (OSASINFRA-2412).
  • filter_tests_list allowlist+blocklist; truncate allow/block txt once before multi-file YAML conversion.
  • Optional lb_tests_guest_cleanup (default false); serial OTE fails if run-test or junit resolver fails.

Related: OSPRH-34941 · pairs with openshift/openstack-test#320 (Amphora per-spec teardown + EqualFold sourceRanges wait).

Test plan

  • serval71 manual lb_tests @ tip (69626c9 / late sourceRanges phase) + openstack-test PR #320: isolated 3/0 fail, amphora phase 2 0 fail, late sourceRanges PASS (ACL allow-all ~2s), OVN 0 fail
  • ci-framework-testproject MR: full osp_verification_5.1_nightly.yaml on serval71 with refs/pull/41/head

@openshift-ci

openshift-ci Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign stephenfin for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tusharjadhav3302
tusharjadhav3302 force-pushed the osp-verification-lb-ci-stability branch from 57dd1c2 to 9ca86a7 Compare September 16, 2026 12:50
ekuris-redhat
ekuris-redhat reviewed Sep 17, 2026
View reviewed changes
Comment thread collection/stages/roles/lb_tests/tasks/cleanup_lb_test_leftovers.yml Outdated
Comment thread collection/stages/roles/lb_tests/tasks/cleanup_lb_test_leftovers.yml
Comment thread collection/stages/roles/lb_tests/tasks/cleanup_lb_test_leftovers.yml Outdated
@tusharjadhav3302
tusharjadhav3302 force-pushed the osp-verification-lb-ci-stability branch 2 times, most recently from 72889ca to cf394be Compare September 17, 2026 07:54
ekuris-redhat
ekuris-redhat reviewed Sep 17, 2026
View reviewed changes
Comment thread collection/stages/roles/openstack_test/tasks/run_openstack_test.yml Outdated
done < {{ tests_to_run_path }}
python3 "{{ openstack_test_ote_resolve_script }}" junit \
"{{ openstack_test_log_path }}" "{{ openstack_test_junit_path }}" || py_exit=$?
exit ${run_exit:-${py_exit:-0}}

@ekuris-redhat ekuris-redhat Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The serial OTE runner initializes run_exit=0, then exits with:
exit ${run_exit:-${py_exit:-0}}

If the JUnit resolver fails, py_exit is ignored because run_exit is already set to 0. The Ansible task therefore succeeds even though JUnit generation failed. The exit status
should preserve either failure, for example:
exit "${run_exit:-${py_exit:-0}}"
with run_exit initialized only when needed, or explicitly combine both statuses.

@tusharjadhav3302 tusharjadhav3302 Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 64fc83c. The serial runner now initializes py_exit=0 and exits non-zero if either run-test or the junit resolver fails.

Comment thread collection/stages/roles/openstack_test/tasks/run_openstack_test.yml Outdated

@ekuris-redhat ekuris-redhat Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as in line 189

@tusharjadhav3302 tusharjadhav3302 Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 64fc83c for the batch run-test path as well (same combined exit logic).

@tusharjadhav3302

tusharjadhav3302 commented Sep 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Ready for another review pass at 64fc83c.

Summary since the last round:

  • Serial OTE + Amphora/OVN allowlists (2becb76)
  • Two-phase amphora with cleanup gated off by default (404384d / bb99bec)
  • Multi-file blocklist truncate fix (bb99bec)
  • Review follow-up: lb_tests_guest_cleanup default false, serial OTE exit status (64fc83c)

serval71 manual lb_tests at bb99bec: isolated 4/4, main amphora 0 failures, OVN 0 failures.

ci-framework-testproject branch osp-verification-pr41-serval71 triggers end-to-end Zuul on serval71 with refs/pull/41/head.: https://gitlab.cee.redhat.com/ci-framework/ci-framework-testproject/-/merge_requests/2711

@tusharjadhav3302 @cursoragent
osp_verification: stabilize LB tests in periodic CI jobs
69626c9 
Run lb_tests before openstack_test, keep [lb] out of the default suite,
and run Amphora/OVN LB OTE serially with provider allowlists. Split
Amphora into isolated, remaining, and late sourceRanges phases; gate
guest cleanup; fix multi-file blocklist truncate and serial OTE exit
status.

Co-authored-by: Cursor <cursoragent@cursor.com>
@tusharjadhav3302
tusharjadhav3302 force-pushed the osp-verification-lb-ci-stability branch from b1076d7 to 69626c9 Compare September 21, 2026 07:22
@tusharjadhav3302 tusharjadhav3302 mentioned this pull request Sep 21, 2026
Open
3 tasks
@tusharjadhav3302 tusharjadhav3302 changed the title osp_verification: exclude LB tests from default openstack suite; stabilize lb_tests stage osp_verification: stabilize LB tests in periodic CI jobs Sep 21, 2026
@tusharjadhav3302

tusharjadhav3302 commented Sep 21, 2026

Copy link
Copy Markdown
Contributor Author

Ready for another review pass at 69626c9 (supersedes the earlier note @ 64fc83c).

Summary since that round:

  • Squashed to a single commit on tip (69626c9)
  • Three Amphora phases in lb_tests: isolated (UDP create + TCP/UDP ETP:Local) → remaining Amphora → late UDP loadBalancerSourceRanges before OVN
  • Pairs with openshift/openstack-test#320 (Amphora per-spec teardown + EqualFold allowed_cidrs wait + sourceRanges failure diagnostics)
  • Review follow-ups from earlier rounds retained: lb_tests_guest_cleanup default false, serial OTE exit status, multi-file blocklist truncate

serval71 manual lb_tests at b1076d7 (same three-phase content as tip) + openstack-test PR #320: isolated 3/0 fail, amphora phase 2 0 fail, late sourceRanges PASS (ACL allow-all ~2s), OVN 0 fail. See OSPRH-34941.

ci-framework-testproject branch osp-verification-pr41-serval71 triggers end-to-end Zuul on serval71 with refs/pull/41/head: https://gitlab.cee.redhat.com/ci-framework/ci-framework-testproject/-/merge_requests/2711

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ekuris-redhat ekuris-redhat ekuris-redhat left review comments
@chenwng chenwng Awaiting requested review from chenwng chenwng is a code owner
@dlaw4608 dlaw4608 Awaiting requested review from dlaw4608 dlaw4608 is a code owner
@eshulman2 eshulman2 Awaiting requested review from eshulman2 eshulman2 is a code owner
@gryf gryf Awaiting requested review from gryf gryf is a code owner
@imatza-rh imatza-rh Awaiting requested review from imatza-rh imatza-rh is a code owner
@mandre mandre Awaiting requested review from mandre mandre is a code owner
@stephenfin stephenfin Awaiting requested review from stephenfin stephenfin is a code owner
@winiciusallan winiciusallan Awaiting requested review from winiciusallan winiciusallan is a code owner

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tusharjadhav3302 @ekuris-redhat