Skip to content

test(breakfix): validate GPUd or Sentinel on GPU nodes (BFX04-01) #595

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
osu wants to merge 1 commit into
base: main
Choose a base branch
from
Open

test(breakfix): validate GPUd or Sentinel on GPU nodes (BFX04-01) #595

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/test-plan.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1999,6 +1999,7 @@ domains:
labels:
- bare_metal
- breakfix
- kubernetes
dependencies:
- LimitedEnv
req_id: BFX04
Expand Down
11 changes: 11 additions & 0 deletions isvctl/configs/providers/aws/config/eks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,17 @@ commands:
TF_AUTO_APPROVE: "true"
NODE_POOL_STATE_FILE: "terraform-delete.tfstate"

# BFX04-01: read-only inventory of Fleet Intelligence/GPUd or
# NVSentinel DaemonSet and pod coverage across every GPU node.
- name: query_node_health_agents
phase: test
command: "python3 ../../shared/breakfix/query_node_health_agents.py"
args:
- "--nodes="
requires_available_validations:
- K8sNodeHealthAgentCheck
timeout: 120

- name: teardown
phase: teardown
command: "../scripts/eks/teardown.sh"
Expand Down
43 changes: 43 additions & 0 deletions isvctl/configs/providers/gb300/config/bare_metal.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.

@abegnoche abegnoche Aug 25, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gb300 is not a provider but nico is one (maybe we should rename to nvidia at some point).
This whole file should be moved into isvctl/configs/providers/nico/config/bare_metal.yaml

# SPDX-License-Identifier: Apache-2.0

# GB300 bare-metal validation configuration.
#
# BFX04-01 performs read-only inspection of NVIDIA Fleet Intelligence Agent
# (GPUd) or NVSentinel. Set tests.settings.bfx04_nodes to comma-separated SSH
# targets for direct systemd inspection.
#
# Usage:
# uv run isvctl test run \
# -f isvctl/configs/providers/gb300/config/bare_metal.yaml \
# --set tests.settings.bfx04_nodes=gb300-node-01,gb300-node-02 \
# -- -k NodeHealthAgentCheck

import:
- ../../../suites/bare_metal.yaml

version: "1.0"

commands:
bare_metal:
phases: ["test"]
steps:
- name: query_node_health_agents
phase: test
command: "python ../../shared/breakfix/query_node_health_agents.py"
args:
- "--nodes={{ bfx04_nodes | default('', true) }}"
requires_available_validations:
- NodeHealthAgentCheck
timeout: 120

tests:
cluster_name: "gb300-bare-metal-validation"
description: "GB300 bare-metal validation"

settings:
region: ""
instance_type: ""
teardown_flag: ""
show_skipped_tests: true
bfx04_nodes: ""

@abegnoche abegnoche Aug 25, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be something like:

    health_agent_ssh_nodes: "{{env.HEALTH_AGENT_SSH_NODES | default('', true)}}"

8 changes: 8 additions & 0 deletions isvctl/configs/providers/minikube.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,14 @@ commands:
phase: setup
command: "my-isv/scripts/k8s/setup_minikube.sh"
timeout: 60
- name: query_node_health_agents

@abegnoche abegnoche Aug 25, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't typically update minikube.yaml since it's in sync with 2 others (k3s, microk8s)
I'd suggest to revert here

phase: test
command: "python shared/breakfix/query_node_health_agents.py"
args:
- "--nodes="
requires_available_validations:
- K8sNodeHealthAgentCheck
timeout: 120
- name: teardown
phase: teardown
command: "my-isv/scripts/k8s/teardown_minikube.sh"
Expand Down
9 changes: 9 additions & 0 deletions isvctl/configs/providers/my-isv/config/k8s.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,15 @@ commands:
args: ["--region", "{{region}}"]
timeout: 600

- name: query_node_health_agents
phase: test
command: "python ../../shared/breakfix/query_node_health_agents.py"
args:
- "--nodes="

@abegnoche abegnoche Aug 25, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: strange to have this empty

requires_available_validations:
- K8sNodeHealthAgentCheck
timeout: 120

tests:
description: "my-isv Kubernetes platform suite (real cluster required)"
settings:
Expand Down
Loading