Vipsharm gpu#676
Open
Medhakulam wants to merge 30 commits into
Open
Conversation
…ped GPU Operator Add scripts/install-nvidia-drivers.sh to bake the NVIDIA data-center driver and its DKMS kernel modules into the Ubuntu base image, so GPU nodes can run the NVIDIA GPU Operator with driver.enabled=false in fully air-gapped environments (no host-side network, no operator driver container). The script derives the target kernel from /lib/modules (not the builder's uname -r), installs ABI-exact headers via install-kernel-headers.sh, forces DKMS build/install/depmod against that kernel, and verifies nvidia*.ko landed in the image kernel tree. It also blacklists nouveau, autoloads the nvidia stack, enables nvidia-persistenced, and rebuilds the initrd. Wire it into the base-image target (Ubuntu/amd64, after the kernel is finalized) behind INSTALL_NVIDIA_GPU_DRIVERS, with tunable branch/type/repo/fabricmanager/ toolkit args. Document in .arg.template and docs/nvidia-gpu-airgapped.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…section Explain that linux-headers/linux-image/linux-modules listed by NVIDIA belong to the driver-container + local-apt-mirror strategy (driver.enabled=true), whereas this integration uses the pre-installed-driver model (driver.enabled=false, no mirror) and satisfies those three requirements at build time in the image. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… semantics Clarify that the installer is version-agnostic (derives CUDA repo tag and kernel codename from the image's os-release), so it works on both Ubuntu 22.04 and 24.04. Note that NVIDIA_DRIVER_BRANCH selects a driver branch (latest patch within it), not an exact point release. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… Operator Add scripts/install-amdgpu-drivers.sh, the AMD counterpart to the NVIDIA installer. Bakes the amdgpu-dkms kernel driver into the Ubuntu base image so AMD Instinct nodes can run the AMD GPU Operator with driver.enable=false in fully air-gapped environments (no host-side network, no operator-built driver). Reuses the same build-time approach: derive the image kernel from /lib/modules (not the builder's uname -r), install ABI-exact headers via install-kernel-headers.sh plus linux-modules-extra, register the AMD driver repo via the version-matched amdgpu-install package (auto-discovered from repo.radeon.com/amdgpu-install/<version>/), install amdgpu-dkms, force DKMS build/install/depmod against the image kernel, and verify amdgpu.ko landed. No nouveau-style blacklist is needed since amdgpu-dkms replaces the in-tree amdgpu module. Wire it into the base-image target behind INSTALL_AMD_GPU_DRIVERS with an AMDGPU_ROCM_VERSION arg (default 7.2.4), alongside a build-time guard making the NVIDIA and AMD pre-install paths mutually exclusive. Document in .arg.template, docs/amd-gpu-airgapped.md, and cross-link from the NVIDIA doc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e-installed Add a "Building the air-gapped content" section to both the NVIDIA and AMD guides. Clarify that pre-installing the driver only lets you drop the driver image (NVIDIA) or the KMM/kaniko/ubuntu build images (AMD) -- the rest of the operator still ships as container images that must be mirrored into the content bundle. Cover the easy-to-miss pieces: NVIDIA still needs container-toolkit (unless also pre-installed on host) and the CUDA validation image, plus the k3s/rke2 containerd config env overrides for Palette Edge; AMD needs cert-manager installed first, a DeviceConfig CR with driver.enable=false, and node labelling. Recommend enumerating exact image tags via 'helm template' and note Palette content-bundle additional-images handling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump the default NVIDIA driver branch from 570 to 580 across the Earthfile ARG, install-nvidia-drivers.sh, .arg.template, and docs. Refresh point-release examples to 580.159.03. The 570 entries left in "e.g. 550, 570, 580" lists are valid-branch examples, not defaults. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… hang On modern NVIDIA data-center GPUs (Ada/Hopper/Blackwell — reproduced on HPE ProLiant Compute DL380a Gen12 with a GH100-class GPU), a CanvOS image built with INSTALL_NVIDIA_GPU_DRIVERS=true would hang forever at first boot inside systemd-udev-settle. Console showed a kernel stack trace ending at gh100_gsp_oneinit+0x2cf/0x440 [nouveau]: initramfs udev auto-loaded nouveau before the rootfs modprobe.d blacklist could apply, and nouveau's experimental GSP-RM path hung on the GPU. udev-settle never returned; the node never came up; nvidia.ko never got to load. Three changes, together sufficient to unbrick boot: * Earthfile: when INSTALL_NVIDIA_GPU_DRIVERS is true, append rd.driver.blacklist=nouveau modprobe.blacklist=nouveau nouveau.modeset=0 to /etc/cos/bootargs.cfg. This is the actual load-bearing fix: rd.driver.blacklist= is honored by dracut/initramfs *before* any udev event fires, so nouveau never binds. Mirrors the existing cgroup-v2 append pattern immediately above. * install-nvidia-drivers.sh: write /etc/dracut.conf.d/95-blacklist-nouveau.conf so BOTH the script's own dracut rebuild AND the later Earthfile-driven dracut rebuild (a) omit nouveau.ko from the initrd entirely and (b) carry the modprobe blacklist file into initramfs. Belt-and-braces in case the cmdline args are ever stripped downstream. * install-nvidia-drivers.sh: drop nvidia_drm from /etc/modules-load.d/nvidia.conf. On headless GPU nodes it serves no purpose and loading it at early boot has a class of KMS/DRM-conflict failure modes. Loads on demand if anything actually needs it. Follow-up: the UKI build path (--extend-cmdline in the build-uki targets) also needs the same nouveau cmdline args when INSTALL_NVIDIA_GPU_DRIVERS is true. Not addressed here since field failures were non-UKI grub boot; worth adding in a subsequent commit. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…upport
Flip the default flavor of the pre-installed NVIDIA kernel modules from
"proprietary" to "open". The closed modules fail RmInitAdapter (0x22:0x56:897)
on Hopper (H100/H200) and Blackwell (RTX PRO 6000 Blackwell, B100/B200/GB200);
`nvidia-smi` on such a host reports "No devices were found" and the entire
GPU Operator stack downstream stays wedged (toolkit init loops, containerd
never registers the nvidia runtime, device plugin never advertises
nvidia.com/gpu, workload pods stay Pending on Insufficient nvidia.com/gpu).
The open modules also work fine on Turing, Ampere, and Ada, so the new
default is safe for every server-class NVIDIA GPU generation Turing and
newer -- essentially every card CanvOS is likely to target. Only Pascal /
Volta (V100/P100/P40) require the closed modules and now need an explicit
NVIDIA_DRIVER_TYPE=proprietary override.
Applied consistently across:
* scripts/install-nvidia-drivers.sh -- runtime default + header comment
* Earthfile -- ARG default
* .arg.template -- user-facing hint
* docs/nvidia-gpu-airgapped.md -- config table plus new "Choosing the
module flavor" section with a hardware/flavor matrix and the RmInitAdapter
symptom description for people googling the error.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The pre-install script silently shipped broken images when AMD's amdgpu-dkms
source did not build against the image kernel (observed on Ubuntu 24.04 HWE
6.17 with the previous 7.2.4 / 30.30.4 default, whose 6.16.13 source fails at
DKMS configure). The verify step accepted the in-tree amdgpu.ko.zst as
sufficient because it only searched anywhere under /lib/modules/$KVER/, and
apt-get failures were papered over. Nodes booted using the in-tree driver
instead of the DKMS one without any signal that this had happened.
Changes:
- Introduce AMDGPU_DRIVER_SOURCE={dkms,inbox}. dkms (default) DKMS-builds AMD's
out-of-tree amdgpu; inbox skips the AMD apt repo entirely and only ensures
the kernel's in-tree amdgpu autoloads. Any other value is a hard error.
- Rename AMDGPU_ROCM_VERSION -> AMDGPU_DRIVER_RELEASE (the URL segment is a
driver-release marker, not a ROCm x.y.z). Bump default 7.2.4 -> 31.30
(amdgpu-dkms 6.19.4, covers 22.04/24.04 kernels through 6.17).
- Fail hard in dkms mode: set -eo pipefail; remove '|| true' shims around
dkms build/install; drop 'dpkg --configure -a --force-all' silent-fallback;
verify amdgpu landed under $MODDIR/updates/dkms AND that dkms status shows
': installed' for the target kernel. Failure prints make.log tail and
points at bumping AMDGPU_DRIVER_RELEASE or switching to inbox.
- Write /etc/canvos/amdgpu-driver-source on-node marker so ops can query
which mode ran and, in dkms mode, the release + module/version + kernel.
- Docs: two-modes explainer up front, snapshot mapping table (2026-07-11)
linking to AMD's canonical system-requirements page, "when the DKMS build
fails" section documenting the bump-or-fallback flow.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…build container)
amdgpu-dkms >= 6.18 (observed on release 31.30 = amdgpu-dkms 6.19.4) invokes
mokutil from the DKMS sign_tool hook to enroll a Machine Owner Key. mokutil
reads /sys/firmware/efi/efivars, which is not exposed inside Docker/Earthly
RUN containers, so it aborts with:
"EFI variables are not supported on this system /
/sys/firmware/efi/efivars not found, aborting."
and the amdgpu-dkms postinst returns non-zero, failing the base-image build.
Drop /etc/dkms/framework.conf.d/canvos-no-mok-signing.conf with sign_tool=""
immediately before apt-get install amdgpu-dkms. DKMS honors this and skips
signing entirely, sidestepping the mokutil invocation. Modules are unsigned;
this path already documents that Secure Boot / UKI is unsupported.
The step-5 error message now lists this as the third common failure mode so
future regressions surface with a pointer at the drop-in.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…tor v1.5.0) Empirical reproduction on the kairos-ubuntu:24.04-core-amd64-generic-v4.0.4 base image (kernel 6.17.0-35-generic) with the AMDGPU_ROCM_VERSION=7.2.4-era script + Abhijit's masking reverted showed that amdgpu-dkms 6.16.13 builds cleanly against Linux 6.17: DKMS reports the module `installed`, all 8 amd*.ko files land under /lib/modules/<kver>/updates/dkms/, and apt-get returns 0. The "EFI variables are not supported on this system / /sys/firmware/efi/efivars not found, aborting" line printed during the 30.30.x postinst is a cosmetic mokutil warning (sign_tool step); the postinst continues past it and finishes normally. The signing config drop-in from ad4c6fa is still valuable as defensive belt-and-suspenders for the 31.x line where the same message is treated as fatal. Given that, revert the previous 31.30 default (which crossed AMD's tech- preview boundary) back to 7.2.1 -- the release AMD's GPU Operator v1.5.0 release notes explicitly pair the operator with. AMD publishes both ROCm-alias and driver-release-marker URL segments at repo.radeon.com/amdgpu-install/, so "7.2.1" is a valid AMDGPU_DRIVER_RELEASE (equivalent to "30.30.1"). Also refactor the docs' "Version alignment" section to lead with the operator → ROCm → driver → kernel chain and explicitly label the 31.x line as tech-preview. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…lure Earthly's -i interactive debugger frequently hands out a broken tty in the buildkit sandbox, making /var/lib/dkms/*/build/make.log unreachable when amdgpu-dkms's postinst fails. Instead of asking users to debug interactively, tail the make.log(s) and a few environment probes (kernel headers state, Module.symvers presence, sign_tool drop-in state, MemAvailable) to stderr before die-ing. The next non-interactive rebuild surfaces the underlying compile error directly in the Earthly output. No behavior change on the happy path. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
amdgpu-dkms's ./configure runs a Kbuild probe to detect CFLAGS. In buildkit's
default unprivileged sandbox that probe fails with
configure: error: cannot detect CFLAGS...
grep: amd/dkms/config/config.h: No such file or directory
make: *** No rule to make target 'amd/dkms/config/config.h'
Reproduced against a plain `docker run --privileged` baseline: same script,
same base image (kairos-ubuntu:24.04-core-amd64-generic-v4.0.4), same kernel
(6.17.0-35-generic), same AMDGPU_DRIVER_RELEASE=7.2.1 -- succeeds end-to-end
with modules landing under /lib/modules/<kver>/updates/dkms/. The only
observed difference is --privileged, so mark the AMD RUN step explicitly.
The wrapper (./earthly.sh) already passes --allow-privileged, and other RUN
steps in this repo use --privileged similarly, so no downstream change needed.
NVIDIA's driver install doesn't use the same ./configure sanity-check step
and continues to work unprivileged; its RUN block is left as-is.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
amdgpu-dkms's ./configure fails silently at "cannot detect CFLAGS..." when Kbuild can't complete the `make -f -` heredoc probe it uses to extract KBUILD_CFLAGS. build-essential covers gcc/make/libc but modern kernel Makefiles unconditionally reach for bc, bison, flex, libelf-dev, libssl-dev, and pahole (from dwarves). --no-install-recommends skips all of those. Enumerate them explicitly to keep the image lean without breaking the AMD DKMS build. dwarves is AMD's own `Recommends:` on amdgpu-dkms -- it just wasn't pulled because we use --no-install-recommends. cpio + xz-utils are added defensively for the initramfs trigger that runs after DKMS install. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ild) Compiles amdgpu-dkms against the kairos base image's kernel in a host-side `docker run --privileged` container, then tars the produced kernel module + firmware + config drop-ins into build/amdgpu-artifact-<release>-<kver>-<digest>.tar.gz. The main Earthly build consumes that artifact via COPY, sidestepping the buildkit RUN sandbox limitation that breaks AMD's ./configure heredoc probe. Cache by (base-image digest x kernel version x AMDGPU_DRIVER_RELEASE); repeat builds skip the ~8-10 min compile. Cache miss reproduces the same behavior we already verified end-to-end via direct docker run on the jump host. Prints "AMDGPU_ARTIFACT_PATH=<abs-path>" on the last stdout line so the earthly.sh wrapper (next commit) can pick it up and thread it into Earthly. Standalone; no other files touched by this commit. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Before dispatching to earthly (build_with_proxy/build_without_proxy), detect INSTALL_AMD_GPU_DRIVERS=true + AMDGPU_DRIVER_SOURCE=dkms from either .arg or CLI overrides, derive BASE_IMAGE via the same tag formula the Earthfile uses, and invoke scripts/prebuild-amdgpu-artifact.sh to compile the amdgpu module on the host. The last line of prebuild output is parsed for the artifact path; it's threaded to earthly as --AMDGPU_ARTIFACT_PATH=<abs>, which the companion Earthfile change consumes via COPY + tar-extract. Inbox mode skips this step entirely. Non-AMD builds skip this step entirely. Existing CLI/.arg semantics for INSTALL_AMD_GPU_DRIVERS, AMDGPU_DRIVER_SOURCE, AMDGPU_DRIVER_RELEASE, and BASE_IMAGE are preserved unchanged. Ubuntu-only guard: prebuild refuses OS_DISTRIBUTION != ubuntu with a clear error, mirroring the mutual-exclusion / Ubuntu-only checks already in the Earthfile for the AMD block. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…+ docs
Complete the prebuild-artifact flow started in the previous two commits.
Earthfile:
* new ARG AMDGPU_ARTIFACT_PATH (empty default)
* base-image AMD block now branches on (AMDGPU_DRIVER_SOURCE, artifact):
- dkms + artifact present -> COPY tarball, extract, depmod, initrd
(fast path used by ./earthly.sh + prebuild helper -- the release path)
- inbox, or dkms without artifact -> unchanged legacy path (works under
`docker run --privileged`; fails under buildkit's RUN sandbox at
AMD's ./configure -- documented)
scripts/install-amdgpu-drivers.sh:
* new AMDGPU_ARTIFACT_PATH fast path before the in-buildkit DKMS block:
extract tarball, verify updates/dkms/amdgpu.ko*, depmod, rebuild initrd,
write the on-node marker at /etc/canvos/amdgpu-driver-source. In-buildkit
DKMS legacy path retained for direct `docker run` invocations and for
scripts/prebuild-amdgpu-artifact.sh's own inner container.
* docstring updated to describe the three execution paths.
docs/amd-gpu-airgapped.md:
* new "two-stage build" explanation covering why the prebuild exists
(buildkit RUN sandbox breaks AMD ./configure) and how it's wired up
behind ./earthly.sh.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…d'd) The prebuild's inner container bind-mounts the install scripts read-only (-v ...:ro). chmod +x fails on RO mounts. Bash can execute scripts without +x when invoked as `bash <path>`; no functional difference otherwise. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previous verify path deleted the tarball via STAGE_DIR trap before the user
could inspect it. Now:
* mv to final path FIRST (before verify), so any failure leaves the file
behind for inspection
* on verify failure, dump the first 40 tar entries to stderr so we can
see what path structure actually landed in the archive
* grep pattern loosened to `updates/dkms/.*amdgpu\.ko` (regex) in case
tar mangled paths in an unexpected way
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ld context) Earthly's COPY expects paths relative to the Earthfile's directory (the repo root), not absolute host paths. The prebuild helper emits an absolute path for its own file-existence checks; the wrapper now converts that to a repo-relative path before threading it into Earthly as --AMDGPU_ARTIFACT_PATH=build/amdgpu-artifact-...tar.gz. Fails fast with a clear message if the artifact somehow lands outside the repo tree. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`build/*` in .earthlyignore was hiding the pre-built AMD driver tarball from Earthly's build context, so COPY build/amdgpu-artifact-...tar.gz failed with "no such file or directory" even though the artifact existed on the host. Add a negation pattern for build/amdgpu-artifact-*.tar.gz so this specific artifact is visible while everything else under build/ (Earthly SAVE ARTIFACT outputs, etc.) stays ignored. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ueue
Reported: node with baked-in DKMS amdgpu (8x MI325X) fails to boot -- drops
into dracut-emergency after `dracut-initqueue: Timed out for waiting the udev
queue being empty`. Screenshots show amdgpu still emitting per-XCP-partition
and per-ring init messages long after boot has given up.
Root cause: rebuilding the initrd (default true) pulled the DKMS amdgpu
modules under /lib/modules/<kver>/updates/dkms/ into the initrd. amdgpu then
loads *before* switch-root, and the sheer volume of udev events it emits on
multi-GPU MI silicon (8 GPUs x 8 XCP partitions + JPEG rings + SMU/PSP/KFD
handshakes) keeps systemd-udev-settle from ever seeing an empty queue.
initqueue times out (~5 min), boot fails, node reboots, loops.
amdgpu is not required to mount rootfs -- NVMe/SATA use their own drivers.
Load it *after* switch-root via /etc/modules-load.d/amdgpu.conf (which the
script was already writing), where there is no timeout pressure. This
matches kairos-a176's observed behavior with the in-tree driver.
Two guards:
1. Flip default AMDGPU_REBUILD_INITRD to `false` (was `true`) in
Earthfile, .arg.template, and install script.
2. Even when someone sets AMDGPU_REBUILD_INITRD=true, drop
/etc/dracut.conf.d/98-canvos-amdgpu-omit.conf with
`omit_drivers+=" amdgpu amdttm amdkcl amd-sched amddrm_ttm_helper
amddrm_buddy amddrm_exec amdxcp "` so dracut still leaves amdgpu out.
Applied consistently across all three code paths in install-amdgpu-drivers.sh
(artifact fast path, inbox mode, in-buildkit DKMS fallback).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…687) * feat: support installer boot from USB media on Renesas xHCI hardware Bundle the Renesas xHCI (USB 3.0) host controller driver into the initramfs and disable PCI reallocation on the installer boot entries so the Palette Edge installer can boot and install from USB media on hardware using that chipset. - Earthfile: write /etc/dracut.conf.d/99-usb-media.conf in base-image before the distro dracut regeneration so xhci_pci_renesas is included in the generated initramfs (add_drivers + force_drivers). - grub.cfg: add pci=realloc=off to the installer menu entries. * feat(fips): bundle Renesas xHCI driver into initramfs for USB media boot The Earthfile skips dracut regeneration for FIPS builds, so the USB media driver config must be added in the ubuntu-fips Dockerfiles where kairos-init regenerates the initramfs (-s init --fips). Add a 99-usb-media.conf (xhci_pci_renesas via add_drivers + force_drivers) to each Ubuntu FIPS version (20.04/22.04/24.04) and COPY it into /etc/dracut.conf.d/ before the kairos-init init step that runs dracut.
Installer ISO was hanging on the same NVIDIA data-center GPUs previously addressed for the installed OS in 1c3a089: nouveau auto-loads in the live ISO's initramfs before switchroot, hits GSP init, and stalls udev-settle indefinitely. Runtime bootargs.cfg had the fix; installer ISO did not. Also block qat_4xxx unconditionally: Xeon Scalable 4th/5th gen hosts with Intel QuickAssist devices load qat_4xxx in initramfs and its probe / firmware-load path adds minutes to boot. CanvOS does not consume QAT acceleration, so blacklisting is safe on every build. Three sites, one arg pair each (rd.driver.blacklist= is the load-bearing flag — dracut honors it before udev fires; modprobe.blacklist= is belt-and-braces for post-switchroot): * overlay/files-iso/boot/grub2/grub.cfg: append rd.driver.blacklist=nouveau,qat_4xxx modprobe.blacklist=nouveau,qat_4xxx nouveau.modeset=0 to all three menuentries (default install, manual install, interactive install). Unconditional — the same ISO installs onto NVIDIA and non-NVIDIA hardware, and blacklisting an absent module is a no-op. * Earthfile CMDLINE default: same three args appended, so enki --extend-cmdline bakes them into the UKI stub for both signed and unsigned installer paths. Addresses the follow-up flagged in 1c3a089 for the UKI build path. * Earthfile base-image bootargs.cfg block: new unconditional RUN for qat_4xxx. Two-branch sed — if the nouveau block above already added rd.driver.blacklist=, splice ,qat_4xxx onto the existing list rather than duplicating the arg key (some cmdline parsers treat repeated keys as last-wins and silently drop the earlier value). Otherwise add a fresh rd.driver.blacklist=qat_4xxx modprobe.blacklist=qat_4xxx pair. Idempotent via outer grep guard. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…GX/DGX/GB200 HGX H100/H200, HGX B200, DGX and GB200 need nvidia-fabricmanager to bring up multi-GPU NVLink; GB200 NVL72 additionally needs nvidia-imex for multi-node NVLink Sharp. Both daemons are userspace, have no kernel side effect, and exit cleanly on hardware that does not need them (fabricmanager: "No NvSwitch found" -> failed/inactive; imex: no nodes_config.cfg -> exits, unit inactive), so default-on is safe on non-target fleets at a ~70-110 MB image cost. Also install libnvidia-nscq-<branch> explicitly alongside fabricmanager -- it is a transitive dep today, listed explicitly so the build fails loudly if the CUDA repo ever drops the auto-dep. IMEX is best-effort: pre-570 driver branches do not publish the package and the install step warns and skips. Opt-out per node type via --NVIDIA_INSTALL_FABRICMANAGER=false / --NVIDIA_INSTALL_IMEX=false. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…esent
The nvidia-fabricmanager-<branch> package ships a unit whose ExecStart is a
wrapper (/usr/share/nvidia/fabricmanager/nvidia-fabricmanager-start.sh) that
NVIDIA added to gate FM startup on the IB fabric being up -- needed for
GB200 NVL72 multi-node NVLink Sharp. The wrapper shells out to `ibstat`
(from infiniband-diags) to probe IB device state; without that binary the
unit fails immediately on every boot, regardless of hardware, with:
"ibstat" command not found! Please install "ibstat".
nvidia-fabricmanager.service: Control process exited, code=exited, status=1
Install infiniband-diags whenever fabricmanager is installed. ~5-8 MB, no
runtime side effect on non-IB hosts. On non-NVSwitch hardware the wrapper
now proceeds to nv-fabricmanager which exits "No NvSwitch found" as
originally expected -- the unit stays failed with no kernel effect and no
restart loop.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… HGX/DGX
NVIDIA's 570+ nvidia-fabricmanager-<branch> unit invokes a wrapper --
/usr/share/nvidia/fabricmanager/nvidia-fabricmanager-start.sh -- that gates
FM startup on the GB200 NVL72 NVLink subnet coming up. The wrapper shells
out to `ibstat` (infiniband-diags) and `nvlsm` (NVIDIA Subnet Manager,
shipped in UFM/MLNX-OFED, not in this package). Installing infiniband-diags
gets past the first gate but the wrapper then dies on missing nvlsm, on
every non-NVL72 topology (standalone HGX H100/B200, single-node DGX,
workstations). FM never runs on any of that hardware.
Install a systemd drop-in at
/etc/systemd/system/nvidia-fabricmanager.service.d/10-bypass-wrapper.conf
that overrides ExecStart back to calling nv-fabricmanager directly (the
pre-wrapper behavior). FM then correctly initializes NVSwitch on HGX/DGX,
and cleanly exits "No NvSwitch found" on non-NVSwitch hosts.
GB200 NVL72 deployments must remove this drop-in via user-data so the
wrapper's IB/nvlsm precondition applies. infiniband-diags is retained
defensively (~5-8 MB) so a user reverting to the wrapper doesn't
re-encounter the ibstat gate.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… all hardware The NVIDIA 570+ fabricmanager unit invokes a wrapper (/usr/share/nvidia/fabricmanager/nvidia-fabricmanager-start.sh) that probes `ibstat` (infiniband-diags) and `nvlsm` (NVIDIA Subnet Manager) before starting nv-fabricmanager -- needed to gate on the GB200 NVL72 NVLink subnet coming up. Without both binaries the unit dies before FM is ever invoked. Prior fix bypassed the wrapper via a systemd drop-in. That worked, but diverges from the vendor path and would break GB200 NVL72 deployments that legitimately need the IB/nvlsm precondition. Verified on a live HGX host that installing nvlsm (2025.10.14-1, from the CUDA repo we already add -- unversioned package name) lets the vendor wrapper succeed on standalone HGX topologies too. FM comes up and GPU Operator pods reach Ready. Install nvlsm alongside fabricmanager + libnvidia-nscq + infiniband-diags, and drop the 10-bypass-wrapper.conf systemd override. Same vendor code path on HGX/DGX and GB200 NVL72 alike. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…led OS Follow-up to 83cfcf6. The previous nouveau block was gated on INSTALL_NVIDIA_GPU_DRIVERS=true (inherited from 1c3a089), which meant non-NVIDIA CanvOS builds that later landed on NVIDIA hardware still hung in GSP init on first boot. The image doesn't know its target hardware at build time — the hang is a property of the host, not the build flag — so blacklisting must be unconditional. Same reasoning as qat_4xxx, which was already unconditional in the same block. Collapses the two prior sed passes (gated nouveau + unconditional qat_4xxx-splice) into one unconditional sed that appends both blacklists plus pci=realloc=off in a single append, mirroring the installer ISO grub.cfg cmdline. Idempotent via the grep guard on "rd.driver.blacklist=nouveau". pci=realloc=off pins the PCI BAR layout to firmware assignments; kernel-side reallocation has caused resource conflicts on some server platforms, and the installer ISO already carries it. UKI CMDLINE default reverted — UKI installer path is out of scope for this change per user direction. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.