test(e2e-proxy): deflake §4 — retry the squid Service-readiness race#346
Merged
Conversation
…+ raw log on failure §4 (the #119 ingestion-egress guard) flaked red on develop (run 29255451968); a re-run of the SAME commit (run 29340987746) passed — a transient, not a #341 regression (#341 never touched this test). Cause: the app pod's section-A curl reaches the in-cluster squid via its Service, but `kubectl rollout status` gates only on the squid pod being Ready, not on its Service endpoints being programmed in kube-proxy. In that window a brand-new pod's first CONNECT to the ClusterIP is refused, and curl gave up before attempting the tunnel (no "Establish HTTP proxy tunnel" line) — so the assertion false-failed. - Add `--retry 5 --retry-connrefused --retry-delay 2` (-m 30) to section A's curl: it rides out the transient connection-refused. A genuine #119 regression still fails all retries, so the guard keeps its teeth. - On the (A) assertion failure, dump the RAW egress-app log + squid pod/endpoints state. This flake surfaced only the filtered B line and a bare "did NOT tunnel"; the raw dump makes any future failure debuggable instead of opaque. bash -n + shellcheck clean. (The k3d e2e can't run locally; CI is the authority.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
👋 Heads-up — Code review queue is at 43 / 30 Above the WIP limit. The team convention is to review existing PRs before opening new work. Open PRs currently in Code review (oldest first):
Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.) |
saadqbal
approved these changes
Jul 14, 2026
saadqbal
added a commit
that referenced
this pull request
Jul 14, 2026
…digest pinning (#345) * test(provision): pin the #303 pre-flight's grep contract with hidden `client list` (Refs #141) (#335) The installer's #303 one-client-per-machine pre-flight (`scripts/lib/provision.sh` `_account_owns_namespace`) shells out to `tracebloc client list --plain` and greps the output for `namespace=<ns>([[:space:]]|$)` to refuse a cross-account re-provision. In the cli repo `client list` is now a HIDDEN cobra command — still callable, but nothing here pinned that the pre-flight keeps classifying the CLI's exact --plain output correctly, nor that it still passes --plain. If either drifts the grep silently fails and #303 stops firing. Add two focused unit tests of `_account_owns_namespace` (the consumer half of the cli#141 contract; the producer half is pinned in the cli repo at internal/cli/client_list_contract_test.go). The fixture mirrors, field-for-field, what cli's runClientList prints under --plain: - owned namespace → rc 0; absent → rc 1; a STRICT PREFIX of a real namespace → rc 1 (pins the ([[:space:]]|$) anchor: an account must not "own" acme-prod-0 just because it owns acme-prod-01); - the pre-flight actually invokes the hidden list WITH --plain; - an unreadable list is rc 2 (fall through to create), distinct from rc 1 (refuse) — the distinction #303 branches on. Assertions use `[ -eq ]` + `grep` (real exit codes, robust on bash 3.2); Linux CI (standard-checks.yml → `bats scripts/tests/*.bats` on ubuntu) is the authority. Mutation-verified: dropping the grep anchor or --plain from provision.sh fails these tests. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(proxy): NO_PROXY covers the cloud metadata IP 169.254.169.254 (#337) The auto-augmented NO_PROXY lists (chart tracebloc.proxyEnv helper, bash _augment_no_proxy defaults, Windows Get-EffectiveNoProxy defaults) omitted 169.254.169.254 — behind a corporate proxy, cloud metadata lookups would be routed through the external proxy (broken IMDS access and an unnecessary place for instance credentials to transit). Add the IP to all three lists in lockstep, pin it in the bats + Pester + helm-unittest expectations, regenerate manifest.sha256 for the two touched installer scripts, bump chart 1.9.3 -> 1.9.4. Ref tracebloc/backend#803 (item J), tracebloc/client-runtime#120 Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * feat(installer): stop-and-check gate — recognize an existing install (#339) * feat(installer): stop-and-check gate — hand a healthy re-run to the home screen Re-running the installer on an already-set-up machine no longer drags the user through full provisioning. A new read-only gate (scripts/lib/assess.sh) runs after the banner and classifies the machine: • healthy — cluster running AND a tracebloc release present AND jobs-manager Ready AND the CLI present → print "Already set up on this machine", hand off to `tracebloc` (the home screen), exit 0. • degraded — cluster stopped / workload not Ready / CLI missing / any partial state → print an honest one-liner, fall through to the normal flow to reconcile. • fresh — no cluster, or a cluster with no release → the normal flow. assess is STRICTLY non-mutating and bounded: read-only `k3d cluster list`, `helm list`/`get values` (reuses detect_installed_client), and a bounded `kubectl get` (--request-timeout). On ANY uncertainty it degrades toward the normal flow — never a false "healthy" that would skip a needed install. The hand-off uses `exit 0` (not exec) so the EXIT-trap cleanup still runs; if the CLI is somehow unresolvable it falls back to a status line and still exits 0. --force / --reinstall (or TRACEBLOC_FORCE_REINSTALL=1) bypasses the gate. A healthy machine still short-circuits under curl|bash (output only, no input). Per-layer surgical repair of a degraded machine is a deliberate fast-follow so this PR stays off provision.sh (clear of the active #838 work). Wires the gate into main() (after print_banner, before the roadmap), adds scripts/lib/assess.sh to the bootstrap FILES + gen-manifest, regenerates scripts/manifest.sha256, and adds scripts/tests/assess.bats. New copy says "secure environment", never "client". PR 2 of 2 (PR 1 = the cli home screen, cli#244). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): assess "healthy" = ALL client workloads Ready + guard awk branch Code-review should-fix items on the stop-and-check gate: 1. _assess_cluster_servers_running: add `|| line=""` to the awk-branch assignment, mirroring the jq branch. awk's `exit` closes the pipe, so under `set -o pipefail` a SIGPIPE from k3d (141) — or any k3d failure — would otherwise propagate non-zero out of the assignment and abort the installer. 2. "healthy" must match the installer's OWN definition of ready. The probe now requires ALL the workloads wait_for_client_ready checks — mysql-client, ${ns}-jobs-manager, ${ns}-requests-proxy — not jobs-manager alone. Previously a machine with jobs-manager up but requests-proxy (training egress) or mysql-client down was classified healthy and short-circuited without reconciling — the false-positive we designed against. Single source of truth: the deployment set is extracted into _client_workload_deployments (common.sh); both wait_for_client_ready (summary.sh) and the assess gate consume it, so they can't drift. Renamed _assess_jobs_manager_ready -> _assess_workload_ready (reason stays `workload-not-ready`); any one workload not-Ready/absent -> degraded. Tests: _assess_workload_ready now covers all-three-Ready plus each workload individually down/absent; classify covers "one down -> degraded" via the real probe and "all three Ready + CLI -> healthy". Mutation-checked: shrinking the shared list to jobs-manager-only fails the mysql-client/requests-proxy/one-down tests. Manifest regenerated (assess.sh, common.sh, summary.sh). provision.sh untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): run the helm client install behind a spinner (no more silent ~10-15s) (#333) * fix(installer): run the helm client install behind a spinner (no more silent ~10-15s) Step 4/5 ran `helm upgrade --install` (and the in-place reconcile) blocking, with all output redirected to the log, so the terminal sat frozen ~10-15s (render + apply + image pull) with no feedback. Wrap both in the existing spin_cmd helper: an animated spinner + message, output still streamed to $LOG_FILE, and on failure the log tail to stderr + the same error-exit. Honours RFC-0002 §2 "progress on every wait" (the principle already applied to the CLI's post-submit waits). - bash -n + shellcheck clean; install-client-helm.bats green (33 tests). - Failure path preserved (error-exit + full output in $LOG_FILE). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(scripts): refresh integrity manifest for spinner change gen-manifest.sh after the install-client-helm.sh spinner edit; static-analysis CI gate requires the sha256 manifest to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * chore(chart): prod-only ingestor digest pin (#1028 item 1) (#334) * chore(chart): prod-only ingestor digest pin via values-prod overlay (#1028) Prod deploys the ingestor image by immutable digest; dev/staging keep floating :0.7 (imagePullPolicy=Always). Base values.yaml keeps digest empty; the new client/values-prod.yaml sets only images.ingestor.digest and is layered at prod-release time (-f values.yaml -f values-prod.yaml). Digest verified live against ghcr.io (multi-arch index, amd64+arm64). Adds resolve-ingestor-digest.sh to re-resolve/verify the digest at each cut instead of hand-typing it. Refs tracebloc/backend#1028. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(digest): --write refuses single-arch pin + verifies sed landed Multi-arch guard now inspects the resolved repo@digest and hard-fails under --write instead of only warning (a single-arch pin breaks arm64 and fails helm-ci ingestor-multiarch). --write also greps the overlay for the digest after sed and errors on a silent no-op instead of falsely reporting success. Bugbot findings on client#334. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(digest): read chart tag without yq; fail loudly, never hardcode 0.7 The no-arg TAG default silently fell back to a hardcoded 0.7 when yq was absent, so a prod cut after the chart tag moved could --write a stale/wrong digest while appearing to follow the chart. Reads images.ingestor.tag via yq when present, else a portable awk parse scoped to the images:->ingestor: block; if neither works it errors and exits non-zero. Bugbot follow-up on client#334. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(digest): tolerate pipefail in the multi-arch platforms capture Under set -o pipefail, a failed inspect or a grep -v that filters every line exits non-zero and aborted the whole script even after the digest resolved. Add || true; an empty platforms then trips multiarch=0 so the guard still fires (ERROR under --write, WARNING otherwise). Bugbot round-3 on client#334. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * docs(security): update §8.3 — backend tokens now bounded + revocable (#302) (#342) §8.3 claimed "Backend tokens never expire"; that finding is now mitigated for interactive DS web sessions via a bounded, revocable 30-day ClientAccessToken (backend#933 + frontend-app#575, shipped through backend#590). Document the mitigation, the intentional carve-out for edge devices / bots (which keep the legacy long-lived DRF Token as non-interactive service credentials, by design), and the residual JS-readable-storage risk tracked as the SEC-06 follow-up in tracebloc/backend. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): Windows one-client guard reads clientId via ConvertFrom-Json (#200) (#340) The one-client-per-machine guard regex-scraped `helm get values` YAML and only matched a DOUBLE-QUOTED clientId. Helm re-serializes values on `get`, so typical clientIds come back unquoted -- the guard silently matched nothing and a re-install could re-point the machine to a different Client ID, defeating the protection from #192 (Bugbot finding on #199). Read the values as JSON instead (`helm get values <rel> -n <ns> -o json | ConvertFrom-Json` -> `.clientId`), which sidesteps YAML quoting entirely and matches the adjacent comment that already claimed ConvertFrom-Json. A release with no user values (literal `null`), a missing clientId key, or unparsable output from one release is skipped without aborting the scan of the remaining releases. Tests: existing guard mocks now serve real-helm-shaped output (JSON for `-o json`, unquoted YAML otherwise), plus new cases for unquoted / single- quoted / double-quoted YAML views, a null-values release mid-scan, and values without a clientId key. Pester (Linux container, lts-7.4): 95 passed, 0 failed, 6 skipped ($IsWindows skips). scripts/manifest.sha256 regenerated (install-k8s.ps1 line only) -- the file is on the signed-manifest surface; gen-manifest.sh --check passes. Closes #200 Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * docs(supply-chain): document the Windows bootstrap (install.ps1) (#336) * docs(supply-chain): document the Windows bootstrap (install.ps1) SUPPLY_CHAIN.md scoped itself to install.sh with zero Windows mentions, while the R8 Windows leg (PR #299, released v1.8.5) has shipped the same guarantee for install.ps1. Add the Windows section, the install.ps1 release asset to the asset table, and a PowerShell verify-by-hand note. Closes tracebloc/backend#957 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(supply-chain): correct the raw-<TAG> Windows bootstrap (fails closed w/o $env:REF) Arturo's review (client#336): the §7 parenthetical told customers they could bootstrap install.ps1 from a pinned <TAG> raw URL, but the committed tag tree still ships the __TRACEBLOC_RELEASE_REF__ placeholder (only the release asset is stamped), so that path refuses to run unless $env:REF is set. Documented the $env:REF requirement + why, instead of implying the raw <TAG> URL works as-is — aligned with §1's $env:REF description and the bash side (which uses raw-<TAG> only for hand-verifying a sub-script against the manifest, not for bootstrap). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * feat(installer): first-time install run-through (1. Downloading / 2. Installing a–f) + healthy re-run bailout (#341) * feat(installer): first-run banner/roadmap + step_header + count_bar helpers - print_banner: new title 'Setting up tracebloc on your machine · <version>' (tracebloc bold-cyan) + rule; TB_VERSION from TRACEBLOC_INSTALL_REF; skips when the bootstrap already drew it (TRACEBLOC_BANNER_SHOWN). - print_roadmap: the '2. Installing' a–f plan. - step_header: bold gerund running headers for steps a–f. - count_bar: honest N-of-M render helper for multi-image pulls. - preflight_sudo: step-b password intro copy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): bootstrap early-bailout + "1. Downloading" run-through - Early bailout: if the tracebloc CLI is present and `tracebloc doctor` reports healthy (bounded, exit-code gated), print the healthy line and exec the home screen — skipping the download entirely. Skipped on --force/--reinstall, TRACEBLOC_FORCE_REINSTALL, the dev/unverified path, or an explicit REF/BRANCH. - Draw the first-run banner here and export TRACEBLOC_BANNER_SHOWN (so install-k8s.sh does not draw a second) + TRACEBLOC_INSTALL_REF. - "1. Downloading" copy: Installer downloaded — N files / Verifying it's authentic (cosign)… / Signature verified / All N files intact — nothing was altered. Local colour palette (common.sh not yet sourced). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): step a — collapsed hardware summary + connectivity spinner - run_preflight renders the run-through's step-a view: one hardware line ('arch · N CPU cores · N GB memory · N GB free disk'), a connectivity spinner + combined 'Connected: …' line, and a 'Local storage (~/.tracebloc)' line. - PF_QUIET_SUCCESS suppresses the per-check ✔ lines only inside run_preflight (folded into the summary); called directly (bats), the checks still print their ✔/info lines, so the unit contracts hold. Warnings + hard-fails always print. - Connectivity probes stay in the foreground (PF_HARD_FAIL propagation) with a no-sleep per-host spinner frame. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): step b — Docker %-bar copy + 'ready' success lines - Real %-by-bytes bar for the Docker Desktop .dmg (single-file curl via download_with_progress); fresh-Mac intro copy on the label. - 'Docker ready' and 'System tools ready (k3d, helm, kubectl)' to match the run-through. Linux (Docker Engine) copy left untouched — a deliberate follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): step c — spinner around the (silent) k3d create - _create_new_cluster wraps the 1-2 min k3d create in a spinner ('Creating your secure environment…'), the real fix for the long silent gap; spin() waits for the backgrounded create so exit-code capture + proxy-config cleanup are intact. - Runtime intro copy; terminology 'compute environment' → 'secure environment'. - _wait_for_api owns the single 'Secure environment ready' (API confirmed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): step d — device-flow sign-in copy + "Registered as" - Reworked sign-in copy: "open the link on this or any device and enter the code" (print-only; the CLI prints the URL/code/wait — no auto-open claim). - Success line → Registered as "<slug>" (the minted namespace = dashboard name). - CLI install call removed (now step b); keeps the has-tracebloc FATAL guard. - KEPT the interim name/location prompt (deployed CLI still hard-requires --name without a TTY, backend#992) + comment: remove when cli#137 ships. - No internal step header (main() prints "d) Registering this machine"). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): step b — "tracebloc CLI ready" on the fully-clean path Fully-clean verdict (usable now + in new terminals) → "tracebloc CLI ready … verified on your PATH", matching the run-through's Docker/System-tools/CLI "ready" pattern. Edge-case lines stay "installed" (installed but not yet usable here). Test assertion updated to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): step e — services count-bar + "tracebloc installed" - Services intro copy (training runner / data manager / live monitor / local database; runs on your machine, data never leaves). - _download_services_progress: honest N-of-M count bar as service images pull (imageID-populated count), bounded + non-fatal; guarded by TB_NO_SERVICE_PROGRESS (set in the bats setup so the mocked-kubectl poll cannot hang). Never a fabricated aggregate %. - Success line "Connected to tracebloc" → "tracebloc installed" (step f/summary owns "Connected"); dropped the internal step 4/5 headers (main owns a–f). - Tests: retarget the 3 assertions + guard the poller. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): step f — rich connected summary matching the run-through - connected summary: "✔ Connected to tracebloc"; Environment/Version/Mode block; "live 🟢" + dashboard; NON-dim "What's next" with 3 numbered steps (tb data ingest / my-use-cases / invite collaborators); prominent "Run tracebloc to get started."; dim footer (Logs · Data) with the reboot tip as the LAST dim line. Dropped the green ━━━ border. - Terminology: "secure environment"; trust claim "never leaves this machine". - _reboot_note stays OS-guarded (Linux: restarts automatically; macOS: open Docker Desktop) — Linux not regressed. - wait_for_client_ready intro reframed as step-f "Connecting…". - Tests updated to the new copy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): main() reorg to the six-step run-through (a–f) + gate slot - main() now: banner → [client#339 gate SLOT] → roadmap → a) Check your machine → b) Install what tracebloc needs → c) Create your secure environment → d) Register this machine → e) Install tracebloc → f) Connect to the tracebloc network, each with a step_header + trailing blank-line pair. - Step b now owns prerequisites AND the tracebloc CLI (moved out of provisioning; step d needs it to sign in). - Gate slot: guarded NO-OP call to assess_existing_install, clearly commented as client#339's — logic NOT implemented here; positioned after banner/before roadmap so it reconciles cleanly with that branch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(installer): regenerate manifest.sha256 for the first-run UX changes Content hashes for the 9 edited sub-scripts (common/preflight/setup-macos/ cluster/provision/install-cli/install-client-helm/summary/install-k8s). No files added or renamed, so install.sh's FILES array is unchanged; gen-manifest.sh --check passes (both bootstrap-sync checks green). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(installer): cover the first-run UX logic (Linux CI is the authority) - common.bats: count_bar (render/clamp/bad-input/divide-by-zero), step_header, print_roadmap (a–f plan), print_banner (version + bootstrap-suppression). - preflight.bats: _pf_hw_summary_line, connectivity combined 'Connected:' line, run_preflight collapsed step-a view (per-check ✔ lines folded away). - install-client-helm.bats: _download_services_progress guards (TB_NO_SERVICE_ PROGRESS / no-kubectl / empty-ns) so the poller can never hang the suite. - install-bootstrap.bats: early bailout — healthy doctor execs home screen (no download), unhealthy does not bail, --force skips the bailout. Not run locally (macOS [[ ]] blindspot + spin/sleep/read block without a TTY); assertions written fail-loud against the exact emitted copy for Linux CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(installer): retarget two assertions to the first-run copy Linux CI caught two stale assertions the local checks couldn't (macOS [[ ]] blindspot): - install-bootstrap happy path grepped the old "installer files verified" line, now "All N files intact — nothing was altered". - _pf_storage_type local-fs test grepped the fstype (ext4), which the first-run redesign moved into the log; the visible line is now the clean "Local storage (…)". Assert that instead. Both tests keep their original intent; no code behavior changed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): address Bugbot #341 — real tracebloc cmd, ~/.local/bin bailout probe, exec fallthrough - summary.sh "What's next" said `tb data ingest`, but there is no `tb` on PATH (the binary is `tracebloc`; the CTA two lines below already says `tracebloc`) → use `tracebloc data ingest`. - install.sh bailout probed `command -v tracebloc` before any PATH prepend, so a healthy CLI in ~/.local/bin (the installer's fallback dir, not on a fresh curl|bash PATH) was missed → forced a needless full re-download. Prepend ~/.local/bin first, mirroring provision_client. - install.sh bailout `exec tracebloc || true; exit 0` silently exited 0 if exec failed (bad interpreter / missing exec bit) — healthy machine, but no home screen and no install. Dropped `|| true`; if exec returns (i.e. it failed), print an actionable line and exit 1. Regenerated scripts/manifest.sha256 (gen-manifest.sh --check passes); retargeted the summary.bats assertion to the corrected command. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): redirect bailout CLI stdin away from the curl|bash pipe (Bugbot #341) Bugbot HIGH (learned rule): under `curl … | bash` the healthy-setup bailout ran `tracebloc doctor` and `exec tracebloc` without redirecting stdin, so the child CLI inherited the install pipe as stdin — a CLI that reads stdin could block the bailout or consume the pipe instead of behaving non-interactively. - `tracebloc doctor` (bounded health probe): stdin ← /dev/null (non-interactive). - `exec tracebloc` (interactive home-screen hand-off): stdin ← /dev/tty (the user's real terminal, not the pipe); the exec-failure fallthrough (incl. no controlling terminal to open /dev/tty) keeps the existing "couldn't launch → exit 1" path. bash -n + shellcheck clean. install.sh is the bootstrap (not in the signed manifest), so no manifest change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): probe /dev/tty openability before the bailout exec (fixes #341 CI) c19b51d's `exec tracebloc </dev/tty` is unconditional, but a readable /dev/tty device node with no controlling terminal (CI, detached sessions) still fails to OPEN — the redirect then aborts the exec, the healthy bailout prints "could not launch" + exit 1, and every job that runs install.sh (bats, unit, prereqs, path-persist) goes red. Probe openability first: `</dev/tty` when it opens, else `</dev/null` — still always execs (never the install pipe), so the hand-off and its bats coverage hold. install.sh isn't a hashed sub-script (manifest unchanged). bats install-bootstrap green; shellcheck clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Merge pull request #344 from tracebloc/test/290-check-drift-cli-invariants test(drift): pin the CLI-assumed chart contract in check-drift.sh * test(e2e-proxy): deflake §4 — retry the squid Service-readiness race + raw log on failure (#346) §4 (the #119 ingestion-egress guard) flaked red on develop (run 29255451968); a re-run of the SAME commit (run 29340987746) passed — a transient, not a #341 regression (#341 never touched this test). Cause: the app pod's section-A curl reaches the in-cluster squid via its Service, but `kubectl rollout status` gates only on the squid pod being Ready, not on its Service endpoints being programmed in kube-proxy. In that window a brand-new pod's first CONNECT to the ClusterIP is refused, and curl gave up before attempting the tunnel (no "Establish HTTP proxy tunnel" line) — so the assertion false-failed. - Add `--retry 5 --retry-connrefused --retry-delay 2` (-m 30) to section A's curl: it rides out the transient connection-refused. A genuine #119 regression still fails all retries, so the guard keeps its teeth. - On the (A) assertion failure, dump the RAW egress-app log + squid pod/endpoints state. This flake surfaced only the filtered B line and a bare "did NOT tunnel"; the raw dump makes any future failure debuggable instead of opaque. bash -n + shellcheck clean. (The k3d e2e can't run locally; CI is the authority.) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): resolve Bugbot findings on #345 (tty handoff, pinned-ref reinstall, jq split) (#347) * fix(installer): resolve Bugbot findings on #345 (tty handoff, pinned-ref reinstall, jq split) Three Cursor Bugbot findings on the develop→main PR: - High — assess.sh `_assess_handoff` ran a bare `tracebloc` with no stdin redirect. Under `curl | bash` that inherits the install pipe, so the interactive home-screen handoff could consume script bytes or block (same class as #341). Redirect </dev/tty when openable, else </dev/null — mirrors the bootstrap hand-off in install.sh; still no `exec` so the EXIT trap runs. - High — install.sh skipped its healthy early-bailout on a pinned REF/BRANCH/unverified request but never told install-k8s.sh's own stop-and-check gate. A pinned-ref re-run would download the new installer and then short-circuit to the home screen, doing nothing. Export TB_FORCE_REINSTALL whenever the bailout is skipped so the downstream gate runs the full flow. - Low — assess.sh `_assess_cluster_servers_running` reintroduced a jq/awk bifurcation; jq is not a guaranteed installer prerequisite (Bugbot #284). Collapse to the single jq-free awk path (the SERVERS column read already there). Adds bootstrap tests asserting the reinstall intent reaches install-k8s.sh; test suites green (assess 31/31, install-bootstrap 15/15). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(supply-chain): regenerate manifest.sha256 for assess.sh change Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): give the healthy-machine hand-off a real terminal on stdout/stderr (#348) Bugbot (Medium) follow-up to #347: main() runs setup_log_file (`exec > >(tee …) 2>&1`) BEFORE the stop-and-check gate, so by the time _assess_handoff runs the shell's stdout/stderr are a pipe to tee, not the terminal. The earlier fix redirected only stdin, so the interactive `tracebloc` home screen still rendered onto the tee pipe instead of a tty (the bootstrap bailout in install.sh avoids this only because it hands off before any tee redirect). Point all three streams at the terminal ($TB_TTY, /dev/tty) when it's openable — bypassing tee for the interactive screen, exactly as the bootstrap does — else fall back to </dev/null and leave stdout/stderr on the pipe (non-interactive/CI). Adopt provision.sh's TB_TTY indirection so the redirect is testable; assess.bats now proves the home screen lands on the terminal (not the pipe) on the openable path and falls back cleanly otherwise. Manifest regenerated. assess.bats 32/32. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): assess gate checks cluster is running before the Helm probe (#349) * fix(installer): check cluster is running before the Helm probe in the assess gate Bugbot (High) follow-up: _assess_classify called detect_installed_client (unbounded `helm list -A` / `helm get values`) BEFORE the cluster-servers-running check. On a stopped cluster (post-reboot or manual stop) the k8s API is down, so Helm hangs — violating the gate's "bounded, never-hang" contract — and, when it finally fails, the machine is mislabeled fresh/cluster-no-release, making the cluster-stopped path effectively unreachable on real re-runs. Reorder so the cheap read-only k3d servers-running probe runs first: a stopped cluster short-circuits to degraded/cluster-stopped without ever touching Helm, and detect_installed_client only ever runs against a live API. Adds an ordering guard test asserting the Helm probe is not invoked on a stopped cluster. Manifest regenerated. assess.bats 33/33. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(e2e-proxy): deflake §4 — retry the app-pod proxy-DNS resolution race The §4 app-pod curl used --retry-connrefused, which retries a refused CONNECT (Service endpoints not yet programmed) but NOT a DNS failure. Under CI load the squid Service name isn't yet in the new pod's resolver when curl runs, so it fails with "Could not resolve proxy" (curl exit 5) — which curl does not retry on its own — and §4 flaked red (PR #349 run 29345383969) even though the squid pod and endpoints were up. Add --retry-all-errors (covers the DNS-resolution error too) and bump to --retry 8. A genuine #119 regression still fails all retries, so the guard keeps its teeth. Test-only; not part of the installer supply-chain manifest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): fail closed on unreadable one-client guard values; fix bootstrap spinner on bash 3.2 (#350) Two Bugbot findings on #345: - Medium (install-k8s.ps1): the Windows one-client-per-machine guard treated an unreadable `helm get values` (fetch failure OR unparsable JSON) for a client-chart release the same as "no client here" — so if the ONLY installed client's values couldn't be read, $existingId stayed empty and the install proceeded, silently overwriting a client it couldn't identify (fail-open). Now record any client release whose clientId we can't read and fail CLOSED: refuse with an actionable message rather than overwrite an unknown client. A parsed release with no clientId (literal `null`) is unchanged — still not a match. Adds Pester coverage for both unreadable paths. - Low (install.sh): the early-bailout spinner stored 3-byte braille frames in a single string and indexed `${frames:i:1}` / `${#frames}` — byte-based on macOS system bash 3.2, so glyphs sliced mid-byte and rendered as garbage. Switch to an array of glyphs indexed by element (matches spin() in lib/common.sh), which is char-correct on 3.2. Manifest regenerated (install-k8s.ps1 hash). install.sh shellcheck clean, .ps1 parses cleanly. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): bound the services progress-bar kubectl calls with --request-timeout Bugbot (Medium): _download_services_progress claims to be bounded/non-blocking, but its two `kubectl get pods` calls omitted --request-timeout. The TB_PULL_TIMEOUT deadline is only checked BETWEEN iterations, so a wedged/ unreachable API makes kubectl block indefinitely — freezing step e's progress bar before the authoritative readiness gate in step f ever runs. Add --request-timeout (default 5s, overridable via TB_PROGRESS_KUBECTL_TIMEOUT) to both calls, mirroring assess.sh's bounded probe — a call that can't reach the API now returns quickly, the loop re-checks the deadline, and step e degrades to the honest "still downloading in the background" line instead of hanging. Manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): proactive hardening sweep (bounded probes, tty handoffs, fail-closed guards) (#352) * fix(installer): bound the services progress-bar kubectl calls with --request-timeout Bugbot (Medium): _download_services_progress claims to be bounded/non-blocking, but its two `kubectl get pods` calls omitted --request-timeout. The TB_PULL_TIMEOUT deadline is only checked BETWEEN iterations, so a wedged/ unreachable API makes kubectl block indefinitely — freezing step e's progress bar before the authoritative readiness gate in step f ever runs. Add --request-timeout (default 5s, overridable via TB_PROGRESS_KUBECTL_TIMEOUT) to both calls, mirroring assess.sh's bounded probe — a call that can't reach the API now returns quickly, the loop re-checks the deadline, and step e degrades to the honest "still downloading in the background" line instead of hanging. Manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): proactive hardening sweep — bounded probes, tty handoffs, fail-closed guards A comprehensive audit of the installer (bash + PowerShell) for the failure classes Bugbot has been surfacing one at a time on #345, fixing the genuine instances in one pass. Bounded probes (no more indefinite hangs against a wedged/unreachable API): - cluster.sh: `kubectl cluster-info` in the _wait_for_api gate (the 60s cap was only re-checked between iterations — a single call could block forever). - summary.sh: cluster-info / get nodes / get pods / logs on the summary + not-ready diagnostic paths. - diagnose.sh: every kubectl call in the support-bundle path (`set +e` stops aborts but NOT hangs); helm calls now run only behind a bounded cluster-info probe (helm has no --request-timeout). - gpu-plugins.sh: `kubectl get nodes` in the verify_gpu poll loop. - common.sh download_with_progress: HEAD probe (-m) + the backgrounded curl (--connect-timeout + --speed-limit/--speed-time stall abort — it was monitored only by `kill -0`, no deadline, no kill). - install.sh: every bootstrap fetch (--connect-timeout/--max-time; retry already present, so a stall becomes retriable). - install-cli.sh: the CLI-installer download (a stall now falls to "install later" instead of hanging the step). curl|bash tty handoffs (stdin is the install pipe; stdout/stderr are the tee): - provision.sh: `tracebloc login` (the credential-mint device flow) now gets the real terminal on all three streams when openable, else </dev/null — same idiom as assess.sh's hand-off. - gpu-nvidia.sh: the reboot `read` read from the EOF pipe with no `|| true`, aborting the whole installer under `set -e` right after a successful driver install; now reads /dev/tty, no-tty => no reboot. - setup-macos.sh: the Docker-arch replace prompt read the pipe (empty answer => meaningless confirm); now reads /dev/tty. Fail-closed guards (a failed check must not read as "safe to proceed"): - install-client-helm.sh detect_installed_client now reports INSTALLED_CLIENT_UNKNOWN=1 when `helm list` FAILS (vs genuinely no releases), and the one-client guard refuses rather than risk overwriting a client it couldn't enumerate. +bats coverage. - install-k8s.ps1 one-client guard: same fix at the `helm list` level (non-zero exit OR non-JSON now fails closed, matching the per-release fix from #350). +Pester coverage. set -e footgun: - common.sh _chart_version: trailing `|| true` so a no-match `grep` (no client release) can't abort callers that assign it under `set -e`. PowerShell 5.1 portability: - install-k8s.ps1: the WSL-path build used a scriptblock `-replace` (PS 6.1+); under Windows PowerShell 5.1 (the bootstrap target) the drive letter wasn't lowercased -> malformed path -> 180s NCT-install timeout. Now -match/$Matches. Deliberately NOT changed (documented): - k3d `cluster create --wait` has no --timeout in EITHER bash or PowerShell — at parity, and the known --wait hang cause (proxy misroute) is already mitigated by the proxy config. Not changing critical-path create behavior speculatively. - The dataset-mount check's inspect-failure no-op is a documented, tested design choice (cluster.bats) — left as-is. - Long tail of Linux-GPU-only driver-download curls (gpu-nvidia/gpu-amd, setup-linux) and misc PowerShell -TimeoutSec: LOW severity, retry-wrapped; left for a follow-up to keep this reviewable. All bash suites green (assess 33, install-client-helm 54, cluster 27, provision, summary, install-cli, bootstrap, preflight); shellcheck --severity=error clean; install-k8s.ps1 parses cleanly; manifest regenerated. Two pre-existing local-env bats failures (_extract_yaml_value '' escape; validate_config) are unrelated and also fail on develop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): fail closed on unknown helm state in the #303 pre-provision guard Bugbot (High) on the sweep: the sweep taught detect_installed_client to signal INSTALLED_CLIENT_UNKNOWN=1 on a failed `helm list` and wired it into the Helm-step one-client guard — but provision_client's #303 pre-flight still only checked INSTALLED_CLIENT_NS. A failed enumeration leaves both globals empty, so provisioning continued to `client create` and could register a dashboard client the later Helm guard then refuses to install — the exact orphan the pre-flight exists to prevent. Fail closed on INSTALLED_CLIENT_UNKNOWN right after detect_installed_client, before any mint — same signal the Helm-step guard keys on. +bats coverage (unknown state refuses before `client create`, no orphan). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Merge pull request #353 from tracebloc/fix/bugbot-345-detect-values-failopen fix(installer): bash guard fails closed on unreadable client values (PS parity) --------- Co-authored-by: lukasWuttke <54042461+LukasWodka@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Arturo Peroni <arturo@tracebloc.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
scripts/tests/e2e-proxy.sh§4 (the client-runtime#119 ingestion-egress guard) flaked red on develop — run 29255451968. A re-run of the same commit (ca9a3352f) passed, confirming a transient, not a regression — and #341 (the commit under test) never touched this file.Cause
Section A's app-pod curl reaches the in-cluster squid via its Service, but
kubectl rollout statusgates only on the squid pod being Ready — not on its Service endpoints being programmed in kube-proxy. In that window a brand-new pod's first CONNECT to the ClusterIP is refused; curl gave up before attempting the tunnel (noEstablish HTTP proxy tunnelline), so the assertion false-failed. The failure was opaque because the diagnostics only print filtered lines — a novel failure showed nothing.Fix
--retry 5 --retry-connrefused --retry-delay 2(-m 30) on section A's curl — rides out the transient connection-refused. A genuine Add request proxy url to jobs manager yaml file #119 regression still fails all retries, so the guard keeps its teeth.Test plan
bash -n+shellcheckclean. The k3d e2e can't run locally; CI (this workflow) is the authority.🤖 Generated with Claude Code
Note
Low Risk
Test-only changes to an e2e shell script; no production runtime, auth, or data paths are modified.
Overview
Deflakes the §4 ingestion-egress check in
scripts/tests/e2e-proxy.shwhen section A’s curl hits the in-cluster squid before its Service endpoints are ready in kube-proxy (pod Ready ≠ ClusterIP routable).Section A’s proxied
curlnow uses--retry 5 --retry-connrefused --retry-delay 2and a longer-m 30so a transient connection refused does not abort before the CONNECT tunnel is attempted; a real #119-style regression should still fail after all retries.If the tunnel assertion still fails, the script now prints the raw
egress-applog plus squid pod and Service endpoint state so CI failures are actionable instead of only showing filtered diagnostic lines.Reviewed by Cursor Bugbot for commit 06801e1. Bugbot is set up for automated code reviews on this repo. Configure here.