Skip to content

fix(server): preserve in-memory SQLite across reconnects - #3174

Closed
EmilienM wants to merge 134 commits into
NVIDIA:mainfrom
EmilienM:fix/3173-preserve-in-memory-sqlite/EmilienM
Closed

fix(server): preserve in-memory SQLite across reconnects#3174
EmilienM wants to merge 134 commits into
NVIDIA:mainfrom
EmilienM:fix/3173-preserve-in-memory-sqlite/EmilienM

Conversation

@EmilienM

@EmilienM EmilienM commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

🏗️ build-from-issue-agent

Summary

Preserve in-memory SQLite state when SQLx replaces the gateway's operational pooled connection. The SQLite adapter now retains a dedicated keepalive connection to the same named shared-cache database, preventing schema and object loss during sandbox cleanup.

Related Issue

Closes #3173

Changes

  • Normalize anonymous in-memory SQLite URLs to a process-unique named database and enable shared cache.
  • Retain a dedicated keepalive connection while leaving the operational pool serialized to one connection.
  • Add deterministic regression coverage proving schema, existing objects, and subsequent writes survive operational connection replacement for both accepted anonymous URL forms.
  • Document the in-memory SQLite lifetime invariant in architecture/gateway.md.

Deviations from Plan

  • Expanded the regression and normalization to cover sqlite://?mode=memory after adversarial review identified that SQLx does not name that anonymous form automatically.

Testing

  • mise run pre-commit passes — blocked by 17 pre-existing missing SPDX headers in unrelated .tekton/ and deploy/konflux/ files.
  • cargo fmt --all -- --check
  • Matched Rust 1.95 cargo clippy -p openshell-server --all-targets --features test-support -- -D warnings
  • cargo test -p openshell-server --features test-support persistence::tests -- --nocapture (63 passed, 3 pre-existing ignored)
  • Targeted explicit-delete and watcher-reconciliation compute tests
  • The unrelated OIDC test that failed during the full concurrent suite passes in isolation
  • E2E tests — not run because no e2e/ files changed and the existing Podman E2E uses file-backed SQLite, so it does not exercise this regression.

Tests added:

  • Unit: persistence::tests::sqlite_in_memory_store_survives_pool_connection_replacement
  • Integration: Existing compute deletion/reconciliation tests verified
  • E2E: N/A; deterministic persistence regression covers the failure boundary directly

Adversarial review:

  • Independent correctness/test review: clean after one confirmed alternate-URL finding was fixed and re-reviewed
  • Independent architecture/maintainability review: clean on both passes

Checklist

  • Follows Conventional Commits
  • Commit is signed off (DCO)

Documentation updated:

  • architecture/gateway.md: documented the in-memory SQLite keepalive invariant

andre-motta and others added 30 commits July 28, 2026 17:04
Multi-stage UBI9 Dockerfiles that compile gateway and supervisor from
source for Konflux hermetic builds. Gateway uses RPM rust + gcc-toolset-14
(C++20 for bundled Z3). Supervisor builds musl from source for static
linking and installs Rust toolchain from prefetched tarballs. Both images
are multi-arch (x86_64 + aarch64) with SHA-pinned base images, RPM
lockfiles, and generic fetcher configs for all external artifacts.
Includes build-local.sh for local hermetic testing via Hermeto.

Co-authored-by: Martin Prpič <mprpic@redhat.com>
Signed-off-by: Andre Lustosa <alustosa@redhat.com>
Regenerate both rpms.lock.yaml files using an unregistered UBI9
container so all URLs use cdn-ubi.redhat.com (public, no auth)
instead of cdn.redhat.com (requires Red Hat CDN CA certificate).
Both x86_64 and aarch64 arches are included, generated via QEMU
emulation in a podman --arch arm64 container.

Remove createrepo_c and Red Hat CDN CA prerequisites from the
build-local.sh script since they are no longer needed.

Signed-off-by: Martin Prpič <mprpic@redhat.com>
…lockfiles

CARRY: regenerate RPM lockfiles from public UBI repos
Konflux mounts prefetched content at /cachi2/output/deps/ not
/cachi2/deps/. Update all generic artifact paths in both Dockerfiles.
Fix build-local.sh to mount hermeto output at /cachi2/output to match
Konflux layout, and disable subscription-manager DNF plugin under
--network=none to prevent RHEL repo injection.

Signed-off-by: Andre Lustosa <alustosa@redhat.com>
CARRY: Fix cachi2 output path for prefetched artifacts
- Pull pipelines use odh-pr tag
- Push pipelines use odh-stable tag
- Target branch: main
Regenerate supervisor rpms.lock.yaml using --bare mode so all
packages and transitive deps are resolved regardless of what the
base image has pre-installed. Fixes missing libatomic on aarch64
(transitive dep of gcc, not needed on x86_64) and ensures runtime
deps like iproute and util-linux-core are always included even
though they're pre-installed in ubi:9.6 but not in ubi-minimal.

Fix build-local.sh to derive the RPM repo arch from the PLATFORM
variable instead of uname -m, so cross-arch builds mount the
correct arch-specific repo file.

Signed-off-by: Martin Prpič <mprpic@redhat.com>
Daily workflow that creates +rhaiv.N tags based on the latest upstream
version tag. When upstream syncs a new vX.Y.Z, the next run creates
+rhaiv.0. Subsequent runs increment N while no new upstream tag appears.
Skips if no new commits since the last rhaiv tag.

Signed-off-by: Andre Lustosa <alustosa@redhat.com>
…-lockfile

CARRY: fix supervisor arm64 build and cross-arch local builds
- Pull pipelines use odh-pr tag
- Push pipelines use odh-stable tag
- Target branch: main
The workflow failed because persist-credentials: false removed the
GITHUB_TOKEN from the git config, blocking the tag push.

Signed-off-by: Andre Lustosa <alustosa@redhat.com>
…reds

CARRY: fix rhaiv-auto-tag to push by keeping checkout credentials
Switch the RHAIV auto-tag workflow to produce dash-separated midstream
tags (vX.Y.Z-rhaiv.N) instead of the plus form (vX.Y.Z+rhaiv.N).

The '+' character is not permitted in container image tags, so dash
tags let tag-triggered Konflux image builds use the git tag verbatim as
the image tag. The '-' to '+' conversion for PEP 440 Python package
versions is handled in the Python build infrastructure.

The upstream-tag filter now excludes both the new -rhaiv form and the
legacy +rhaiv form so any existing +rhaiv tags don't interfere with
latest-upstream detection.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Martin Prpič <mprpic@redhat.com>
CARRY: create -rhaiv.N tags instead of +rhaiv.N
The Konflux supervisor and gateway builds previously produced binaries
reporting version 0.0.0: the Dockerfiles copy Cargo.toml (workspace
version 0.0.0) without patching it, and .git is absent from the build
context so build.rs cannot derive a version via git describe.

Accept an OPENSHELL_VERSION build-arg, strip the leading "v", and export
it as OPENSHELL_GIT_VERSION for the cargo build. openshell-core::VERSION
prefers OPENSHELL_GIT_VERSION over CARGO_PKG_VERSION, and both the
supervisor (openshell-sandbox) and gateway (openshell-server) binaries
report that constant, so a tag build now stamps e.g. 0.0.93-rhaiv.0.

The build-arg is empty by default, so push and pull-request builds are
unaffected and continue to report the workspace Cargo version.

The odh-konflux-central tag PipelineRuns pass OPENSHELL_VERSION={{git_tag}}.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Martin Prpič <mprpic@redhat.com>
CARRY: stamp rhaiv version into konflux binaries
red-hat-konflux Bot and others added 21 commits August 28, 2026 02:07
Signed-off-by: red-hat-konflux <126015336+red-hat-konflux[bot]@users.noreply.github.com>
…in/lock-file-maintenance

chore(deps): refresh rpm lockfiles
Update all Konflux gateway, supervisor, and CLI builder and runtime
base images to pinned UBI 9.8 manifests.

Signed-off-by: Emilien Macchi <emacchi@redhat.com>
Build the openshell-gateway package after gateway binary composition moved
out of openshell-server. This ensures the runtime image contains the binary.

Signed-off-by: Emilien Macchi <emacchi@redhat.com>
CARRY: migrate base images to RHEL 9.8
Build both with `--no-default-features --features defaults-without-telemetry`.
The alias matters: a bare `--no-default-features` would also drop
`in-tree-compute-drivers` from the gateway and `bundled-ca-roots` from the
supervisor.

Only the gateway emits telemetry, and its binary no longer contains the
telemetry endpoint. The supervisor flags are a no-op today and are carried for
lockstep.

Resolves AIPCC-30286

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Martin Prpič <mprpic@redhat.com>
CARRY: compile telemetry out of gateway/supervisor
Signed-off-by: red-hat-konflux <126015336+red-hat-konflux[bot]@users.noreply.github.com>
Signed-off-by: red-hat-konflux <126015336+red-hat-konflux[bot]@users.noreply.github.com>
…in/lock-file-maintenance-vulnerability

chore(deps): refresh rpm lockfiles [SECURITY]
…in/lock-file-maintenance

chore(deps): refresh rpm lockfiles
Keep a dedicated SQLite connection alive so pool connection replacement
retains the shared in-memory schema and objects.

Closes NVIDIA#3173

Signed-off-by: Emilien Macchi <emacchi@redhat.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

Thank you for your submission! We ask that you all sign our Developer Certificate of Origin before we can accept your contribution. You can sign the DCO by adding a comment below using this text:


I have read the DCO document and I hereby sign the DCO.


1 out of 6 committers have signed the DCO.
✅ (andre-motta)[https://github.com/andre-motta]
@moulalis
@mprpic
@mchamsed-rh
@alexxfan
@EmilienM
You can retrigger this bot by commenting recheck in this Pull Request. Posted by the DCO Assistant Lite bot.

@EmilienM

EmilienM commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #3175, which is based directly on NVIDIA/OpenShell main and contains only the issue #3173 fix. This PR accidentally inherited downstream carry commits from the local base.

@EmilienM EmilienM closed this Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(server): in-memory SQLite loses objects schema during Podman sandbox cleanup

7 participants