Skip to content

fix(runtime): consolidate patch correctness and retire unsafe shortcuts - #202

Closed
f0rr0 wants to merge 23 commits into
mainfrom
f0rr0/consolidate-patch-correctness
Closed

f0rr0 wants to merge 23 commits into
mainfrom
f0rr0/consolidate-patch-correctness

Conversation

@f0rr0

@f0rr0 f0rr0 commented Sep 7, 2026 •

Copy link
Copy Markdown
Owner

September 27 refresh onto current main

The earlier correctness work has been selectively ported onto main df2b112e866f8ea76d320d7b3e172cf7be764177, preserving its current source/build graph, resources/extensions, seedless browser startup, and extracted Postmaster executor. Existing PR history is preserved; no force push.

Not yet merge/release-ready: Windows/MSVC Sys exception recovery remains a blocker for the complete supported matrix. A production early stack guard is not enabled. Fresh Postmaster concurrent runtime and full supported-platform/extension qualification remain outstanding.

Implemented

  • Retire the 13 rejected embedded WASIX patches, with reasons retained in the patch ledger; port guest recovery and Rust/browser host ABI changes together.
  • Port native startup cleanup, process/session boundaries, trusted identity, checked entropy and bounded output. Stop disabling native fsync by default.
  • Use strict-memory AOT and reject stale artifact profiles instead of relabelling binaries.
  • Split Postmaster Wasmer/libc patches into ordered series while preserving main's first-party executor; strict patch replay and preparation receipts replace unsafe application assumptions.
  • Reuse existing owner tests/contracts instead of restoring obsolete policy machinery. No new public environment flags.
  • Fix two newly exposed integration gaps: omitted broker application_name overriding role defaults, and a file-wide feature gate silently skipping core Rust runtime tests.

Fresh local evidence

  • Native Linux core build, C ABI/process-boundary smoke, broker identity/wire integrations, 108 SDK and 3 broker unit tests.
  • Fresh WASIX core/initdb and production browser host; Chromium memory, IndexedDB and OPFS startup, error recovery, COPY callback failure/reuse, identity and persistent close/reopen.
  • Fresh strict AOT; seven PostgreSQL regressions, 17 existing runtime cases, plus actual 10,000-row COPY OUT/recovery in both memory and directory modes.
  • Strict source replay: 22 native PG, 31 embedded WASIX PG, 27 browser-host, nine Postmaster Wasmer and eight libc patches. Postmaster owner tests/lint pass; full new Postmaster producer/runtime not yet qualified.
  • First hosted run 36310352902 exposed the release metadata requirement for a single-parent tip; documentation follow-up ef4821f now follows the history-preserving merge. The full local release-tools:metadata check passes on this tip; policy was not weakened. Hosted CI must still qualify the new SHA.

These are correctness checks, not a comparative performance or release qualification claim. Native fsync-on can cost write latency intentionally.

New dependencies: experiments, not production upgrades

  • Wasmer 7.4.2 / WASIX 0.704.2 passed isolated Linux LLVM/AOT, exception recovery, memory-growth and real host-module probes. Worth a separate full SDK compatibility and matched-workload evaluation; no Windows or stack-exhaustion fix established.
  • @wasmer/sdk 0.18 ran SQL and large local browser transfers. It exposes a command-oriented API, not our embedded exports; the tested PostgreSQL data did not survive command restart in the same Sandbox. A supported JavaScript PGDATA mount/persistence contract is still needed. Its tested PostgreSQL package is Oliphaunt-derived, not ElectricSQL PGlite.
  • Production Wasmer/browser-host pins remain unchanged.

Current implementation, evidence, retained assets and next steps.
Follow-up tracking: #201.

Historical September checkpoint (not current qualification)

Latest update — 41c74d8 (2026-09-08)

Not merge-ready. The Windows/MSVC exception-recovery blocker remains. The new execution-stack guard is a tested review candidate, not enabled in consumer packages.

This update pushes the completed resource-budget cleanup and the actual stack-safety candidate code:

  • Centralize existing transport/link-time budgets, preserve their values, and document ownership, tradeoffs, PostgreSQL/PGlite comparisons and the distinction between linear and execution stacks. No new environment flag or larger production stack.
  • Add four candidate patches and focused regression probes: Wasmer VM accounting/EH guard, fixed Wasmer API query, PostgreSQL early check, and Rust import registration. Each passes clean application against its documented base.
  • Keep those patches outside production series and dependency selection. Published Wasmer 7.2.1 lacks the new API; Windows/browser contracts and recovery margins are not qualified. This PR does not pretend to deliver the new guard to downstream consumers yet.
  • Remove ordinary-host-call stack snapshots from the candidate. Its successful fixed query has no allocation, lock, arbitrary callback or host-stack switch. Missing accounting and the 64 KiB engine floor fail terminally; PostgreSQL retains a separate 512 KiB early-recovery reserve and the engine retains its 1 MiB default.
  • Preserve code and evidence without importing binaries, producer trees, the benchmark framework or historical result archives.

Candidate evidence, not exact-head release qualification

  • Wasmer VM: 62/62 debug and 62/62 release tests; baremetal compile check.
  • 1,000-cycle scalar ABI/nested-stack probe; 100 serialized-AOT deep throw/catch and rethrow cycles, followed by expected terminal stack failure.
  • Real PostgreSQL memory/directory recovery: repeated SQLSTATE 54001, subsequent queries, PL exception catches, savepoint rollback and directory reopen.
  • 144 successful benchmark children against the retained previous guarded direct-binding candidate, with identical guest/AOT and SDK dependency versions. RTT improved 2.6% memory and 3.6% in the directory follow-up. The initial +8.2% directory temporary-INSERT warning did not reproduce at that magnitude: the eight-pair follow-up measured +1.7%; COPY measured −2.0%. Small overheads remain uncertain; no blanket non-regression claim.
  • PR preparation: pinned Moon repo:docs-policy sdk-contracts:all passed (9 tasks, 5 cached), profile-budget test passed, both pq_flush ABI tests passed, regression-probe Rust formatting passed, all four candidate patches passed git apply --check, and staged whitespace checks passed.

Before activation: settle/consume the upstream VM/API contract (coordinate Wasmer #6913), qualify recovery margins and error cleanup across supported targets, resolve the Windows/browser contract, then integrate pins/imports/artifact invalidation together. No release build or exhaustive qualification was rerun merely to publish this review update.


Summary

Previously published checkpoint: 0f4a996. Not merge-ready: Windows/MSVC Wasm exception recovery is blocked in the consumed Wasmer runtime. The known formatting and Postmaster source-root fixes are pushed. No Windows target/test is removed or waived, and the repeat optimization remains held outside this PR.

Consolidate the completed patch-correctness work on current main, without importing the research harness, historical result archives, old binaries or producer trees.

  • Recover Native trusted-session, startup/cwd cleanup, host process-boundary and COPY/cancellation fixes while preserving main's ABI 10 and public SDK APIs.
  • Replace WASIX host-longjmp symptom handling with live guest-local recovery and typed outcomes. Integrate the matching Rust/browser host ABI, bounded protocol/tool output, entropy and fail-closed lifetime/transport changes.
  • Remove 13 rejected embedded WASIX patches, including LIKE, int4 comparator shortcuts, semaphore reset and scalar atomics. Preserve the seek, parser, WAL-sync and JSONB optimizations already merged on main.
  • Use the fixed strict-memory AOT profile llvm-opta-ro_ftable throughout production, caches, manifests and all eight carriers; reject incompatible legacy overrides and artifacts.
  • Clean the ordered Native (22), embedded WASIX (31), Postmaster PostgreSQL (7), and browser-host (27) patch series, with patch-local motivation and focused checks. The Postmaster runtime has separate ordered Wasmer (9) and libc (8) series. Do not claim the inherited runtime bundle is fully qualified or upstream-ready.
  • Align generated-header installation/fingerprints, ABI export policies, Moon cache inputs and the existing core-only fast-path packaging check.

This is intentionally a substantial correctness integration, not a small isolated performance patch: guest/host ABI and Native process ownership must land together. It excludes optional compiler/runtime experiments and the obsolete pre-N-API Node filesystem path.

Follow-up implemented at 51789da

  • Preserve PostgreSQL's Native fsync=on default instead of silently adding -F; explicit fsync=off remains available. Fresh Linux ICU direct and installed-broker checks verify both settings, identity/login policy, session reset and nondefault-database settings.
  • Fix the ICU collation import failure without permitting subprocess execution, and move the signal-mask declaration to its type-owning header to fix Windows runtime/extension compilation.
  • Replace the Postmaster monoliths with 8 Wasmer + 7 libc logical patches. Strict application produces exactly the previous source trees. Separate Wasmer0009 disables unsafe nonvolatile-memory assumptions in all four compiler routes and invalidates old artifact identities.
  • Postmaster source/shell units and both freshly compiled product/CLI strict-memory identity tests pass. The corrected-profile hosted Postmaster producer/qualification at 51789da also passed (job); the subsequent libc0008 full carrier remains subject to the new head's CI.
  • SDK header mirrors, Clippy and the build graph failures are corrected. The repository-pinned Moon toolchain runs locally; SDK contracts and graph tests pass.

Additional follow-up at 8e45be7

  • Initialize the WASIX host-selected role as the real PostgreSQL session principal during startup. Enforce catalog login/admission, role/database defaults and login triggers; prevent RESET ROLE/DISCARD ALL from restoring bootstrap-superuser rights. Delete the late SET ROLE workarounds from Rust direct/proxy and browser direct/worker paths.
  • Preserve typed startup SQLSTATE/details for Rust SDK callers and the exact original error bytes for proxy clients, using the existing decoder.
  • Make libc sigsetjmp evaluate both arguments once while keeping setjmp in the live caller. C/C++ O0/O2 host checks and actual C/C++ Wasm O2 execution pass; the old macro fails the negative control. The new helper/header was tested over the retained EH sysroot on a freshly built strict Wasmer runner, not mislabelled a full sealed-carrier rebuild.
  • Add inherited memcmp alignment, byte-order and protected-page boundary checks; no new speedup is claimed. Update ordered-series source locks and expose the newly established POSIX signal-mask limitation truthfully.
  • Reuse the active AOT-enabled serializer executable instead of rebuilding a shared bare xtask path which another worktree can replace. No new public environment flag or research framework.

Hosted failure fixes prepared at 94d674d

  • Initialize and consistently propagate the Postmaster libc source root before capability validation. A no-build behavioral test executes the real script prefix/inventory with default/custom roots and verifies missing-libc rejection.
  • Correct the single Biome formatting failure. The full 263-file JS formatting path list now passes locally using the exact pinned Biome version.
  • These fixes plus profile/experiment, timeout and Windows-blocker documentation are pushed at 0f4a996. All material producers finished before the push: iOS extensions and Linux/macOS AOT passed on 8e45be7; Windows serialized/validated/installed AOT successfully but crashed during SQL-error recovery in its Rust regression suite. Neither the failed parent run nor its successful individual jobs are represented as exact-head green qualification.

Final focused validation

  • Fresh portable guest/support/initdb, standard and ICU seeds, strict AOT packaging/artifact checks pass. Strict PostgreSQL replay: 22 Native / 31 WASIX / 7 Postmaster, no fuzz/rejects.
  • Rust on those fresh assets: 211 library tests, 19 runtime tests, 7 proxy tests, 7 PostgreSQL regression tests pass with extensions enabled. This is a different feature scope from the initial tools-enabled test count below.
  • The final tools-enabled configuration additionally passes 223 library tests, 7 public-API tests and 5 doctests.
  • Real Chrome memory/IndexedDB recovery and COPY floor plus configured-role direct→worker→direct reopen, defaults, login triggers, reset and privilege/NOLOGIN rejection pass. TypeScript 343 tests / 36 files, binding and example typechecks pass.
  • SDK contracts, graph unit tests, WASIX source/artifact guards, Postmaster canonical source/shell units and source lock verification pass. Rust formatting and Clippy pass for WASIX with/without extensions and xtask with/without the serializer feature.
  • Final benchmark: 220 independent children, all correctness and actual durability/version oracles pass. Prior corrected candidate and final identity candidate retain matching RTT; final/strict-main RTT is +3.5% memory / +2.3% directory. Historical narrow-directory ~10% warning did not repeat, but final/prior wide temporary INSERT warns +5.1% memory / +4.2% directory. No blanket non-regression claim.
  • Stock PostgreSQL 18.4 context still shows final narrow INSERT 1.41x memory / 1.64x directory, wide INSERT about 1.60x, and mixed OLTP 1.35x / 1.51x. Two stock-server process replicates and differing transports make this descriptive context, not paired cross-engine proof. The checked-in report contains the full nine-case table, scale, settings and provenance. Equal affinity was not exclusive CPU isolation; unrelated shared-host activity remained.

Initial 34f173f validation

  • Strict PostgreSQL 18.4 patch replay: 22 Native / 30 WASIX / 7 Postmaster, no fuzz or rejected hunks.

  • Fresh Linux standard Native library: ABI conformance, configured identity/login policy, six startup rejection cases, signal/mask/timer ownership, cancellation and COPY timeout/backpressure, five cwd cases and broad C smoke passed.

  • Rust WASIX: 215 unit tests, 5 doctests, 7 public API tests passed.

  • Browser TypeScript: 343 tests / 36 files and typecheck passed; six pinned-host source-contract tests passed.

  • Real isolated Chrome: memory and IndexedDB both passed repeated syntax/division error recovery, PL/pgSQL exception catch, set-based INSERT, 1 MB COPY callback abort/drain/reuse; IndexedDB close/reopen preserved all 1,000 rows.

  • Fresh portable guest/support/initdb built; standard and ICU seed initialization/profile probes passed. Strict AOT compiled, packaged and passed artifact validation; all eight actual generated carrier identities were checked.

  • Shared protocol schema/generated views, compiled C bridge ABI, 64 MiB aggregate tool-output policy, SJLJ artifact guard, patch-review freshness, Postmaster source lock, focused libc/runtime checks, Rust formatting and documentation policy passed.

  • assets verify-committed passed using the existing core-only build flag; full extension catalog validation remains the default outside that explicitly scoped path.

  • Fresh Rust WASIX on the new portable guest and strict AOT: 18/18 runtime-smoke and 7/7 PostgreSQL regression tests passed. These exercise memory/directory storage, sync/async callback error and panic recovery/reuse, prepared parameters, INSERT/savepoints, rollback/cancellation, directory ownership/reopen, backup/restore and recovery after an abruptly killed child. These are focused cases, not exhaustive crash qualification. Rust COPY-specific coverage was not available in these suites; actual browser COPY coverage is listed above.

Boundaries and follow-up

  • Windows/MSVC is a merge blocker. Job 101905963703 reaches an explicit panic in wasmer-vm 7.2.1's MSVC throw() implementation during savepoints_error_recovery_and_indexed_updates, exit 0xe06d7363. Official v7.4.0 and current upstream main still contain the unsupported MSVC exception stubs; an ordinary dependency upgrade is not a fix. Genuine engine exception support or a separately qualified compatible guest is required. Skipping tests, catching the panic or silently falling back to Native would not preserve the guest recovery/isolation contract.

  • The retained repeat experiment is held, not promoted: 184 correctness/settings-checked children show wide temporary INSERT 41.5% / 33.5% faster and logged wide 29.0% / 20.7% faster (memory/directory). A separate 72-child targeted+A/A recheck resolved the preconstructed-directory warning but retained a prepared-directory warning of approximately 6–7% against both baseline controls, predominantly in transaction body. Shared-host interference remains a limitation. Exact candidate artifacts and original baselines remain retained; no experimental framework is staged, and neither durability nor fill-chunk tuning is a demonstrated remedy for the warning.

  • SQL statement_timeout is not a reliable CPU-bound deadline in current Rust WASIX. Actual baseline/candidate probes expose inherited libc timer-field and Rust unit/one-shot/delivery gaps. Native timeout/reuse and injected interrupt tests do not prove Wasm delivery. Public Rust/usage docs now explicitly distinguish accepting the GUC or timing out a caller future from stopping guest execution; issue Follow up patch consolidation: signal semantics, performance profiling, and qualification #201 records the coordinated remedy.

  • No release-ready or blanket non-regression claim. The initial 324-child strict-profile comparison and final 220-child identity screen are separate evidence sets; the later one does not erase the earlier warnings. All accepted directory reports actually verify fsync/synchronous_commit/full_page_writes=on. No Native durability-change timing is inferred from the WASIX screen.

  • Inherited Postmaster POSIX signal masks are not implemented. Pinned pthread_sigmask returns success/no-op, sigpending returns EINVAL, and handler sa_mask is ignored. The capability inventory now says unsupported. The macro fix does not repair delivery masking; correct remediation needs coordinated Wasmer/libc mask/pending delivery semantics and PostgreSQL stress tests. This is a high-priority known limitation, not an observed corruption or new-regression claim.

  • Four completed prior/final ABBA profile captures collected 45,283 samples on unchanged strict guest/AOT payloads, using existing public Wasmer function extents for diagnostic address maps. The wide workload's repeat(chr(...),96) makes 96 one-byte copies per row; imported-copy self samples account for 24–27%, while direct statistics-counter samples are about 0.45%. This supports a bounded general repeat experiment, not causal attribution of the +5.1% warning. No new optimization is promoted. The user-authorized temporary profiling setting was restored to its original perf_event_paranoid=4 immediately after capture; no persistent setting changed.

  • Hosted Native builds and the portable WASIX, Postmaster and iOS-extension producers passed at parent 51789da. The later 8e45be7 head also passed Native/extension and portable WASIX producers plus Linux/macOS AOT, but exposed the known formatting/root failures and Windows exception blocker. That parent run was cancelled by the push only after material producers finished, with app/E2E work remaining. 0f4a996 must qualify in its own run 34177975360; parent results are not relabelled as final-head results.

  • Native now preserves fsync=on by default; durable-write latency can increase. Functional reopen tests still do not establish crash durability. No WASIX durability relaxation is introduced.

  • Guest and hosts/AOT must be rebuilt together. Old profile manifests cannot be relabelled; terminal guest failures cannot be treated as recoverable SQL errors.

  • Browser host runtime code was built before mail-header normalization. Every patch diff is byte-identical before/after that metadata-only step; the original build marker is retained and a local before/after equivalence receipt records both identities. This is not labelled an exact-release artifact.

  • Local Moon affected-project resolution, SDK contracts and graph unit tests pass with the repository-pinned toolchain.

  • Broad platform/ICU, extension, hostile-extension/resource, crash and release qualification remain separate work.

The complete prioritized remainder is issue #201: inherited Postmaster signal-mask semantics, a profile-supported repeat experiment and unresolved wide-INSERT warning, remaining platform/runtime qualification and upstream review. Optional compiler/ICU/ownership ideas are conditional research, not mandatory features. Rejected CRC/lazy-globals/unsafe shortcuts are not automatically revived.

See docs/internal/PATCH_CONSOLIDATION_20260907.md for the consumer implications and explicit retirement table. The original research state is preserved in a separate local recovery commit and is not staged here.

@vercel

vercel Bot commented Sep 7, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
oliphaunt-docs Ready Ready Preview Sep 27, 2026 9:50am UTC

* refactor: localize product tooling and simplify release qualification

* fix: qualify rendered CI plans through owned tooling checks

* fix: propagate attestation verifier failures on Bash 3.2

* fix: isolate owner tests and repair native build setup

* fix: let cancelled CI aggregates release the run queue

* refactor: trim maintainer bootstrap and own installer tests

* fix: avoid duplicate docs checks and type native SDK smoke helpers

* refactor: remove oversized fixtures and keep tests with their owners

* fix: exercise LLVM installation and share lifecycle test support

* test: qualify the standalone native lifecycle runner locally

* fix: isolate commit-range planning from caller input

* fix: cache Moon content-addressed outputs across project edits

* fix: reject malformed Moon query envelopes

* fix(ci): repair cold client builds and Windows backup paths

* fix(ci): repair cache restores and carrier qualification

* fix(ci): qualify unprivileged carriers and provision evidence dependencies

* fix(postmaster): include release asset finalizer in source checkout

* refactor: model product dependencies and simplify qualification

* fix: make qualification work on clean platform runners

* fix: gate CI consumers and preserve validated build outputs

* refactor: isolate packaged dependencies and centralize tool pins

* fix: qualify packaged consumers without hidden checkout state

* fix: include runtime preflight in build dispatch fixture

* fix: resolve Git Bash and validate portable Windows packages

* fix: allow shell setup verification to run in CI

* fix: qualify native packaging and WASIX consumers at their owning stages

* fix: restore ICU build inputs and bound pgwire shutdown reads

* fix: make CI consumers use complete isolated product inputs

* fix: repair artifact consumers and restore full extension qualification

* fix: provision Bun and consume declared Swift source archive

* fix: complete mobile SDK runtime dependency packaging

* fix: complete independent iOS resource packaging and qualification

* fix: let the React Native plugin own seed pod registration

* fix: isolate iOS implementation headers and bound browser readiness

* fix: include mobile ABI jobs in the build gate

* refactor: consolidate projects under src and reduce CI duplication

* fix: resolve remaining source migration paths

* fix(ci): cache verified tools on PRs and back off downloads

* fix: resolve ICU inputs and canonical package notices

* fix: align relocated extension and consumer inputs

* fix: package WASIX generated metadata from src

* fix: serialize overlapping source fetches

* fix: preserve installed SDK resource discovery and seed integrity

* fix: integrate SDK resource and host boundary fixes from PR 211

* test: verify integrated SDK APIs through installed consumers
* fix(ci): import Swift signing keys and preserve release test timeouts

* fix(test): use the pinned Bun launcher for every test entrypoint
* refactor: organize sources by native and WASIX runtime families

* fix(ci): pin the upstream Swift signing key bundle

* fix(ci): preserve Swift source paths in iOS artifact staging
* fix(sdk)!: align resource loading and extension selection

Load independently versioned WASIX extension packages, materialize native
Rust resources automatically, and initialize browser storage without a seed.
Align typed extension selections, native TypeScript restore, version-based
WASIX compatibility, and Swift documentation with the resource contracts.

BREAKING CHANGE: Native extension lists use typed selections, and native
TypeScript restore accepts a directory storage descriptor instead of a path.

* fix(test): make native consumer restore checks portable

* fix(wasix): preserve worker clocks and extension resources
@f0rr0

f0rr0 commented Sep 27, 2026

Copy link
Copy Markdown
Owner Author

Superseded by #218, created against current main to correct the stale GitHub comparison. Exact same head commit ef4821f; no source changes or lost work. Verified #218 reports 248 files, +59,360 / -43,720 against df2b112. This PR and branch remain available for historical discussion/evidence. Replacement is draft; Windows recovery and outstanding qualification blockers remain.

@f0rr0 f0rr0 closed this Sep 27, 2026

This branch was successfully deployed

1 active deployment
Preview — ef4821f5 Deployed Sep 27, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant