Rebuild Kuber.jl on OpenAPI.jl 1.0 - #69
Merged
Merged
Conversation
tanmaykm
changed the title
tanmaykm
marked this pull request as ready for review
September 2, 2026 03:31
…l 1.0 The trial's four phases, squashed. `src/ApiImpl/api/` is gone; in its place one generated module per Kubernetes group version, each carrying its own models, operations and embedded JSON Schemas, plus a generated `registry.jl` holding the six lookup tables the verb layer resolves through — GROUP_MODULES, MODULE_GVS, KIND_TYPES, OPS, OP_PARAMS, OP_BODIES. These replace api_typemap.jl/api_versions.jl and every string-munging `eval` lookup. `helpers.jl` and `simpleapi.jl` are rewritten against the 1.0 runtime. KuberContext now holds one Runtime.Client per group module — a client is bound to its module's compiled _SPEC and cannot be shared — plus HTTP.jl 2.x request options and a retry condition built from an actual characterization of the runtime's exception types (test/characterize_retries.jl records what it found, rather than guessing). Strict generation and strict response validation are on and stay on: a SchemaValidationError against a real cluster means the document lies, and the fix is a patch rule in patch_k8s_spec.jq, never validate_responses=false.
Four lifecycle bugs, all found by writing the acceptance criteria down and then failing them: the consumer closing the public stream is the only stop signal; a clean close must re-watch from the last resourceVersion; an in-stream 410 must not restart without one; and a 200-with-no-events must not spin the re-watch loop. Mid-chunk aborts recover rather than propagating. test/watch_recovery.jl covers all of it against a fake apiserver. CI's cluster version is load-bearing now, which it was not on 0.2.x: responses are validated against the schemas the client was generated from, so the workflow pins kind and its node image to the same k8s minor the specs came from. A spec bump has to move that pin with it.
OpenAPIv1ConsumerGaps.md starts here — a survey of what JuliaRun and the JuliaHub monorepo need that this branch did not yet provide, written as stable C…/G… identifiers so the items can be cited and ticked off rather than renumbered. C1 was the blocker: consumers plugged their own generated layer in through KuberContext(apimodule), which this branch removed. src/register.jl answers it — Kuber.register!/unregister! merge an out-of-tree layer's six registry tables into the shipped ones from the registering package's __init__, validating the whole registration before merging any of it. That splits C1 into a mechanism (closed) and a content question, and rescopes it by what consumers actually reach. Also here: `Kuber.is_retryable`, the exception classification consumers lost when OpenAPI.Clients.is_request_interrupted went away; re-listing instead of replaying when a resourceVersion expires (G1), because watching from no resourceVersion replays current state as ADDED and never mentions what was deleted in the gap; and capturing group documents from a live cluster, which is how metrics.k8s.io/v1beta1 ships again — aggregated APIs are not in release-tag specs.
The G-series, squashed. What consumers do that the suite never touched: caller-driven watch re-establishment and event continuity across the re-watch seam; selector-scoped watches across all namespaces; the kinds consumers actually write; put!(ctx, O, dict), the form production writes go through; Secret data/stringData round-tripping; a cluster-scoped Node patch; the shapes consumers read off a live result; and the retry loop driven against injected failures. Two of these were not test-only. JSON patches did not encode at all (G16/C8) — k8s declares one object schema for all five patch media types, but a JSON Patch is an array of RFC 6902 operations, so OP_BODIES maps media type to body type. And the live suite was orphaning Job pods and leaving state behind, which is why it now clears leftovers before it runs.
Patch rule §7 collapses the single-element allOf that k8s wraps every property $ref in to hang a description on it. Read literally that mints a type per use site; collapsing it halved the layer, 2252 types -> 1098, and is what makes PodList.items eltype Pod rather than a positional copy. It is scoped to property schemas and array items rather than walked recursively, because apiextensions' JSONSchemaProps has properties *named* allOf/nullable/items and a recursive walk corrupts it. Rule §8 declares resourceVersion on single-object reads, which the apiserver honours and k8s documents only on lists. k8s_retry is an explicit loop rather than Base.retry, so a 429's Retry-After can lengthen the wait. max_tries counts attempts, not retries, and _call_options sets retry=false on every call so HTTP.jl's own retry layer is not multiplying underneath — before that, max_tries=1 meant ten requests and a mutating call was retried once despite all_apis=false. Kuber.getpropertyat/haspropertyat replace the OpenAPI.Clients accessors 1.0 dropped, which JuliaRun uses at 49 sites. Resource limits and requests are open structs, not Dicts (G12) — kuber_props reads them.
custom.metrics.k8s.io was captured from a cluster running prometheus-adapter and then deliberately not shipped. Its operations carry neither x-kubernetes-group-version-kind nor x-kubernetes-action, and they address metrics through a three-variable path, so emit_registry.jl and the verb API cannot carry them: shipping it would mean KIND_TYPES entries with no OPS, which is worse than not shipping. The document is kept as evidence in gen/openapi_v1/reference-captures/, never in specs/, which the chain globs. Zero callers sealed it. Two capture-mode bugs the exercise exposed: SPECS_CAPTURED listed only the current run's files, so capturing a second group silently erased the first group's record, and the merge that fixed it dropped blocks it did not recognize instead of keeping them verbatim. G5 split along what compression can reach. G5a is the CI-sized half — several apiserver-initiated clean closes with timeout_seconds, asserting exactly-once delivery and decoding real BOOKMARK frames, which nothing covered before. Its first CI run failed and the failure was real: the events-only watch form lists internally to learn where to resume and discards that list, so an object created in that window is never announced. Seeding resource_version from an explicit get fixes it, and the README now says so. G5b stays a manual probe — an hours-long watch tracking descriptors and live bytes.
The gaps document grows a checklist arranged by repo rather than by finding, ordered by risk, so each consumer team can read one section. JuliaHubK8sApi.jl is dropped rather than regenerated: Kuber's shipped layer is a superset of what consumers actually reach, so there is no out-of-tree layer left for them to register. The dependency line comes out last, after the references are gone. C10 was found by asking whether JuliaRun could be ported from these documents alone. It could not — the survey had covered OpenAPI.Clients and missed two uses of the OpenAPI module proper. OpenAPI.APIModel has no successor (1.0 models share no supertype), and OpenAPI.to_json is gone with JSON.json a silently wrong replacement: it emits lowercase field names and "ABSENT" strings that a cluster rejects.
PR JuliaComputing/OpenAPI.jl#103 merged as d0b7acb and 1.0.0 was tagged at cdcf203, so Project.toml carries no [sources] entry any more: OpenAPI = "1" resolves from the General registry. The pin move in between is worth recording, because it produced a false green. Pkg resolved the *new* rev name onto the *old* tree — the gitignored Manifest recorded 1ff9ba8's git-tree-sha1 under cdcf203's repo-rev — so a regeneration, the registry gate and the whole integration suite all passed while exercising the code the pin had supposedly moved away from. The break it hid was loud once the tree was right: generated modules emitted SchemaEngine.Dialect positionally and cdcf203 makes that constructor keywords-only. Regenerating against a correctly resolved tree fixed it and moved no consumer-facing name — registry.jl came out byte-identical. C14 records the discipline: after moving a [sources] rev, delete Manifest.toml and check the resolved git-tree-sha1 against `git rev-parse <rev>^{tree}`, because Pkg.status() shows the rev, not the tree. Verified against the released version on that footing: regeneration byte-identical (18 modules, 141 kinds, 468 operations), offline suites 6098 assertions, live suite against k3s v1.35.4 clean, and CI green on Julia 1.11, 1 and nightly. Also lands the JuliaRun port's write-back into the gaps document — JuliaRun is ported and its own integration suite passes end to end, which is the first evidence the shipped layer is enough: nothing needed Kuber.register!.
tanmaykm
force-pushed
the
openapi-v1-trial
branch
from
September 2, 2026 03:40
cab9ce4 to
d2b9262
Compare
krynju
approved these changes
Sep 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A rebuild of Kuber.jl on OpenAPI.jl 1.0, replacing the entire generated API layer and rewriting the verb layer against the new runtime. It follows
OpenAPIv1TrialBranchPlan.md;OpenAPIv1RewriteNotes.mdis the evaluation behind it.Read
OpenAPIv1TrialResults.mdfirst. It is the implementation record: what was built, the five places implementation contradicted the plan and why, the measured numbers, and what is left. Where the plan and the results doc disagree, the results doc is current.The blocker is gone. JuliaComputing/OpenAPI.jl#103 merged on 2026-08-29 and OpenAPI.jl 1.0.0 was released on 2026-09-02.
Project.tomlcarries no[sources]entry any more —OpenAPI = "1"resolves from the General registry. The release is tagged atcdcf203, the commit the generated layer was last regenerated against, so the released tree and the committed generated layer are the same tree: rerunning the whole generation chain against the registry version reproduces it byte for byte.Shape of the change
src/ApiImpl/generated/— one generated module per Kubernetes group version (K8sV1,K8sAppsV1, …), 18 group versions, each carrying its own models, operations and embedded JSON Schemas. Replaces the oldsrc/ApiImpl/api/tree.src/ApiImpl/generated/registry.jl(also generated) — the lookup tables the verb layer resolves through:GROUP_MODULES,MODULE_GVS,KIND_TYPES,OPS,OP_PARAMS,OP_BODIES. 141 kinds, 468 operations. These replaceapi_typemap.jl/api_versions.jland all the string-munging plusevallookups.src/register.jl—Kuber.register!/unregister!merge an out-of-tree generated layer's copy of those tables into the shipped ones. This is the replacement for 0.2.x'sKuberContext(apimodule), and it is how CRD groups and operator-owned APIs plug in. Validates the whole registration before merging any of it.src/helpers.jl,src/simpleapi.jl— rewritten.KuberContextnow holds oneRuntime.Clientper group module (a client is bound to its module's compiled spec and cannot be shared), HTTP.jl 2.x request options, and a retry condition built from an actual characterization of the runtime's exception types rather than guesswork (test/characterize_retries.jlrecords the findings).gen/openapi_v1/— the reproducible generation pipeline: fetch pristine OpenAPI v3 documents from akubernetes/kubernetesrelease tag, patch them withpatch_k8s_spec.jq(eight rules), generate in strict mode, emit the registry. See its README.get,list,put!,update!,delete!,watch,sel) is unchanged in shape.Strict generation and strict response validation are on, throughout. A
SchemaValidationErroragainst a real cluster means the document lies and the fix is a new patch rule — there is novalidate_responses=falseanywhere insrc/. Several of the eight patch rules were found exactly that way. Two are worth knowing about: §7 collapses the single-elementallOfk8s wraps every property$refin, which halved the layer (2252 types → 1098) and is what makesPodList.itemseltypePodrather than a positional copy; §2 makes array properties nullable, because Go nil slices marshal asnull— a watchBOOKMARKonly decodes at all because of it.What changes for callers
ABSENT, notnothing— the one semantic change to watch for.nothingnow means an explicit JSONnull. Code testing "not set" withx.field === nothingmust useKuber._field(x.field) === nothing. In practice the trap ishasproperty(model, :field), which is now always true._suffix on collisions with Julia reserved words:metadata.resourceversion,obj.apiversion,event.type,svc.spec.type_. Type names are unchanged (IoK8sApiCoreV1Pod). Case is deliberately not folded, so a missed camelCase name fails rather than resolving quietly — except throughgetpropertyat, which answersnothing.Dicts.metadata.labels/annotationsentries live inadditional_properties; usekuber_props(pod.metadata.annotations)["key"].KuberEventwith an already-typedevent.object; thekuber_objround-trip is gone.apps/v1'sStatusis not core's. Comparekuber_kind(res) == "Status", not the type — this matters fordelete!.set_timeoutsetsrequest_timeout; watches deliberately carry no overall deadline (bound them withtimeout_seconds).KuberException— no(result, response)tuples.Kuber.is_retryablereplacesOpenAPI.Clients.is_request_interrupted, andKuber.getpropertyat/haspropertyatreplace the accessors 1.0 dropped.OpenAPIv1ConsumerGaps.mdis the working document for all of this — every break with a stableC…/G…identifier, what it costs, and a per-repo checklist.Watches
The subtlest part, and worth reviewing closely. There are no dedicated watch operations — the deprecated
/watch/paths are deliberately not patched back in. Watching iswatch=trueon the list op with an accept-scoped codec (application/json;stream=watch), which fires only for calls that ask for it, because a real apiserver replies bareapplication/json. Since the generated call returns at the response head,list's watch branch pumps the stream inline and returns only when the watch is over — that is what keepswatch(processor, ctx, list, O)and itsfinally close(stream)blocks working.Four watch-lifecycle bugs were found and fixed during the trial (consumer-close as the only stop signal, resume from the last
resourceVersion, in-stream 410 restarting without one, and a re-watch spin on a 200-with-no-events).test/watch_recovery.jlcovers all of it against a fake apiserver, and the live suite covers the long-lived path: a watch driven across several apiserver-initiated clean closes, asserting exactly-once delivery and decoding realBOOKMARKframes.Testing
CI is green on Julia 1.11, 1 and nightly (run 33587911204), resolving
OpenAPI v1.0.0from the General registry, with the live suite running against a Kubernetes 1.35kindcluster — not skipped.test/registry.jltest/register.jltest/helpers.jltest/simpleapi.jltest/retries.jltest/watch_recovery.jltest/runtests.jllive suiteregistry.jlis the generation gate: every table entry resolves and no deprecatedwatch*operation leaks in. The live suite is skipped with a warning when no server is reachable. Its assertion count varies run to run (870/890/895 observed on the same cluster) because two loops assert per observed event — the watch-event sweep and theBOOKMARKframes a three-second watch happens to receive.Note that CI's cluster version is now load-bearing, which it was not on the 0.2.x line: responses are validated against the v1.35.4 schemas the client was generated from, so the workflow pins kind
v0.32.0with thev1.35.5node image. A spec bump has to move that pin with it.Not in
runtests.jl, kept as manual probes:test/characterize_retries.jl(pins the runtime's exception types — rerun at 1.0.0, all six classifications unchanged),test/watch_latency.jl(median 8.5–11.6 ms reactions, 0 missed events) andtest/watch_longevity.jl(an hours-long watch with descriptor and live-bytes tracking).Consumers
JuliaRun is ported, and its own integration suite passes end to end against k3s v1.35.4 — all 23 sections, including the parallel ones that run the ported JuliaRun inside the pod from an image rebuilt off this branch. That is the strongest evidence the shipped layer is enough: nothing needed
Kuber.register!.The port also retired
JuliaHubK8sApi.jlrather than regenerating it — Kuber's shipped layer is a superset of what consumers actually reach. What the port cost, the four findings the survey did not predict (C11–C14), and the remaining per-repo checklist are all in the gaps document.Known limitations
metrics.k8s.io/v1beta1ships, captured from a live cluster viafetch_specs.sh --from-cluster— aggregated APIs are not in upstream release-tag specs. Node and pod metrics work against any cluster running metrics-server.custom.metrics.k8s.iowas captured, evaluated, and deliberately not shipped. Its operations carry neitherx-kubernetes-group-version-kindnorx-kubernetes-actionand address metrics through a three-variable path, soemit_registry.jland the verb API cannot carry them; shipping it would addKIND_TYPESentries with noOPS. The captured document is kept as evidence ingen/openapi_v1/reference-captures/. See C5.Kuber.register!is untested in anger. It is covered offline against a hand-written fake group module; the first real registration will be whatever CRD or aggregated group someone needs next.