Proposed publisher API, generated from the consumption specification - #147
Proposed publisher API, generated from the consumption specification#1470x73746F66 wants to merge 19 commits into
Conversation
Signed-off-by: Chris Langton <chris@vulnetix.com>
Signed-off-by: Chris Langton <chris@vulnetix.com>
- Add authentication schemes
- Create discovery endpoints (/.well-known/tea/{id})
- Standardise property naming between consumer/publisher APIs
- Implement advanced filtering for components and artifacts
- Add lifecycle status endpoints
Signed-off-by: Chris Langton <chris@vulnetix.com>
|
Thank you for updating the Publisher spec! 💯 It was certainly way of out of date! However, I notice that the proposed spec has also a lot of subtle differences (schema names, parameter names, schema contents) from the current Consume API. Should we mark this PR as draft and work on it after the hackaton? |
Signed-off-by: Chris Langton <chris@vulnetix.com>
Signed-off-by: Chris Langton <chris@vulnetix.com>
Signed-off-by: Chris Langton <chris@vulnetix.com>
Signed-off-by: Chris Langton <chris@vulnetix.com>
taleodor
left a comment
taleodor
left a comment
There was a problem hiding this comment.
Thank you for the Pull Request! Did first read through for the document, there may be more things to discuss later, but raised issues in the comments that require further work and / or discussion.
| post: | ||
| description: Create TEA Product entry for the supplied product identifier | ||
| operationId: createTeaProduct | ||
| requestBody: |
There was a problem hiding this comment.
Similar to above, most of those properties are not currently part of the Consumer API spec. Additionally, it would be helpful to create a shared Schema object as we did throughout the Consumer API.
There was a problem hiding this comment.
As above, refer to PURL in the consumer API
There was a problem hiding this comment.
Again, don't see how this is relevant to PURL at all.
There was a problem hiding this comment.
You're going to have to be clear then, because of you're referring to the markdown spec and find these then perhaps the consumer spec should be updated after beta1 to align like this has
There was a problem hiding this comment.
See my previous comment, those are listed in markdown as possible identifiers, we shouldn't have a separate field for each type and instead expand TeaIdentifier. We may also want to be a little stricter about identifier types - it's possible that markdown needs to be updated here as well.
There was a problem hiding this comment.
The consumer API returns the constructed PURL as a dynamic response property
To be able to do that, the publisher needs to gather data elements and store them to construct the PURL
A string that is a full PURL can be described by th publisher as a loaded optional string, where some values are duplicated every API call
Or an API spec that implements the intent of returning a partial PURL could do so wth partial PURL support by not including some of the additional elements I added
I believe reduction of duplicates, and addition of fields available in a PURL, is the best way to implement the spec.
If we do anything lese it is either adding duplication, and it is putting the responcibility onto publishers to know how to construct a PURL, and being optional this will probabl be skipped...
If we want to do ourselves a favour later when we implement the PURL support for searching, we should be forward thinking now
There was a problem hiding this comment.
The consumer API returns the constructed PURL as a dynamic response property
A single entity may have several PURLs - that is the state of the PURL spec at the moment, I specifically brought this question on the last PURL community meeting. Having consistent and unique PURLs is a noble goal, but I don't think that is achievable.
As a basic example, imagine a Docker image that also has code hosted on GitHub. You may have PURL pkg:github/... and pkg:docker/... for the same thing where both are valid. That is also part of the reason why separate TEI is needed.
Therefore, I believe we should actually leave it to publishers to define their identifiers.
Finally, PURL is very domain (type) specific and things like sku and barcode are not even basic PURL elements, so not sure why they are here in this context.
| $ref: '#/components/responses/404-object-by-id-not-found' | ||
| tags: | ||
| - TEA Component | ||
| /component/{componentIdentifier}: |
There was a problem hiding this comment.
Suggest using same name for {uuid} across the spec and refer to the Schema uuid definition as in the Consumer API.
There was a problem hiding this comment.
that makes sense is we merge them and the meaning has no distinctions from publisher and consumer. i.e. a publisher does not have the primary key yet (until the database record is inserted) so it cannot be the same entity as the resultant response entity in the consumer API spec.
It is common for request models to be distinct from response models in all API designs - this is why I deveated deliberately
| $ref: '#/components/operations/standardDelete' | ||
| tags: | ||
| - TEA Release | ||
| /collection: |
There was a problem hiding this comment.
We should discuss whether explicit CRUD on Collections should be allowed, see #152
There was a problem hiding this comment.
A lot of the referenced issue has a mixture of obviously good ideas but I believe that dogma might be winning over critical thinking a little here. Without question why these views are better than allowing publishers the flexibilities is problematic, we automatically impose an arbitrary constraint. Perhaps constraints should be sparingly applied to a stnadard like TEA
There was a problem hiding this comment.
For me this is actually purely implementation prospective. I think we should add calls related to adding artifacts, then it becomes more clear that having both CRUD on collections and on artifacts conflicts with each other logically. For now, since artifact CRUD is not there it may be less visible.
There was a problem hiding this comment.
CRUD is a por description for this spec, given R for read is not apparent and therefore chaining requests that rely on prior call results with new unknown UUIDs can be a logical constraint which only a read can resolve..
Better to reduce chaining requests for creation, and offer incremental limited updater methods expected to be called at much later dates, dependant on having a timely GET request for freshness
Would you agree a single POST with everything is both concise, organised, and performant? With consideration any complications that can be addressed with a GET + PATCH chain should be back ported to the POST to maintain these desirable characteristics and avoid dependency on complicated chaining logic?
There was a problem hiding this comment.
This makes sense to me, but right now we have collection type update enum, which has things like Vex Updated, Artifact Updated, Artifact Removed, Artifact Added. If you do batch requests where you both add and update artifacts, this concept becomes murky.
Another issue with batch requests is that it eventually would lead to conversation about collection own lifecycle - which to me is more bureaucracy without much gain. I prefer a system where I upload an artifact and it becomes visible right away, instead of a system where I need to first upload an artifact (or a batch) and then approve a new collection on top of that. This is a point to discuss though.
There was a problem hiding this comment.
It's not only "approve" - it's in many cases "sing the collection" too. It's like a commit.
There was a problem hiding this comment.
We should have a conversation on this.
My current implementation idea is that signing should happen automatically on the backend - we can add constraints that artifacts should be signed, etc.
In SaaS scenarios, we can easily have dozens of releases per day for a single component and there can be many components. Each of those releases can have multiple collection versions. I cannot imagine a human signing all of them.
There was a problem hiding this comment.
Keyless signing (Cosign signature) is definitely a superior choice, however we'll need to support signing with Tuff, CMK, and PGP
There was a problem hiding this comment.
Cosign as a service is not a superior choice for a lot of reasons. The software is cool though. Signing is discussed elsewhere and we need to open that discussion.
Signed-off-by: Chris Langton <chris@vulnetix.com>
|
Awesome feedback @taleodor ty. I have some cnversations resolved and others with clarifications t odiscuss |
Signed-off-by: Chris Langton <chris@vulnetix.com>
…release paths Signed-off-by: Chris Langton <chris@vulnetix.com>
|
Just pinging to see if this still is an active PR that we need to merge as the basis of future work on the publisher API or if it's superseded by other PRs already merged. |
|
There are conflicts that needs to be handled. |
|
Update PR description, and rebased main. Publisher spec is not generated, so consumer updates are shared with publisher spec going forward |
…s and distribution
The document did not validate as OpenAPI 3.1, and the way it failed
mattered. `components.operations` is not a field OpenAPI defines, so the
three $refs pointing into it never resolved and every delete operation
declared no responses at all — a generator reading this produced deletes
that returned nothing. Redocly reported 9 errors; it now reports 0.
Error responses carried `application/json: {}`, an empty schema that says
nothing about what comes back. Publication runs in release pipelines, so
failures need to be actionable without a human reading prose: there is
now an `error` schema with a machine-readable code.
Three things were missing rather than broken.
Artifacts. The draft could create products, components, releases and
collections but had no way to publish the artifact that carries the SBOM,
VEX or attestation the exchange exists to move. Creating an artifact and
uploading its bytes are separate operations, because an SBOM is routinely
tens of megabytes and a publisher who must resend all of it to fix one
metadata field will not fix the field. Content goes up per format in its
own media type rather than base64 in JSON, stored bytes are immutable so
a recorded checksum cannot start describing different content, and
Content-Digest is verified so a truncated retry fails loudly instead of
publishing corruption under a checksum that claims otherwise.
Signatures. The consumption API exposes only a signatureUrl, leaving a
consumer to infer the scheme from the bytes. The publisher knows it, so
artifactSignature records the scheme, key id, certificate chain and
transparency-log entry — which is what makes a signature checkable rather
than merely present.
Distribution. TEA describes how data is fetched, not who may fetch it.
That boundary works for a consumption API and cannot hold for a
publication one, where the same server holds public material, material
shared with named counterparties, and internal material. Three
visibilities, plus mirroring to other TEA servers. Inheritance narrows
and never widens — a child may restrict what it inherits and a server
rejects any attempt to loosen it, because otherwise marking a product
private would guarantee nothing when any artifact beneath it could be
made public. GET /distribution/{uuid} reports declared against effective
and names the ancestor responsible, since the gap between those two is
where accidental disclosure hides.
Idempotency-Key on creates, because pipelines retry and a timeout that
actually succeeded otherwise yields a duplicate found by a consumer
rather than by the publisher.
The README records what changed and three open questions: releases are
not split into product and component as the consumption API splits them,
there is no publisher-side read surface, and publishTo does not say
whether a failed mirror blocks local publication.
Signed-off-by: Christopher Langton <chris@vulnetix.com>
# Conflicts: # contributors.md # spec/publisher/openapi.json
…d the drift
The publisher document was a hand-maintained copy of the consumption
specification and had fallen four minor versions behind it. The
consumption API is at 0.4.0; the copy still carried schemas from 0.0.3,
renamed along the way — artifactFormat beside the real artifact-format,
typeUuid beside uuid, a private artifactChecksum beside checksum. A
combined document therefore held two definitions of the same concept,
each free to move independently, with nothing to say which one a
publisher should believe.
It is now generated. build.mjs merges spec/openapi.yaml with a
publication overlay, so everything shared is a $ref into the consumption
specification's own definitions and the publisher version is copied from
the consumer rather than chosen. The build refuses to run if the overlay
declares a schema, parameter, response, request body or path method the
consumption specification already owns — the overlay may only add.
Alignment corrections this surfaced:
- Releases are split into /productRelease and /componentRelease. The
consumption API has distinguished the two since 0.1.0 and a single
/release could not say which it was creating.
- Collections are addressed through their release. The consumption
specification states a collection's UUID matches its release and that
an update only changes the version, so it is not an independently
created object: PUT /productRelease/{uuid}/collection, not
POST /collection.
- Errors reuse the consumption envelope. publisher-error-response has the
same { error } shape as error-response over an enumeration that repeats
OBJECT_UNKNOWN and OBJECT_NOT_SHAREABLE and adds the failures only a
writer can hit, so clients branch on one enumeration rather than two.
- distribution became access-policy, because the consumption
specification already uses release-distribution for a downloadable
build with its own URL and checksums. Reusing the word for access
control would have been actively misleading.
Enforcement of the access policy needs no new consumer behaviour: a
denied consumer gets OBJECT_NOT_SHAREABLE, which already exists in
unknown-error-type.
The generated document now adds no lint errors of its own. It reports 24,
and linting spec/openapi.yaml alone reports the same 24 — 23 operations
missing summaries and one OpenAPI 3.0 `nullable` on
pagination-details.nextPageToken. Those belong upstream rather than
patched here, where they would put the fork at odds with a file upstream
owns. CI compares the two counts and fails only if the overlay makes
things worse.
Signed-off-by: Christopher Langton <chris@vulnetix.com>
createTeaDistribution and deleteTeaDistribution, on the component release because that is where the consumption schema puts release.distributions. distributionId is absent from the request body by design: the server derives it, and a publisher-supplied identifier would let one tenant claim another's. identifiers stays optional — Homebrew and Scoop have no accepted purl type, and an identifier naming a coordinate that does not resolve is worse than none, because a consumer will follow it. Regenerated openapi.yaml. Redocly reports the same 24 errors as before, all inherited from the upstream consumption specification. Signed-off-by: Christopher Langton <chris@vulnetix.com>
Audited the overlay against a working TEA server and its console — Vulnetix
Resolve — rather than against itself. Five things the specification could not
express, each of which the implementation had to answer on its own.
A product release could not state its components. The consumption API lists
`components` among productRelease's required members, so every release a
conformant publisher could create had to be served with an empty list: a claim
about the product rather than a gap in the record.
An artifact could not say which distributions it describes. `distributionIds`
exists in the consumption artifact and was unreachable from the publication
side, which matters exactly where an SBOM is not one document.
There was no publisher read surface, and the consumption API cannot substitute
for one: it answers what a reader is entitled to see, so an object correctly
withheld and an object that was never created are indistinguishable from
outside. `GET /publications` and `/publications/{uuid}/releases` report what
was published alongside the policy in force for it.
`public` asked for "a separate confirmation" without naming one, so the
implementation invented `?confirm=public`. An unspecified requirement is one
every publisher satisfies differently, which is the failure this document
exists to prevent, so the parameter is now specified.
`publishTo` named targets nothing could configure and left failure semantics
undefined. Targets are now registered through `/publicationTargets`, mirroring
is asynchronous and never blocks the local write, per-target state is reported
on the access policy, and a server that does not mirror answers
`MIRRORING_UNSUPPORTED` instead of accepting the request and discarding it.
One change to the consumption specification: an empty alternative in the global
`security`. Without it `visibility: public` cannot mean what it says, because
discovery has no step at which a credential could be obtained and a server that
answers an anonymous request is non-conformant for doing so. It is a widening,
so nothing conformant breaks. build.mjs gives every overlay operation an
explicit credential requirement in return, so anonymous writes stay
non-conformant.
Redocly still reports the same 24 errors as the consumption specification
alone; the overlay adds none.
Signed-off-by: Christopher Langton <chris@vulnetix.com>
0x73746F66
changed the title
The step has failed on every run since it was added, and not for the reason it looks like: `npx redocly lint` exits non-zero whenever it reports an error, both documents report 24 by inheritance, and GitHub Actions runs the step under `bash -e`. With `set -o pipefail` the pipeline inherits that exit status, so errexit killed the script at the first assignment, before either count was echoed or compared. The log showed an exit code and no output, which is what made it read as a lint regression. Discard redocly's exit status in the helper, where the count already carries everything the status would have told us. Two things worth keeping while in here. Empty output now fails loudly rather than counting as zero errors, because a gate that passes when the linter did not run is worse than no gate. And the diagnostic lint in the failure branch gets `|| true`, so errexit cannot cut it short before the explicit exit. Verified against the step body extracted from the workflow itself, under `bash -e`: equal counts exit 0; a linter producing no output exits 1 with a message; a publisher document carrying one extra error exits 1 and reports the delta. Signed-off-by: Christopher Langton <chris@vulnetix.com>
The inheritance rule said a child may narrow what it inherits and must not widen it, and described resolution as taking the nearest declared policy. Those two statements do not combine into the guarantee the document claims. ACCESS_WIDENS_PARENT rejects a declaration that is too wide at the moment it is written. It says nothing about a parent narrowed afterwards. So under nearest-wins, an artifact that had declared public goes on being public once its product is set to private, because the artifact's own declaration is the nearest one and therefore becomes the answer. Marking a product private is then a statement about a single row, which is the opposite of what the section says it is for. Resolution is now the narrowest declaration anywhere between the object and the root. Where several levels declare shared, entitlements intersect and the earliest expiry applies, since a grant on a child cannot create an entitlement its parent withheld. Descendants' declarations are explicitly not rewritten when an ancestor narrows. Folding is reversible and rewriting is not: widening a product back restores each descendant to its own declaration and no further, which is what makes a temporary embargo something a publisher can undo. Redocly still reports the same 24 errors as the consumption specification alone. Signed-off-by: Christopher Langton <chris@vulnetix.com>
Every em dash in the publisher overlay, its README, the generator and the one comment added to the consumption specification is rewritten. Each was replaced according to the grammar of its own sentence, using a colon, a comma, a full stop or a parenthesis as the clause required, rather than by substituting one character for another. Wording only. `npm run check` reports the document still in sync, and redocly reports the same 24 errors on both documents. Signed-off-by: Christopher Langton <chris@vulnetix.com>
|
@oej i've added open questions in the PR description, but so far as having a consumer compliant publisher spec, generating and extending fromt he consumer is hard to argue agianst, if there is no objections we should merge this and break out the outstnadings into discrete work |
4 participants
Proposes a publisher API for TEA: how software gets into a transparency log. The
consumption specification already says how it comes out.
spec/publisher/openapi.yamlis generated. It isspec/openapi.yamlplus thepublication operations in
spec/publisher/overlay.yaml, merged byspec/publisher/build.mjs. Anything the two share is a$refinto theconsumption definitions, and the build fails if the overlay redefines something
the consumption specification already owns. The publisher version is the
consumer version by construction.
The earlier draft was a hand-maintained copy and had drifted to 0.0.3 schemas
under parallel names:
artifactFormatbesideartifact-format,typeUuidbeside
uuid. Generating the document removes that possibility instead ofcorrecting the symptom. Reasoning for each decision is in
spec/publisher/README.md.What the overlay adds
productReleaseandcomponentReleaseto match the consumption model.PUT /productRelease/{uuid}/collection, because the consumption specification saysa collection's UUID is its release's and an update only bumps the version.
updateReasonis required.often tens of megabytes, and a publisher who has to resend all of it to fix
one metadata field will not fix the field. Uploads verify
Content-Digest(RFC 9530).
transparency-log entry. The consumption API exposes only
signatureUrl, so aconsumer currently has to work out the scheme from the bytes before it can
verify anything.
known about it. They hang off a component release, which is where
release.distributionsis defined.private,shared(named organisations, optional expiry)or
public, inherited from artifact to collection to release to product.GET /publications, reporting what an organisation has published and who canread it.
publishTo, mirroring to targets registered at/publicationTargets.Idempotency-Keyon creates, because pipelines retry.Two rules in the access policy do most of the work. Inheritance narrows and
never widens, so a server refuses an attempt to loosen a child with
ACCESS_WIDENS_PARENT; without that, marking a product private would guaranteenothing, because any artifact beneath it could be made public on its own. And a
change to
publicrequiresconfirm=public, because no later request recallswhat has already been fetched.
None of it asks anything new of a consumer. A reader denied by a policy gets the
existing
OBJECT_NOT_SHAREABLE.One change to the consumption specification
spec/openapi.yamlgains an empty alternative in its globalsecurity, makingauthentication optional. This is the only edit to a file the working group owns,
and the only part of this PR that needs a decision rather than a review.
Without it the publication API cannot mean what it says.
visibility: publicisdefined as readable without authentication, and the discovery sequence (take the
domain from a TEI, fetch that host's discovery document, call the root it names)
has no step at which a credential could be obtained. A specification that
requires one everywhere closes its own entry point, and a server that does serve
a public object anonymously is non-conformant for it.
The change only widens what is allowed, so nothing conformant breaks. A server
may still refuse every anonymous request. Every operation in the consumption
document is a read, so nothing becomes writable. In exchange,
build.mjsgiveseach of the 24 write operations an explicit
bearerAuth/basicAuthrequirement, so anonymous writes stay non-conformant.
If the working group would rather leave the document alone, the alternative is
prose saying servers may relax it for public objects, and accepting that the
document and the behaviour disagree.
Compatibility with 0.4.0
The consumption specification is otherwise untouched.
diffbetween this branchand the copy that implementation validates against reports the security block
above and nothing else.
redocly lintreports 24 errors on the generated document and the same 24 onspec/openapi.yamlalone: 23 missing operation summaries and onenullableinpagination-details. CI compares the two counts and fails if the publisherdocument is ever worse than the specification it is built from.
Checked against a deployed implementation, Vulnetix Resolve.
23 of 23 consumption operations exercised, 160 test cases, 0 failures, 85
responses validated against the upstream schemas and all conforming. The
Insights API adds 2 operations and 27 cases, with 16 responses validated against
CycloneDX 1.6.
All four access modes work end to end. An anonymous consumer resolves a TEI at
the root advertised in
/.well-known/tea, then reads a public release, itscollection, its artifacts and their bytes with no credential at any step.
sharedgrants name organisation UUIDs with an optional expiry, and an expiredgrant is not a grant.
privateis the default and owner-only, withGET /publicationsas the way an owner confirms it. Discovery resolves bothurn:tei:uuid:andurn:tei:purl:, listings filter onidTypeandidValue,and retrieval walks release to collection to artifact to content. A refusal
answers
OBJECT_UNKNOWNand not 403, so nobody can enumerate anotherpublisher's UUIDs.
Those 160 cases cover the consumption operations. The publication operations are
implemented and in use, and the implementation now checks itself against this
document: every error code it emits must be a value
publisher-error-typedeclares, and every operation this document declares must be routed. They are
not yet driven by a conformance suite. That is the next piece of work, not a
claim being made here.
What implementing it found
Five things this document could not express, all fixed in it now.
A product release could not state its
components, which the consumptionproductReleaselists as required. An artifact could not setdistributionIds.publicasked for a separate confirmation without saying what one was, so ourimplementation uses
?confirm=public. AndpublishTonamed targets thatnothing could configure, with failure semantics the README itself admitted were
undefined.
Open questions
Artifact versions. The consumer API addresses artifacts as
/artifact/{uuid}/{artifactVersion}, but create and update do not say how apublisher advances that version.
Mirror authentication runs one way only. Nothing says how a receiving server
decides whether to accept a mirrored object, or how it records that the object
arrived as a copy. Consensus across transparency logs requires this to be resolved, or consensus is not possible and there is nothing trustworthy across transparency exchange servers.