Gossipsub: make partial message part forwarding normative - #737
Open
MavenRain wants to merge 2 commits into
Open
Gossipsub: make partial message part forwarding normative#737MavenRain wants to merge 2 commits into
MavenRain wants to merge 2 commits into
Conversation
Promote forwarding of individually verifiable message parts from a motivation-level possibility to normative requirements, and bump the partial-messages spec revision to r1. The new Forwarding Message Parts section specifies that applications SHOULD validate and forward individually verifiable parts before the full message is known, MUST NOT forward parts that failed validation, and that implementations MUST accept incremental partial publishes for a group. This codifies behavior that is already deployed: the rust-libp2p implementation (libp2p/rust-libp2p#6275) places no completeness requirement on published partials, and Lighthouse (sigp/lighthouse#8314) validates and republishes incomplete data columns as cells arrive. Signed-off-by: Onyeka Obi <softwareengineerasaservant@isurvivable.cv>
gmelodie
reviewed
Sep 2, 2026
Comment on lines
+71
to
+72
| [Forwarding Message Parts](#forwarding-message-parts)), allowing us to | ||
| reduce the store-and-forward delay [2]. |
Contributor
There was a problem hiding this comment.
Suggested change
| [Forwarding Message Parts](#forwarding-message-parts)), allowing us to | |
| reduce the store-and-forward delay [2]. | |
| [Forwarding Message Parts](#forwarding-message-parts)), helping to | |
| reduce the store-and-forward delay [2]. |
| transferred. In order for fanout and gossip messages to be useful, the | ||
| Application MUST be able to send partial messages to these peers. | ||
|
|
||
| ## Forwarding Message Parts |
Contributor
There was a problem hiding this comment.
maybe
Suggested change
| ## Forwarding Message Parts | |
| ## Forwarding Message Parts Without Knowing Full Message |
Comment on lines
+206
to
+210
| Some applications can validate a message part on its own, without the full | ||
| message. Such a part is individually verifiable. For example, a node can | ||
| validate a DAS cell against its KZG commitment without the other cells in the | ||
| column. | ||
|
|
Contributor
There was a problem hiding this comment.
maybe we could just remove this paragraph
Suggested change
| Some applications can validate a message part on its own, without the full | |
| message. Such a part is individually verifiable. For example, a node can | |
| validate a DAS cell against its KZG commitment without the other cells in the | |
| column. |
| partial messages for the topic, without waiting for the full message. This | ||
| removes the store-and-forward delay of full message relay [2]. | ||
|
|
||
| An application MUST NOT forward a part that failed validation. |
Contributor
There was a problem hiding this comment.
Suggested change
| An application MUST NOT forward a part that failed validation. | |
| An application MUST NOT forward a part that failed validation or that cannot be individually validated. |
| To support this, implementations MUST accept a partial message publish for a | ||
| group before the application has the full message, and MUST accept repeated | ||
| publishes for the same group as the application obtains more parts. | ||
| Implementations SHOULD track per-peer state so that repeated publishes only |
Contributor
There was a problem hiding this comment.
Implementations SHOULD track per-peer state
Maybe you can be a bit more descriptive on how this is supposed to happen implementation-wise? I know many of these details are for each implementation to decide, but I found that "track per-peer state" is a bit too vague
also, nit:
Suggested change
| Implementations SHOULD track per-peer state so that repeated publishes only | |
| Implementations SHOULD track peer state so that repeated publishes only |
Comment on lines
+225
to
+227
| Parts that are not individually verifiable MUST NOT be forwarded before the | ||
| full message validates. Applications with such parts fall back to | ||
| store-and-forward relay of the reconstructed full message. |
Contributor
There was a problem hiding this comment.
i think if we apply my third suggestion we can get rid of this part
Suggested change
| Parts that are not individually verifiable MUST NOT be forwarded before the | |
| full message validates. Applications with such parts fall back to | |
| store-and-forward relay of the reconstructed full message. |
Address the first review round on the normative part forwarding section of the partial-messages spec: - Rename the section to "Forwarding Message Parts Without Knowing Full Message" and update the Motivation link to match. - Soften the Motivation bullet from "allowing us to reduce" to "helping to reduce" the store-and-forward delay. - Drop the paragraph that defined individually verifiable parts; the Terms section already defines a Message Part as the smallest verifiable part of a message. - Fold the "cannot be individually validated" case into the single MUST NOT forward rule and remove the now redundant closing paragraph. - Replace "track per-peer state" with "track peer state" and add a concrete description of that state: for each peer and Group ID an implementation keeps the parts it believes the peer has and the partsMetadata it last sent, sends only the difference on repeated publishes, and drops both records after a bounded number of heartbeats. This matches the rust-libp2p implementation (libp2p/rust-libp2p#6275). Signed-off-by: Onyeka Obi <softwareengineerasaservant@isurvivable.cv>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The partial-messages spec mentions forwarding parts before full receipt only as a motivation-level possibility ("partial messages could also be forwarded"). This PR promotes that behavior to normative requirements and bumps the revision to r1.
The new Forwarding Message Parts section specifies:
The Motivation bullet now points at the new section instead of the "could" phrasing.
Why normative, and why now
This codifies behavior that is already deployed rather than proposing new behavior:
publish_partial: it caches the application's partial, supports incremental re-publish of a growing group, and skips sends whose metadata adds nothing.PartialDataColumnAssembler::merge_partialsreturnsupdated_partials, whichgossip_methodspublishes viapublish_partial).Making the forwarding rule normative gives other implementations and applications a clear interop target for the store-and-forward latency win that motivated the extension, and pins down the safety condition (individual verifiability plus validation before forwarding) that the deployed code already honors.
Relationship to the v1.4 work
Complementary to #654 / #720: those define fragmentation and pipelined relay for opaque payloads at the protocol layer, while this change covers applications with application-defined verifiable parts on the shipped v1.3 partial-messages extension.