Protocol Stack Refactor

[tfpauly]

This is a big one! Thanks for your patience in reviewing this -- it's all very entangled, so can't meaningfully be broken up. This is a very large API break for adopters, but allows us to push through performance and architectural bottlenecks that otherwise would not be possible.

Here is some AI-assisted explainer content to help in your review and evaluation.

Summary

Protocol-to-protocol dispatch stops going through a type-erased ProtocolInstanceReference and instead goes through concrete, per-direction linkage structs that switch on a closed enum of framework protocols, while all mutable per-instance state moves into a ~Copyable NetworkContext.EventContext threaded inout through every call.

Major changes

1. ProtocolInstanceReference is gone

Protocols/ProtocolInstanceReference.swift is deleted and zero references remain in Sources. It
splits in two:

• InstanceIdentifier (new — Protocols/InstanceIdentifier.swift) is a pure identity/hash token
  wrapping a NetworkStateIndex, plus an optional parent index.
• The dispatch half moves into Protocols/BaseProtocolLinkages.swift (+2,548 lines — the biggest
  single file to read).

Each Base{Inbound,Outbound}{Datagram,Stream}Linkage holds a ProtocolType enum naming every
framework protocol and switches on it:

public struct BaseInboundDatagramLinkage: InboundDatagramLinkage, @unchecked Sendable {
    enum ProtocolType {
        case unknown
        case udp(NetworkStateIndex)
        case ip(NetworkStateIndex)
        case tcp(NetworkStateIndex)
        case demux(NetworkStateIndex)
        case datagramEndpointFlow(ProtocolInstanceBox<DatagramEndpointFlowProtocol<BaseDatagramLinkageFamily>>)
        case quicPath(ProtocolInstanceBox<QUICPath>)
        #if !NETWORK_EMBEDDED
        case external(any ExternalInboundDatagramLinkage)
        #endif
    }
    ...
}

So every hop is a direct call on a known type — no existential, no unspecialized generic. Read one
linkage method (e.g. handleConnectedEvent) and the other ~50 are the same shape.

2. NetworkContext.EventContext is a ~Copyable struct passed inout everywhere

public struct EventContext: ~Copyable {
    let globals: NetworkContext.Globals
    let scheduler: any NetworkContext.Scheduler
    let schedulerIsDefault: Bool

    internal var protocolEventStates = NetworkGappyArray<ProtocolEventManagerState>()
    ...
}

It owns protocolEventStates and is effectively an exclusive-access token: holding it proves you're
on the context's scheduler. Almost all of the QUIC diff (QUICConnection.swift +1,263, and the rest
of QUIC/) is mechanical in eventContext: threading — skim it; the interesting bits are elsewhere.

The convention introduced throughout: the no-context overload is the external entry point (it
does fromExternal { … }), and the in:-taking overload is for callers already inside the stack.

/// This is an external entry point: call it from code outside the protocol stack.
public func invokeConnect() {
    fromExternal { eventContext in
        invokeConnect(in: &eventContext)
    }
}

/// Requests that the lower protocol start connecting, using an already-acquired event context.
public func invokeConnect(in eventContext: inout NetworkContext.EventContext) {
    lower.invokeConnect(for: self.identifier, in: &eventContext)
}

Other notable changes

• Unregistering protocol instances from the context on teardown is now enforced with preconditions, and there is no more async during a deinit of a protocol. This is far better, but requires precision in teardown.

• OneToOneProtocol and ManyToManyProtocol no longer have special cases around attaching that don't work on embedded. Generic types are more consistently used.

Test / packaging shape

• New SwiftNetworkTestHarness target (Package.swift). The harness protocols move out of
  SwiftNetwork proper, so the NETWORK_NO_TESTING_HARNESS cases disappear from the production enum.
  Both test targets, SwiftNetworkBenchmarks, and the three tools now depend on it.

• TestProtocolLinkages.swift is the test of the extension point. It
  subclasses BaseNetworkProtocolStorage and defines a parallel Test* linkage family for protocols
  the framework doesn't know about:

  public struct TestDatagramLinkageFamily: DatagramLinkageFamily { ... }
  public final class TestNetworkProtocolStorage: BaseNetworkProtocolStorage { ... }

Performance

Comparison of main against this branch for IPUDPTransfer, QUICStreamLoad -stream-count 100000 -download-size 10000, and QUICTransfer.

All three benchmarks improve: IPUDPTransfer ~18% cheaper, QUICStreamLoad ~8.7% cheaper,
QUICTransfer ~1.4% cheaper.

Benchmark	Metric	main	branch	Change
IPUDPTransfer	CPU samples	4902	3993	−18.5%
	Cycles @4.0 GHz (derived)	4.97 G	4.05 G	−18.5% (−0.92 G)
	CPU time	1.242 s	1.011 s	−18.5%
	Wall clock	1.241 s	1.015 s	−18.2%
QUICStreamLoad	CPU samples	16266	14853	−8.7%
	Cycles @4.0 GHz (derived)	16.30 G	14.88 G	−8.7% (−1.42 G)
	CPU time	4.074 s	3.720 s	−8.7%
	Wall clock	4.207 s	3.835 s	−8.8%
QUICTransfer	CPU samples	47588	46913	−1.4%
	Cycles @4.0 GHz (derived)	53.52 G	52.76 G	−1.4% (−0.76 G)
	CPU time	13.380 s	13.190 s	−1.4%
	Wall clock	16.151 s	15.756 s	−2.4%

QUICStreamLoad detail

Many concurrent streams rather than one bulk transfer, so per-flow identity and event dispatch
carry much more of the profile (crypto is 28% of CPU here vs 42% in QUICTransfer). Self-time by
category, whole run:

Category	main	branch	Δ
Value-witness copy / destroy	661	239	−422	−64%
Other runtime	1747	1494	−253	−14%
Exclusivity checks	1720	1484	−236	−14%
ARC (retain/release)	838	636	−202	−24%
memmove / memset	795	658	−137	−17%
Dictionary / hashing	291	202	−89	−31%
Framework + tool code	3704	3617	−87	−2%
Generic metadata / witness	417	343	−74	−18%
Protocol thunks	80	48	−32	−40%
malloc / free	1237	1286	+49	+4%
Crypto (corecrypto / CommonCrypto)	4612	4686	+74	+2%
Total	16266	14853	−1413	−8.7%

[agnosticdev]

First pass complete. Looks good!

[agnosticdev]

What is the wrapping here of ProtocolInstanceBox about?

[tfpauly]

That's to make sure it is a type that is hashable and equatable on ===

[agnosticdev]

Is self set to borrowing here to provide flexibility for non-copyable types or is this left over from something?

[tfpauly]

Yes, it lets it work for non-copyable

[agnosticdev]

So this would be the external stream protocol (or one of them) that the server side folks could adopt then, correct?

[tfpauly]

Correct, and this is demonstrated in the test code

[agnosticdev]

Awesome!!!

[agnosticdev]

Ohhhhh yeah!!

[rpaulo]

This style is more confusing IMO