fix(routing): never re-send a delivery the callee is still holding (#1172)

[rbuergi]

Fixes #1172

Is #1172 the cluster's root? It is the cluster's REPORTER, and this is the third root behind it

feature:routing-back-pressure is one log site, not one defect. All 7 open issues are the same
RoutingGrain critical; the fingerprint folds in the activation id and episode number, which is why
one saturation event on 2026-09-19 fanned out into six separate tickets (#4908's own body says so).
#1172 is simply the oldest instance, filed 2026-08-10.

Across the 10 log samples recorded in the 7 issues, every single one reads 64 route dispatches in flight and routing pool subscribing 0 — 64 because ReportSaturation fires on the one increment
that crosses the threshold and then latches, so 64 is the only value it can print; 0 because the
pool's in-flight counter spans only source.Subscribe, which for a route leg returns as soon as the
grain call is initiated. Neither number carries information. The number that does is deepest:

shape	samples	reading
destinations 0, deepest 0	8 of 10	breadth — 64 grain-branch legs in flight, nothing waiting on anything
destinations 1, deepest 62 on one cache/…	2 of 10 (fcf31578#14, 67341d12#7)	head-of-line on a single destination

This PR fixes the mechanism that put up to 7× more legs into that budget than the traffic justified,
and held each one ~7× longer. It is the third root named at this site: #1341 was O(node-size) JSON
patch construction on hub turns, #1358 was a genuine slot LEAK (SubscribeThroughPool terminating an
observer in neither direction when the drain cancelled it). This one is a classification error
rather than a cost.

What I measured, and the defect

Orleans' ResponseTimeout is a caller-side give-up timer, not a cancellation — when it fires the
request has already been handed to the target activation and stays in its work queue until that
activation gets to it. So a rejection and a timeout are opposite facts about who holds the request:

• OrleansMessageRejectionException ("… to invalid activation. Rejecting now.") — the callee
  REFUSED it and holds nothing. Re-invoking re-resolves placement and the message lands on a
  fresh activation. This is the case the retry was built for (Retry a transient pod-hub delivery rejection instead of NACKing on the first attempt #2314).
• TimeoutException — the callee ACCEPTED it and has not answered yet. Re-sending duplicates
  it.

The retry gated on IsTransientFailure, which matches TimeoutException. And nothing on the receive
path can recognise a repeat: MessageHubGrain.DeliverMessage ends in
hub.DeliverMessage(delivery) — an unconditional post onto the target hub's queue — and I found no
reader of IMessageDelivery.Id that dedupes anywhere in src/. So that is duplicate handler
execution, not a retry.

The standing rationale for the generosity was "it is bounded by a retry budget". The budget is
bounded in attempts (6) and its delays (250 ms → 3 s, 9.75 s total) are sized for a rejection,
which Orleans returns instantly. A timed-out attempt costs the transport's whole ResponseTimeout
instead:

fault class	attempts	wall clock per delivery	copies queued at the callee
rejection	7	~10 s	1
response timeout	7	~3 m 40 s	7

And the coupling runs the wrong way, which is what makes it an amplifier rather than waste. A
response timeout on a delivery leg happens precisely when the destination is slow — a per-node hub
whose HubReady has not emitted, which for an _Activity/compile address means an in-mesh NodeType
compile — or when the silo is CPU/thread starved. In both cases the retry multiplies the work of the
thing that was already too slow, at the moment it has least capacity, while each leg holds one
inFlightRoutes slot for the whole 3 m 40 s. #1172's original evidence is dominated by
_Activity/compile and _Activity/import targets; on this reading those are not incidental, they are
the addresses whose HubReady takes longest, i.e. exactly the ones the ladder re-sent to seven times.

The fix — a third predicate, not a narrowed one, and no bound is touched

IsTransientFailure               "is another attempt CONCEIVABLE?"      unchanged
  ⊇ IsResendableDeliveryFailure  "may we SEND THIS REQUEST AGAIN?"      new gate
      ⊇ ClassifyDeliveryException == ShuttingDown
                                 "should the SENDER keep recovering?"  unchanged

IsResendableDeliveryFailure = !IsResponseTimeout(ex) && IsTransientFailure(ex), walking the
exception GRAPH (ExceptionChain) rather than the InnerException line — these faults arrive
through Rx Catch arms and two-transport AggregateExceptions where which fault sits at index 0 is a
race. It gates both non-idempotent retry sites: RoutingGrain.DeliverToGrainObservable (serving
IMessageHubGrain.DeliverMessage and IPodHubGrain.Deliver) and
OrleansRoutingService.DispatchObservable (IRoutingGrain.RouteMessage, whose callee queue is the
one that reached 541 deep in the 2026-08-07 incident).

IsTransientFailure is deliberately left intact — its other caller,
AttachWithBoundedRetry's IPodHubGrain.Attach claim, IS idempotent (it sets flags and re-pins an
activation), so re-sending it after a timeout costs nothing and is how the claim converges (#2633).
Narrowing in place would have silently disarmed that retry. It is also still the right answer to "is
this fault transient" and is pinned as such by OrleansDirectoryInstabilityClassificationTest and
StreamPostTimeoutAttributionTest, both of which stay green untouched.

No retry count, timeout, pool size or threshold is changed, and nothing is swallowed. The fault
reaches the same arm it reached after the retries were exhausted, and ClassifyDeliveryException
already answers a bare TimeoutException with the terminal ErrorType.Failed — deliberately, because
telling a consumer "transient" arms an unbounded resubscribe against a plausibly-wedged target. The
sender therefore gets the identical verdict it always got, one ResponseTimeout after the first
attempt instead of seven of them later.

Negative control

TimedOutDeliveryIsNotResentTest (6 facts, 694 ms, no cluster, Scheduler.Immediate).

Reverted only the two gate expressions back to IsTransientFailure, left everything else in
place, rebuilt Release:

failed ATimedOutDelivery_IsSentExactlyOnce
  Expected value to be 1 because an Orleans response timeout means the CALLEE ACCEPTED this
  delivery and has not answered yet … but found 7.
failed ATimeoutCarriedInsideAnAggregate_IsAlsoNotResent(timeoutFirst: True)
  Expected value to be 1 because a timeout anywhere in the aggregate is the same fact as a bare
  one … but found 7.
total: 6  failed: 2  succeeded: 4

The found 7 is the defect stating its own size. The 4 that stay green are the ladder pins and the
rejection path — i.e. the control on the other side of the change:
ARejectedDelivery_IsStillResentUntilTheBudgetIsSpent asserts a rejection still spends its full
budget both before and after, so the fix cannot be "the retry was disabled". timeoutFirst: False
also passes in both directions and is honest about it (an InnerException-only walker could not see
index 1 at all) — the doc page says which half is the discriminator and which is the graph-walk pin.

Restored, verified byte-identical to the saved diff, re-ran: 6 of 6 pass, plus 68 of 68
neighbouring routing/classification/dispatcher tests and 601 of 601 documentation tests.

What the other six would inherit, and what they would NOT

Inherited — the shape all 8 deepest 0 samples show gets up to 7× fewer legs in the 64-slot
budget and each held for one ResponseTimeout instead of seven, so crossings become rarer and
episodes shorter. On the reading above that is the dominant contributor to the eight breadth samples.

NOT inherited, and I am not claiming it is:

• Head-of-line, deepest 62 on one cache/… (RoutingGrain repeatedly hits 64-dispatch back-pressure threshold on memex-portal pods, with one episode head-of-line blocked on a single cache destination #4897, RoutingGrain dispatch pool saturates with all 64 slots in flight; one destination holds 62 queued legs (head-of-line block), fleet-wide today #4899, Routing grain hits dispatch back-pressure threshold on both portal pods simultaneously; one pod shows a 62-deep head-of-line queue on a single cache destination #4900, RoutingGrain back-pressure recurs: 62-deep head-of-line pileup behind cache hub legs persists past issue #4903's window #4904, RoutingGrain dispatch slots held indefinitely under ThreadPool starvation; back-pressure log fans out into per-activation incidents #4908). Independent
  root: OrderedRouteDispatcher's FIFO key is the destination address while the ordering
  invariant it protects (SynchronizationStream's receive-side monotonicity guard) is per stream
  — so all traffic to one process's cache hub is serialised whether or not any two frames have an
  ordering relationship at all. That over-broadness is what lets one slow destination stack ~62 legs.
  Already ticketed at sev:M; deliberately out of scope here rather than filed again.
• The unbounded claim→subscribe window. A slot is claimed at dispatch/enqueue, but the leg's own
  timeouts are Rx operators inside the cold observable and do not start until
  SubscribeThroughPool obtains a ThreadPool thread. The report's own text names this; it is not
  closed by this change.
• routing pool subscribing is a structurally uninformative field — 0 in all 10 samples, and
  ~always 0 by construction. Noted for whoever next touches that report.

So: the sev:H framing in #1172's body ("dispatches that never complete", "the Orleans
NonReentrancyQueueSize limit", "all further routing on that silo stalls") was already disproven
in its own comments, and every root named for it is now fixed — #1341, #1358, and this. The
residual load and head-of-line shapes at that log site are tracked at sev:M/L on the six newer
tickets, which is where I think they belong; reopen if you read the disposition differently.

Verification

• dotnet build -c Release -warnaserror, one project per invocation, 0 Error(s) 0 Warning(s):
  MeshWeaver.Connection.Orleans, MeshWeaver.Hosting.Orleans,
  test/MeshWeaver.Hosting.Orleans.Test (which transitively builds the only two dependents,
  Hosting.Orleans.TestBase and the test project itself — nothing else in src/, test/ or
  samples/ references either touched project), test/MeshWeaver.Documentation.Test.
• No public surface added or removed (IsTransientFailure and both new predicates are internal), so
  no Pairs-with: / Implementers: / Mirror-sync: obligation.
• No What's New file — per release now, not per change. The durable record is the doc page
  Doc/Architecture/ATimedOutDeliveryIsStillHeldByTheCallee, listed in the Architecture topic map
  (ArchitectureTopicMapTest enforces that).
• Addresses to recycle on deploy: none. The change is compiled src/ on the silo path, not node
  content and not a NodeType, so it takes effect with the image roll.

🤖 Generated with Claude Code

[Copilot]

Copilot review overview

🟢 Approval recommended

The change is narrowly scoped to retry gating, is backed by deterministic tests that fail on the regression and pass with the fix, and does not introduce async/await into production routing code paths.

Review effort: Lite
Findings: None

What changed in this PR

This PR fixes a routing back-pressure amplifier by ensuring Orleans delivery retries do not re-send a delivery after a response timeout (where the callee may still be holding/processing the original request), while keeping bounded retries for genuinely resendable/transient failures (e.g., Orleans rejections).

Changes:

• Gate non-idempotent delivery retries on a new “resendable” predicate (IsResendableDeliveryFailure) that excludes response timeouts (walks the exception graph via ExceptionChain).
• Add a deterministic unit test suite proving: timeouts are sent exactly once; rejections still spend the retry budget; aggregate exception ordering doesn’t change the decision.
• Add an architecture doc page explaining the timeout vs rejection distinction and the three-predicate “ladder”, and link it into the architecture index.

File	Description
test/​MeshWeaver.Hosting.Orleans.Test/​TimedOutDeliveryIsNotResentTest.cs	Adds regression + control tests covering timeout non-resend, rejection resend, and aggregate exception ordering.
src/​MeshWeaver.Hosting.Orleans/​RoutingGrain.cs	Switches DeliverToGrainObservable retry gate from IsTransientFailure to IsResendableDeliveryFailure; adds the new predicate.
src/​MeshWeaver.Connection.Orleans/​OrleansRoutingService.cs	Switches RouteMessage retry gate to IsResendableDeliveryFailure; introduces IsResponseTimeout (exception-graph walk) and the resendable predicate.
src/​MeshWeaver.Documentation/​Data/​Architecture/​ATimedOutDeliveryIsStillHeldByTheCallee.md	New architecture note documenting the mechanism and the predicate ladder.
src/​MeshWeaver.Documentation/​Data/​Architecture.md	Adds the new architecture note to the Architecture topic map.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Test Results (shard 4)

0 tests   0 ✅  0s ⏱️
0 suites  0 💤
0 files    0 ❌

Results for commit 524d126.

[github-actions]

Test Results (shard 1)

0 tests   0 ✅  0s ⏱️
0 suites  0 💤
0 files    0 ❌

Results for commit 524d126.

[github-actions]

Test Results (shard 3)

0 tests   0 ✅  0s ⏱️
0 suites  0 💤
0 files    0 ❌

Results for commit 524d126.

[github-actions]

Test Results (shard 5)

0 tests   0 ✅  0s ⏱️
0 suites  0 💤
0 files    0 ❌

Results for commit 524d126.

[github-actions]

Test Results (shard 2)

601 tests   601 ✅  27s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit 524d126.

[github-actions]

Test Results (shard 0)

  1 files    1 suites   1m 57s ⏱️
300 tests 300 ✅ 0 💤 0 ❌
302 runs  302 ✅ 0 💤 0 ❌

Results for commit 524d126.

[github-actions]

Test Results

  2 files    2 suites   2m 24s ⏱️
901 tests 901 ✅ 0 💤 0 ❌
903 runs  903 ✅ 0 💤 0 ❌

Results for commit 524d126.