Skip to content

Allow overriding the ID generator (event_id / trace_id / span_id) for snapshot-restored runtimes such as AgentCore Runtime V2 #24728

Description

@dmnlk

Problem Statement

We run a Node.js service on Amazon Bedrock AgentCore Runtime, platform version V2. On V2 every new instance is restored from a single memory snapshot of an already-initialized process instead of booting from scratch (the same model as Lambda SnapStart, but without restore hooks). Anything the SDK computed or cached before the snapshot is shared by every restored instance.

This breaks the uniqueness of the IDs the SDK generates:

  • uuid4() (packages/core/src/utils/misc.ts) uses crypto.randomUUID(). In Node, randomUUID() serves UUIDs from a 128-entry entropy cache (lib/internal/crypto/random.js, kBatchSize = 128). Sentry.init() already consumes entries from that cache (every Scope constructor calls generateTraceId()), so the snapshot contains a partially consumed cache. After a restore, the first ~125 uuid4() calls return the same values on every instance.
  • Once the cache is exhausted it is refilled from OpenSSL's DRBG, whose state is also in the snapshot and only reseeds after 256 requests or 1 hour (OpenSSL 3.x defaults). So the window is not just the cache; within the first hour after a runtime is created or updated, restored instances keep producing identical IDs.
  • event_id, trace_id (generateTraceId) and span_id (generateSpanId) all come from uuid4().

Observed effect: the first error captured on each restored instance has the same event_id. Sentry keeps only the first event with a given ID and drops the others as duplicate, so an error that hits N freshly started instances shows up as a single event. Trace and span IDs collide the same way, so distributed traces from different instances are merged.

We have verified the same mechanism empirically for other ID sources in the same environment (20 instances restored concurrently, first request on 19 of them produced only 3 distinct random values from crypto.randomBytes).

Nothing in the SDK can detect a restore, so this is not something the SDK can fix on its own. What is missing is a supported way to plug in our own ID source. Today:

  • beforeSend can rewrite event.event_id and the envelope uses the rewritten ID, but captureException() / captureEvent() still return the original ID (hintWithEventId.event_id), so lastEventId() and user-feedback linking point at an ID that was never stored.
  • hint.event_id works for manual captures but not for errors captured by integrations.
  • There is no way at all to influence trace_id / span_id generation.

SDK: @sentry/node 10.70.0 (the relevant code is unchanged on main), Node.js 24.

Solution Brainstorm

An SDK option to override the ID generator, applied to event_id, trace_id and span_id alike, for example:

Sentry.init({
  // called for every event_id / trace_id / span_id; must return 32 hex chars
  idGenerator: () => readFromDevUrandom(16).toString('hex'),
});

or, at a lower level, letting uuid4() consult a client-level override instead of always using crypto.randomUUID().

For snapshot-restored runtimes the only entropy source in Node that is reseeded on restore is the kernel CRNG, i.e. reading /dev/urandom directly (AWS documents os.urandom() as the safe source for AgentCore V2). With such a hook we could route all Sentry IDs through it. A smaller improvement that would already help would be to make beforeSend's rewritten event_id flow back into the captureException return value, but that still leaves tracing IDs uncovered.

Additional Context

Priority

React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions