Skip to content

Fix telemetry regressions from the applicationinsights 3.x upgrade - #2843

Merged
Akanksha Jain (jainakanksha-msft) merged 5 commits into
mainfrom
fix/telemetry-otel-regression
Oct 12, 2026
Merged

Akanksha Jain (jainakanksha-msft) merged 5 commits into
mainfrom
fix/telemetry-otel-regression

Conversation

@jainakanksha-msft

@jainakanksha-msft Akanksha Jain (jainakanksha-msft) commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Summary

Since Azurite 3.37.0 (released 24 August 2026), Application Insights has been receiving far more Azurite telemetry, and it arrives as unknown_service:node with no version. The cause is the applicationinsights upgrade from 2.9.6 to 3.x (#2660, then 3.16.0 in #2726).

Version 3 is a compatibility layer over OpenTelemetry. It accepts several settings Azurite used without errors but ignores them. The existing unit tests called the helper functions directly rather than going through the SDK, so they didn't catch this.

What went wrong in 3.x

Setting Azurite relied on Behaviour in 3.x Effect
context.tags[cloudRole] = "Azurite_V1.0" No longer sets the role. It is copied into every item as a custom field instead. cloud_RoleName = unknown_service:node and a stray ai.cloud.role custom dimension
Version read automatically from package.json Removed application_Version is blank
samplingPercentage = 1 setup()/start() and the default global client use the distro's default 5 traces/second rate limiter, which replaces the 1% ratio Instances below 5 requests/s send almost every request, about 100× the previous volume (for example Q_Messages_Dequeue polling)
addTelemetryProcessor(removeRoleInstance) No-op (it only logs a warning) The raw machine name is sent as cloud_RoleInstance
Global setup()/start() Installs global OpenTelemetry providers, HTTP instrumentation, Live Metrics and host/OS resource detection in the process host.id, host.name and os.version are sent in a per-batch _OTELRESOURCE_ metric. Live Metrics pings are sent. Global OpenTelemetry state is replaced inside the VS Code extension host.
maxBatchSize = 0 Not supported; telemetry is batched The Stop event can be lost at exit

Changes (src/common/Telemetry.ts)

  • Isolated clients: Create clients with new TelemetryClient(connectionString, { useGlobalProviders: false }) and stop calling setup()/start(). Each client gets its own providers and its own sampling ratio, and nothing global is installed.
  • Role, version and instance: Set them through the OpenTelemetry resource:
    • service.name = Azurite_V1.0, which keeps existing queries and dashboards working.
    • service.version = the Azurite version.
    • service.instance.id = SHA-256 of the hostname, matching the previous privacy-review behaviour.
  • Resource metric: Disable the per-batch _OTELRESOURCE_ metric. The environment variable is set only while the exporters are constructed, then restored.
  • Sampling: Track requests under ROOT_CONTEXT, so another OpenTelemetry setup in the same process (for example another VS Code extension) can't make its sampled span the parent and bypass 1% sampling.
  • Request counts: Set microsoft.sample_rate on requests so the envelope sampleRate is 1 again. itemCount then still estimates total requests, as it did with 2.x.
  • Stop event: Flush the clients when it is sent.
  • authorization property: Only report known scheme names (SharedKey, SharedKeyLite, Bearer). Anything else is reported as Other, so a malformed header can't leak a credential. Normal values are unchanged.
  • Dependencies: Add @opentelemetry/api and @opentelemetry/resources as direct dependencies. Both were already installed transitively at the same versions (1.9.1 and 2.10.0).

Evidence

I ran the built Azurite against a local fake ingestion endpoint (only the Azure Monitor HTTPS calls were redirected), with real Blob, Queue and Table SDK traffic:

Scenario Before After
~4 req/s queue polling, 100 dequeues: request records sent 97 2–3 (≈1%)
903-request burst: request records sent 6 (rate-limited) 9–12 (≈1%)
cloud_RoleName unknown_service:C:\Program Files\nodejs\node.exe Azurite_V1.0
application_Version (blank) 3.37.0
cloud_RoleInstance raw hostname SHA-256 hash
_OTELRESOURCE_ metric (host.id, host.name, os.*) sent not sent
Live Metrics pings sent none
Stop event delivered (polling run) no yes

Custom dimensions are back to Azurite's own properties. ai.cloud.role and _MS.ProcessedByMetricExtractors no longer appear.

Tests

tests/common/Telemetry.test.ts now sends telemetry through the real applicationinsights SDK to a local HTTPS ingestion server. It checks:

  • role, version and hashed instance
  • sampleRate is 1
  • no _OTELRESOURCE_ metric is sent
  • the environment variable is restored

A separate test sends 200 requests inside another library's sampled span and checks that only about 1% are sent. With the root-context isolation removed, all 200 were sent and the test failed. The PR also adds tests for the authorization allow-list.

  • mocha tests/common/Telemetry.test.ts: 9 passing
  • npm run build and npm run lint pass

Not changed (follow-ups)

  • source: still the raw user-agent header, which can include an app-provided prefix. This behaviour predates 3.x.
  • url and responseCode: still empty or 0 on request records, as they have been since 3.x. The 3.x exporter only copies them for spans with an HTTP method attribute, and trackRequest only sets one when no name is given. Azurite always sets a name, so this needs a separate change.

After merge

Ingestion should drop once users update to the next release, not before. After that, check that cloud_RoleName == "Azurite_V1.0" and application_Version are set, and that daily volume is back under the cap.

applicationinsights 3.x is an OpenTelemetry shim that silently ignores
several settings Azurite relied on. Since 3.37.0, telemetry reported
unknown_service:node with no version, sent close to every request instead
of 1% (a default 5 traces/second limiter replaced the ratio), sent the raw
machine name plus host and OS details, and installed global OpenTelemetry
providers in the Azurite and VS Code extension host process.

- Use isolated TelemetryClient providers instead of setup()/start().
- Set service.name (Azurite_V1.0), service.version and a hashed
  service.instance.id on the resource.
- Disable the per-batch _OTELRESOURCE_ metric.
- Track requests from the root context so other OpenTelemetry
  instrumentation can't bypass sampling, and set microsoft.sample_rate
  so itemCount keeps estimating total requests.
- Flush when the stop event is sent.
- Only report known authorization scheme names.
- Test through the real SDK with a local ingestion endpoint.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 9, 2026 09:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Shutdown paths do not await the asynchronous telemetry flush, so stop-event delivery remains unguaranteed.

2 open findings
What changed in this PR

Fixes Application Insights 3.x telemetry regressions by isolating OpenTelemetry providers, restoring metadata and sampling, and tightening authorization reporting.

Changes:

  • Restores role, version, hashed instance, and 1% request sampling.
  • Prevents global instrumentation and resource-metric leakage.
  • Adds SDK-level telemetry and authorization tests.
  1. Impact analysis

    • Package: applicationinsights, @opentelemetry/api, and @opentelemetry/resources
    • Version change: OpenTelemetry packages promoted to direct dependencies.
    • Usage: Shared Blob, Queue, Table, CLI, and VS Code telemetry.
    • Impact and risk: Improves telemetry privacy and accuracy; shutdown flushing remains unreliable because its promise is not awaited.
  2. Build and conflict resolution

    • Commands run: None locally; PR reports npm run build and npm run lint.
    • Issues found: No dependency or lockfile conflicts identified.
    • Result: Reported build and lint pass.
  3. Tests

    • Tests added or updated: Real SDK ingestion, sampling isolation, resource metadata, and authorization allow-list tests.
    • Coverage: Core regressions are covered; stop-event shutdown flushing is not exercised.
    • Result: One test incorrectly assumes the resource-metric environment variable was initially unset.
  4. Test suite

    • Commands run: None locally; PR reports mocha tests/common/Telemetry.test.ts.
    • Passing: 9 reported.
    • Pending or failing: None reported.
  5. Changelog

    • Entry added: Yes.
    • Location: ChangeLog.md, Upcoming Release → General.
  6. PR comments

    • Threads reviewed: No prior review threads provided.
    • Actions taken: Added two review comments.
    • Unresolved items: Await telemetry flushing during shutdown and make environment restoration testing hermetic.
  7. Self-rating

    • Score: 8/10
    • Reasoning: Reviewed every changed file and validated Application Insights 3.16 behavior, but did not execute the suite locally.
  8. Iteration

    • Improvements made after self-review: Traced all shutdown callers and confirmed from upstream SDK source that flush() returns an asynchronous promise.
  9. Summary

    • Why the update matters: Restores expected telemetry volume, identity, privacy, and process isolation.
    • Risk if not applied: Excess telemetry and unintended host metadata continue to be emitted.
    • Final status: Changes required for reliable shutdown flushing and robust testing.
File Description
src/​common/​Telemetry.ts Implements isolated clients, metadata, sampling, flushing, and authorization filtering.
tests/​common/​Telemetry.test.ts Adds real-SDK regression tests.
package.json Adds direct OpenTelemetry dependencies.
package-lock.json Records direct dependency declarations.
ChangeLog.md Documents user-visible telemetry fixes.

🧠 Review effort: Balanced


💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/common/Telemetry.ts
Comment thread tests/common/Telemetry.test.ts
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 12, 2026 09:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Shutdown does not await asynchronous flushing, and the production sample-rate path lacks direct verification.

2 open findings
Previously missed (1)

In code that hasn't changed since last review

Medium severity Assert TraceRequest sample rate and internal property cleanup

tests/​common/​Telemetry.test.ts:182

This SDK-level test manually supplies microsoft.sample_rate, so it still passes if TraceRequest stops adding the attribute or adds the wrong value. The sampling test below calls TraceRequest but only checks the number of envelopes, not their sampleRate. Route a production TraceRequest through the capture client and assert its exported envelope has sampleRate === 1 and no leaked internal property.

🧠 Review effort: Balanced

Node already strips surrounding whitespace from header values.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Preserve a pre-existing resource metric env var value in the test
  instead of assuming it starts unset.
- Verify TraceRequest itself sets the 1% sample rate and doesn't leak it
  as a custom property.
- Verify TraceStopEvent flushes both clients.
- Describe stop event delivery accurately in the changelog.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 12, 2026 09:43
@jainakanksha-msft

Copy link
Copy Markdown
Member Author

Re the Copilot finding "Assert TraceRequest sample rate and internal property cleanup" (tests/common/Telemetry.test.ts): fixed in 013b965. A new test sends a request through TraceRequest to the local ingestion server and checks the exported envelope has sampleRate === 1, with no microsoft.sample_rate or _MS.* custom property.

I also checked that each new test catches the bug it's meant to: with the wrong sample-rate attribute, the flush removed from TraceStopEvent, or the env var not restored, the matching test fails.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The implementation matches the stated telemetry corrections and has focused integration coverage with no unresolved defects.

0 open findings

2 resolved since last review

🧠 Review effort: Balanced

@jainakanksha-msft
Akanksha Jain (jainakanksha-msft) merged commit aa49d15 into main Oct 12, 2026
46 checks passed
@jainakanksha-msft
Akanksha Jain (jainakanksha-msft) deleted the fix/telemetry-otel-regression branch October 12, 2026 10:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants