feat: Add TrackDurationOf, TrackMetricsOf, TrackJudgeResult, TrackToolCall

[mattrmc1]

Summary

Adds the compound tracking methods and new event types to ILdAiConfigTracker. Callers can now measure operation duration via a callable wrapper, extract metrics from an operation result in one call, record judge evaluation outcomes, and track tool invocations. The legacy TrackDurationOfTask and TrackRequest methods are preserved but marked [Obsolete].

Six new methods on ILdAiConfigTracker (and implementations on LdAiConfigTracker):

TrackDurationOf<T>

public Task<T> TrackDurationOf<T>(Func<Task<T>> operation);

Accepts a callable (not a pre-started task) so the tracker controls when execution begins — duration measurement starts at invocation, not at some earlier Task.Run call site. Uses Stopwatch for wall-clock precision. Duration is recorded even if the operation throws (via finally). Emits $ld:ai:duration:total.

Replaces TrackDurationOfTask<T>(Task<T>), which is now [Obsolete("Use TrackDurationOf instead.")].

TrackMetricsOf<T>

public Task<T> TrackMetricsOf<T>(Func<T, AiMetrics> metricsExtractor, Func<Task<T>> operation);

All-in-one wrapper that tracks duration, success/error, and optional token usage from a single operation. Flow:

1. Starts a stopwatch, invokes operation.
2. On success: stops timer → TrackDuration → calls metricsExtractor(result) → TrackSuccess/TrackError based on AiMetrics.Success → TrackTokens if AiMetrics.Tokens is non-null.
3. On exception: stops timer → TrackDuration → TrackError → re-throws.

Replaces TrackRequest(Task<Response>), which is now [Obsolete("Use TrackMetricsOf instead.")].

TrackJudgeResult

public void TrackJudgeResult(JudgeResult result);

Records a judge evaluation outcome. The event is silently dropped when result.Sampled == false or result.Success == false — this prevents noisy/invalid scores from polluting metrics.

When emitted, the event uses result.MetricKey as the track event name and result.Score as the metric value. If result.JudgeConfigKey is non-null, it's merged into the track data alongside the standard runId/configKey/variationKey/version fields.

TrackToolCall

public void TrackToolCall(string toolKey);

Emits a $ld:ai:tool_call event with toolKey merged into the track data. Unlike most tracker methods, this is not at-most-once — it may be called multiple times to record every tool invocation in a run (each emits a separate event with metric value 1).

TrackToolCalls

public void TrackToolCalls(IEnumerable<string> toolKeys);

Convenience batch method — iterates toolKeys and calls TrackToolCall for each.

New types

AiMetrics — immutable record holding Success (bool) and optional Tokens (Usage?) for use with TrackMetricsOf.

JudgeResult — immutable record holding MetricKey, Score, Sampled, Success, and optional JudgeConfigKey for use with TrackJudgeResult.

AgentConfigs event fix

AgentConfigs now fires only the aggregate $ld:ai:usage:agent-configs event. It calls the private BuildAgentConfig path internally — it does NOT call the public AgentConfig() method and does NOT fire individual $ld:ai:usage:agent-config events. Tests updated to assert Times.Never on individual events.

Test plan

• dotnet build succeeds across netstandard2.0, net462, net8.0
• dotnet test --framework net8.0 passes
• LdAiConfigTrackerTest covers the new tracker surface:
  • TrackDurationOf_MeasuresDuration — verifies wall-clock measurement via a 50ms delay
  • TrackMetricsOf_SuccessPath_TracksAllMetrics — verifies duration + success + tokens all emitted
  • TrackMetricsOf_ErrorPath_TracksErrorAndRethrows — verifies duration + error emitted, exception propagated
  • TrackJudgeResult_SampledFalse_NoEventEmitted — verifies silent drop
  • TrackJudgeResult_SuccessFalse_NoEventEmitted — verifies silent drop
  • TrackJudgeResult_SuccessPath_EmitsCorrectEvent — verifies metric key, score, judgeConfigKey in data
  • TrackToolCall_DataIncludesToolKey — verifies $ld:ai:tool_call event with toolKey in data
  • TrackToolCall_NoAtMostOnce_EmitsMultipleEvents — verifies repeated calls emit separate events
  • DeprecatedShims_StillCallable — verifies [Obsolete] methods remain functional
• LdAiClientAgentJudgeTest.AgentConfigs_FiresOnlyAggregateEvent — verifies NO individual $ld:ai:usage:agent-config events
• LdAiClientTest.AgentConfigs_OnlyBatchEventFired — same assertion from the client-level test
• Reviewer confirms method signatures, event names, and at-most-once semantics match the cross-SDK contract (AITRACK §1.1.4, §1.1.12, §1.1.13, §1.1.15)

---

Note

Medium Risk
Changes public tracker contracts and telemetry event shapes; behavior is well-tested but callers migrating off obsolete APIs need to adopt the new wrappers.

Overview
Extends ILdAiConfigTracker with compound wrappers and new event types for the server AI SDK.

TrackDurationOf times a Func<Task<T>> (so measurement starts at invocation), records $ld:ai:duration:total even on failure, and supersedes TrackDurationOfTask, which stays but is [Obsolete]. TrackMetricsOf runs an operation, records duration, then applies success/error and optional tokens via a new AiMetrics extractor; TrackRequest is obsolete in favor of this pattern.

TrackJudgeResult emits judge scores under a caller metric key when sampled and successful, optionally merging judgeConfigKey into track data. TrackToolCall / TrackToolCalls emit $ld:ai:tool_call with toolKey and are not at-most-once. Supporting types AiMetrics and JudgeResult are added; MergeTrackData enriches events for judge and tool tracking.

Tests document AgentConfigs aggregate-only usage events (no per-key $ld:ai:usage:agent-config) and cover the new tracker APIs plus deprecated shims.

^{Reviewed by Cursor Bugbot for commit 1e44b8d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[jsonbailey]

You should wrap this in its own try catch and log a warning if we fail to extract metrics.

[mattrmc1]

Thinking about this more...

The metricsExtractor is a delegate that comes from the user if I'm not mistaken. I think it's better to let the exception happen if it's due to the metricsExtractor failing. I can still log a warning and re-throw but that seems like noise since the exception carries a stacktrace to the user and the error will hit our logs anyway.

My vote: keep it the way it is