Builder task-time estimator

A terminal-first TypeScript project that refreshes OpenRouter performance data and runs two independent estimates concurrently:

1. a pure-AI judgment of autonomous Builder turns, output tokens per turn, and automated tool/test time
2. a log-trained estimate of Builder turns, output tokens per turn, and calibrated task lifecycle time

It also runs a deterministic, blind A/B-style backtest against the completed timings in the log.

Quick start

npm install
$env:OPENROUTER_API_KEY = "your-key"
npm run profile
npm run estimate -- --task "Add an admin audit-log export with permissions and end-to-end tests"
npm run backtest

OPENROUTER_API_KEY is required for the pure-AI pass. The statistical pass still completes if the AI request fails, but estimate exits with code 2 so a missing pass is visible to automation.

Use a longer task specification:

npm run estimate -- --task-file ./task.md

Machine-readable output and timing overrides are available:

npm run estimate -- --task "Fix the mobile quote flow" --json
npm run estimate -- --task "Fix the mobile quote flow" --throughput 55 --latency 1.2
npm run estimate -- --task "Fix the mobile quote flow" --routing nitro
npm run estimate -- --task "Fix the mobile quote flow" --ai-model minimax/minimax-m3

Build the distributable CLI with npm run build, then run:

node dist/cli.js estimate --task "Add inbound email processing"

Two parallel estimation passes

After performance data is ready, estimate starts both passes before awaiting either one. They do not consume one another's result.

Pass 1: pure AI autonomous-work estimate

The CLI calls OpenRouter Chat Completions with a strict JSON schema. The estimator sees only the task specification and current timing—it does not see task-build-log.json, historical durations, or the statistical prediction.

Its prompt explicitly estimates an autonomous AI Builder rather than a human team. It returns:

• Builder-model turns, including expected self-correction and repair turns
• output tokens per Builder turn
• elapsed autonomous tool time for file operations, builds, tests, browsers, and shell commands
• uncertainty bounds, assumptions, rationale, and confidence

The AI pass uses:

AI model seconds = AI turns × live latency
                 + AI turns × AI output tokens/turn ÷ live throughput

AI autonomous completion seconds = AI model seconds
                                 + AI-estimated automated tool seconds

It excludes human approvals, person-hours, queue pauses, scheduling delays, and provider outages. The task specification is sent to OpenRouter, so do not submit secrets.

Pass 2: log-trained statistical estimate

The parser attributes agent-turn-complete events to the most recent actor-start. Training therefore uses only turns and token counts generated by the Builder actor whose normalized model is minimax/minimax-m3; Reviewer turns from other models are not incorrectly charged to MiniMax.

The feature estimator fits two small ridge regressions on the task title and available task-body text:

1. log(Builder turns)
2. log(output tokens / Builder turn)

The ridge penalty is selected using leave-one-out validation on the training partition only. Predictions are bounded by robust training quantiles because 25 eligible records are not enough to justify unconstrained extrapolation.

For a timing profile with latency L seconds and output throughput R tokens/second:

raw model seconds = predicted turns × L
                  + predicted turns × predicted output tokens/turn ÷ R

active completion seconds = raw model seconds × lifecycle calibration

The lifecycle multiplier is the training-set median of actual active completion time divided by the same equation using observed Builder turns and output tokens. It absorbs the historical review, tool, test, and orchestration lifecycle. It is not a claim that input-token prefill is free; OpenRouter exposes time-to-first-token and output throughput publicly, while a separate public input-prefill rate is not available in the cited snapshot.

Live OpenRouter performance and five-minute cache

Before every non-help command, the CLI checks a disk cache and refreshes expired data from:

GET https://openrouter.ai/api/v1/models/{author}/{slug}/endpoints

The cache lifetime is exactly five minutes. Its default location is .cache/openrouter/; set TASK_TIME_CACHE_DIR or pass --performance-cache to change it.

• balanced uses the median p50 throughput and latency across healthy endpoints with both measurements.
• nitro uses the healthy endpoint with the highest p50 throughput and its corresponding p50 latency.
• CLI --throughput and --latency values override the selected live values.
• If refresh fails, the last stale cache is used with a warning.
• If the API has no complete latency/throughput pair, the CLI visibly falls back to the bundled snapshot.

The endpoint response sometimes contains provider/uptime records while latency_last_30m and throughput_last_30m are null. That is treated as missing performance evidence, not as zero latency or throughput.

The live API currently returns latency samples such as 681 for 0.681 seconds even though its published schema example uses fractional seconds. The normalizer detects the millisecond representation from the full percentile set and converts it to seconds; a regression test covers this wire shape.

OpenRouter documents the model endpoint API, structured outputs, and provider performance routing.

Bundled fallback snapshot

The defaults in data/openrouter-minimax-m3.json were captured on 2026-07-22 from the official MiniMax M3 model page:

• balanced: 42 output tokens/s and 1.99 s latency, using the recent MiniMax-provider averages as the central reference
• nitro: 51 output tokens/s and 0.60 s latency, using the best-provider headline figures

OpenRouter states that latency/throughput statistics vary by provider and are tracked over rolling windows. These values are a last-resort fallback, not the normal source when live p50 endpoint metrics are present.

Blind A/B-style backtest

npm run backtest makes one out-of-sample prediction for every eligible MiniMax task using grouped cross-validation:

• A: training-partition median turns and median output tokens/turn
• B: task-feature ridge predictions
• completed duration is hidden until that task is scored
• identical normalized titles stay in the same fold, preventing duplicate phase names such as “Implement application logic” from leaking across train/test
• both arms use the same OpenRouter timing profile and lifecycle-calibration method

The command reports MAE, median absolute error, RMSE, MAPE, percentage within 25%, and every task-level prediction. This is an observational model comparison, not a randomized causal experiment. If A wins, the correct conclusion is that the text features did not generalize on this small sample—not that the backtest failed.

Using the frozen fallback values (--throughput 42 --latency 1.99) with --seed blind-v1 --folds 5, A has 18m 53s MAE and 38.6% MAPE; B has 18m 24s MAE and 37.5% MAPE. B improves MAE by only 2.6%, raises within-25% coverage from 36% to 48%, and has slightly worse RMSE (23m 57s versus 23m 06s). Live-default results can move with provider performance. Treat the task-aware result as exploratory: changing timing or --seed can change the winner because the sample is small.

Data-quality decisions

The source currently has 28 valid task records. The default MiniMax analysis uses 25 and excludes two GPT-5.6 Luna Builder records plus one GPT-5.4 Mini Builder record.

The normalizer also handles these known issues:

• repeat Builder/reviewer repair cycles and multiple task-start events
• explicit “stopped by request” intervals and completed lifecycles later reopened; those gaps remain in wallSeconds but are removed from the primary activeSeconds target
• missing task bodies, which fall back to title-only features
• completionObservedAt values that precede completedAt; the inconsistent observation field is profiled but not used as the completion target

Both estimates are active autonomous runtime. They intentionally exclude future queue pauses, human stops, and provider outages. The statistical pass learns historical repair-cycle effects; the pure-AI pass judges likely repair work from the task text without historical data.

Verification

npm run check
npm run build
npm run profile
npm run backtest

The tests include parser assertions against the supplied log, actor/model attribution, external-pause removal, grouped-fold leakage prevention, and end-to-end estimator/backtest checks.

They also verify the exact five-minute cache boundary, refresh behavior, live p50 timing selection, strict AI-response parsing and arithmetic, missing-key behavior, and that both estimate passes start before either finishes.