FBDetect Pipeline (Reconstruction)

A containerized, CLI-driven reconstruction of Meta's FBDetect regression-detection pipeline, as described in "FBDetect: Catching Tiny Performance Regressions at Hyperscale through In-Production Monitoring" (SOSP '24, Best Paper Award).

Honesty note: this is a teaching-scale reconstruction of the algorithm's shape (same 7 stages, same ordering, several details pulled directly from the paper), not a reimplementation of FBDetect itself — the real system is inseparable from Meta's fleet-wide stack-trace/gCPU infrastructure, which this project has no access to.

• New to this? Start with docs/EXPLAINED.md — a plain-English, no-algorithms walkthrough of why each stage exists.
• Want the technical detail? See docs/ARCHITECTURE.md for the full stage-by-stage fidelity comparison against the paper.

The 7 stages

1. Change Point Detector (CUSUM + EM)
2. Went-away Detector (now with real SAX-based NewPattern detection, via pyts)
3. Seasonality Detector (STL, gated on autocorrelation)
4. SOMDedup (fast first-pass clustering, features enriched with tsfresh variance/FFT)
5. Cost-shift Detector
6. PairwiseDedup (correlation OR name-similarity merge)
7. Root Cause Analysis (ranks changelog candidates by time proximity + text similarity)

Requirements

• Docker (tested against Colima on macOS)

Everything else (numpy, pandas, scipy, scikit-learn, statsmodels, pymannkendall, minisom, matplotlib, pyts, tsfresh) is declared in pyproject.toml and installed inside the image as the fbdetect package — see Dockerfile.

Build

docker build -t fbdetect:latest .

Usage

The CLI (src/fbdetect/cli.py, installed as the fbdetect command) supports three modes via --mode:

stages1-3 — real single-metric data

Runs the change-point / went-away / seasonality detectors against a real CSV (timestamp,value columns).

docker run --rm \
  -v "$(pwd)/data:/data:ro" \
  fbdetect:latest --mode stages1-3 --input /data/ec2_cpu.csv

Optional: validate the detection against NAB's own ground truth

NAB ships hand-labeled real-anomaly windows for every benchmark file (labels/combined_windows.json in the NAB repo), independent of anything this pipeline computes. Download that file into data/ and pass it with --nab-labels to check whether the detected change point falls inside Numenta's labeled window for ec2_cpu_utilization_5f5533.csv:

curl -o data/nab_labels.json \
  https://raw.githubusercontent.com/numenta/NAB/master/labels/combined_windows.json

docker run --rm \
  -v "$(pwd)/data:/data:ro" \
  fbdetect:latest --mode stages1-3 --input /data/ec2_cpu.csv --nab-labels /data/nab_labels.json

The pipeline reports change_point_in_labeled_window: true/false plus the matched window, if any -- an external corroboration signal distinct from the pipeline's own internal verdict (went-away/seasonality filters). It's the closest analog stages1-3 has to stage 7's changelog correlation in full-real mode.

full-synthetic — all 7 stages, synthetic multi-metric scenario

Stages 4-7 need multiple metrics sharing a root cause plus a changelog — neither exists in a single real metric, so this mode uses a synthetic scenario built for exactly that purpose (src/fbdetect/scenarios/synthetic_demo.py).

docker run --rm fbdetect:latest --mode full-synthetic

full-real — all 7 stages, real multi-service telemetry

Runs against real Prometheus-scraped microservice telemetry from an actual injected fault (BARO/RCAEval sample — see Data below).

docker run --rm \
  -v "$(pwd)/data:/data:ro" \
  fbdetect:latest --mode full-real --input /data/rca_simple_data.csv

Visualization

Add --output-dir /out (with a mounted volume) to any mode to get per-metric change-point plots, a Stage 6 cluster scatter + group overlays, a Stage 7 root-cause score chart, and a report.html that ties it all together:

docker run --rm \
  -v "$(pwd)/data:/data:ro" \
  -v "$(pwd)/output/full-real:/out" \
  fbdetect:latest --mode full-real --input /data/rca_simple_data.csv --output-dir /out

then open output/full-real/report.html.

Data sources

Both datasets are real (not synthetic) and pulled from public, open-source benchmarks — no data in data/ was generated by this project.

File	Source project	Direct source	License	What it is
data/ec2_cpu.csv	NAB — Numenta Anomaly Benchmark	data/realAWSCloudwatch/ec2_cpu_utilization_5f5533.csv	AGPL-3.0	Real AWS CloudWatch CPU utilization for one EC2 instance, 5-min interval, 4032 samples (~14 days)
data/rca_simple_data.csv	BARO / RCAEval (FSE '24)	baro GitHub release 0.0.4/simple_data.csv	MIT (BARO)	Real Prometheus-scraped per-service telemetry (CPU/memory/workload/error/latency-p50/p90, 721 samples × 57 metrics, 1-sec interval) captured from an actual running deployment of Google's Online Boutique microservice demo during a real injected resource-stress fault

Both were fetched directly from the links above with curl — see docs/ARCHITECTURE.md for the real service topology the second dataset came from (with the pipeline's Stage 7 finding overlaid on it), and docs/ARCHITECTURE.md for how these compare to the data Meta's actual FBDetect runs on in production.

Project layout

pyproject.toml                   package metadata, dependencies, console-script entry point
Dockerfile                       pip installs the fbdetect package into the image
src/fbdetect/
  cli.py                         entrypoint (installed as the `fbdetect` command); --mode dispatch
  fast_cusum_em.py               Stage 1: CUSUM+EM change point detector
  sax_features.py                Stage 2: SAX-based NewPattern / SignificantRegression
  pipeline.py                    Stages 1-3 orchestration (single metric)
  full_pipeline.py               Stages 4-7 orchestration (multi-metric)
  visualize.py                   PNG plots + HTML report generation
  scenarios/
    synthetic_demo.py            synthetic multi-metric scenario (full-synthetic mode)
    real_rca.py                  real BARO/RCAEval scenario (full-real mode)
data/                            real datasets (see table above)
docs/
  EXPLAINED.md                            plain-English walkthrough of why each stage exists
  ARCHITECTURE.md                         pipeline architecture + paper fidelity comparison
  service_architecture_healthy.png        source system topology, healthy baseline
  service_architecture_incident.png       source system topology, with pipeline's finding overlaid
output/                          example run artifacts (regenerable via --output-dir)

Inside the container, docker run ... fbdetect:latest --mode ... invokes the installed fbdetect console script (fbdetect.cli:main) — not a bare python cli.py — same as running fbdetect --mode ... after a local pip install ..