`dataform compile` ~4x slower since @dataform/core 3.0.61 (CLI only) — encoded graph now crosses the vm2 bridge as a host Buffer

[ashish10alex]

Summary

dataform compile became roughly 4x slower in @dataform/core 3.0.61 and has stayed slow through 3.0.69 (current latest). This affects the CLI only — the hosted Dataform service shows no regression on the same project.

Root cause: after the protobufjs 7.5.8 -> 7.6.3 bump in 3.0.61, Writer.finish() returns a host Buffer rather than a Uint8Array, and the CLI runs Dataform Core inside a vm2 sandbox, so the entire encoded graph now has to cross the sandbox/host boundary. Details below. The added cost scales with the volume of column descriptors in the project, so documentation-heavy projects are hit hardest. A real project of ~400 actions with ~39k documented columns went from ~3s to ~10s per compile.

Bisected with the CLI pinned at 3.0.69 throughout, varying only @dataform/core:

@dataform/core	compile
3.0.50	3.0s
3.0.55	3.1s
3.0.60	3.2s
3.0.61	10.3s
3.0.65	10.1s
3.0.69	10.2s

Root cause

protobufjs picks its output representation at runtime: Writer.finish() returns a Node Buffer if one can be resolved, and a plain Uint8Array otherwise. The CLI runs Dataform Core inside a vm2 sandbox, so which one comes back decides whether the encoded graph crosses the sandbox/host boundary.

• 3.0.60 — Buffer does not resolve inside the sandbox. The compiled graph is encoded into a Uint8Array and never crosses the bridge.
• 3.0.61+ — with protobufjs 7.6.3 it resolves. finish() allocates one host Buffer (3.6 MB for the project below), which crosses the vm2 boundary, where vm2 walks it with containsDangerousConstructor and thisSafeGetOwnPropertyDescriptor.

I profiled this by running compilation in-process through vm2 using the pattern in testing/run_core.ts, which reproduces the regression outside the CLI's forked child (2.16s vs 5.86s):

	total	vm2 bridge	GC	host Buffer allocations
core 3.0.60	2.41s	0.03s	0.04s	0
core 3.0.69	5.72s	2.33s	0.46s	1 x 3.6 MB

The bridge accounts for roughly 83% of the regression. Call path of the heaviest frame, leaf first:

containsDangerousConstructor      bridge.js:2208          1135 ms
thisFromOtherWithFactory          bridge.js:3851
thisFromOther                     bridge.js:3565
apply                             bridge.js:2848
alloc / allocUnsafe               setup-sandbox.js:934 / :948
Writer.finish                     @dataform/core bundle.js
main                              @dataform/core bundle.js

Instrumenting the host Buffer.allocUnsafe confirms the count: 0 calls on 3.0.60, exactly 1 call of 3.6 MB on 3.0.69. It is a single large object, not many small ones — the cost is vm2 inspecting it as it crosses.

This accounts for every symptom: the cost scales with the size of the encoded graph (hence with column descriptor volume and with description length), peak memory roughly triples, Node version is irrelevant, --json is irrelevant, and the hosted service is unaffected because it does not run core through this sandbox path.

The core-side changes in 3.0.61 (getContents(), .md handling in core/compilers.ts) are dormant unless a project uses them and are not involved.

Fix direction: keep the encoded graph off the vm2 boundary as a host Buffer. Returning a Uint8Array, or converting before it crosses, should restore 3.0.60 timings while keeping the dependency bump.

Reproduction

The trigger is the volume of column descriptors. The script below builds a throwaway project with N tables x M documented columns — no markdown, no getContents(), no JS, nothing but config { columns: ... } blocks — and times both versions with peak RSS:

#!/usr/bin/env bash
set -euo pipefail

DIR=$(mktemp -d)
TABLES=${TABLES:-90}
COLS=${COLS:-435}

mkdir -p "$DIR/definitions"
cat > "$DIR/workflow_settings.yaml" <<'YAML'
defaultProject: test-project
defaultLocation: EU
defaultDataset: test
defaultAssertionDataset: test_assertions
YAML

node -e "
const fs = require('fs');
const dir = process.argv[1], nt = +process.argv[2], nc = +process.argv[3];
for (let t = 1; t <= nt; t++) {
  const cols = {};
  for (let c = 1; c <= nc; c++) {
    cols['COL_' + c] = 'Description for column ' + c + ' with some reasonable length of documentation text';
  }
  const select = Object.keys(cols).map(k => '1 as ' + k).join(', ');
  fs.writeFileSync(
    dir + '/definitions/t' + t + '.sqlx',
    'config { type: \"table\", columns: ' + JSON.stringify(cols) + ' }\nselect ' + select + '\n'
  );
}
" "$DIR" "$TABLES" "$COLS"

echo "project: $DIR  ($TABLES tables x $COLS documented columns = $((TABLES * COLS)))"
for V in 3.0.60 3.0.61; do
  echo "{\"dependencies\":{\"@dataform/core\":\"$V\"}}" > "$DIR/package.json"
  (cd "$DIR" && npm install --silent >/dev/null 2>&1)
  echo "--- @dataform/core $V ---"
  /usr/bin/time -l dataform compile "$DIR" --json --timeout=5m >/dev/null 2>"$DIR/t.txt" || true
  grep -E "real|maximum resident set size" "$DIR/t.txt" || true
done

Result at the default 90 x 435 (39,150 documented columns):

@dataform/core	compile	peak RSS
3.0.60	2.2s	376 MB
3.0.61	8.7s	1074 MB

Both symptoms reproduce from the same input, which is why I suspect this and #2219 share a root cause.

Scaling

Holding the shape fixed and varying the descriptor volume (90 tables unless noted):

documented columns	3.0.60	3.0.61
9,810	1.62s	3.06s
19,620	1.69s	4.91s
39,240 (180 tables)	2.68s	9.49s
39,150, descriptions shortened to 1 character	2.00s	3.97s

The added cost is roughly linear in total column-descriptor volume (~0.17 ms per documented column at ~75-character descriptions), and shrinks with description length, so it tracks serialized size rather than field count alone.

Control: 200 tables with no column documentation compile in 1.32s on 3.0.60 and 1.19s on 3.0.61 — unaffected. The regression only shows up once actions carry column descriptors.

Scope: CLI only, hosted service unaffected

Compiling the same project through the Dataform API with CreateCompilationResult shows no regression. Both arms are the same repository with identical content apart from the pinned core version, and both produced 400 compilation actions:

	core version	compile (5 runs)	median
Hosted API, branch	3.0.50	3.76 / 3.73 / 5.37 / 3.54 / 3.51	3.73s
Hosted API, workspace	3.0.69	1.94 / 2.81 / 1.54 / 3.05 / 2.82	2.82s
Local CLI	3.0.60	—	3.2s
Local CLI	3.0.69	—	9.6s

CreateCompilationResult is synchronous — querying the result immediately after a 2.98s POST returns all 400 actions — so these are compile times rather than queue times. They include the API round trip, so the hosted figures are upper bounds.

Two candidate explanations for the local slowdown that I ruled out:

• Node version. core 3.0.69 on the same project: Node 20.19.0 -> 7.8s, 22.13.0 -> 7.3s, 23.6.1 -> 9.5s, 24.14.0 -> 9.6s. Slow on all of them.
• Output serialization. dataform compile --json 9.58s vs dataform compile (no --json) 10.43s. Not the JSON encoding of the graph.

So whatever changed in 3.0.61 costs time in the way the CLI runs compilation locally, while the hosted path is unaffected. That may be why this has not shown up in your own monitoring.

I can only observe the hosted service from outside, so I cannot confirm how it executes core. Given the root cause above, the natural explanation is that it does not run core inside a vm2 sandbox the way the CLI does, so there is no boundary for the encoded graph to cross.

Relationship to #2219

#2219 reports a memory increase at the same version boundary, and the repro above shows both the slowdown and a ~3x peak RSS increase from one input, so they are probably the same underlying change.

That issue attributes the cause to #2164 (getContents() reading markdown descriptions). I don't think that holds: the repro here never calls getContents() and contains no markdown files at all, yet reproduces both symptoms. That seems consistent with @kolina's own suggestion on that thread that it is the dependency upgrades in 3.0.61.

The root cause section above supersedes this: the shared mechanism is the encoded graph crossing the vm2 bridge as a host Buffer, which explains the memory increase as well as the slowdown.

Environment

• macOS, Node v24.14.0
• @dataform/cli 3.0.69 for every run; only @dataform/core varied
• 3.0.69 is the latest published version, so there is currently no release with the fix

Worth noting for anyone else hitting this: propertyGraphs support requires >= 3.0.65, so pinning back to 3.0.60 to recover compile speed also means giving up property graphs.

[ihistand]

Another data point from a real project with a different descriptor profile. Both symptoms reproduce; the memory increase matches yours almost exactly, the time increase is smaller because our documented-column volume is lower.

Hardware: MacBook Air M4, 16 GB RAM, macOS, Node v26.8.2 — a laptop, not a VM or CI runner.

Method: scratch copy of our production Dataform repo, @dataform/cli@3.0.69 via npx for every run, only dataformCoreVersion in workflow_settings.yaml varied. One warm-up per version (core install), then 3 timed runs with /usr/bin/time -l. Times include npx startup.

Project scale (from compile --json at 3.0.69):

metric	value
actions	1,618
actions with column docs	772
documented columns	13,258
avg description length	32 chars
compiled JSON	5.7 MB

Results:

@dataform/core	compile (3 runs)	peak RSS
3.0.69	11.43 / 11.23 / 11.59 s	977 / 1,090 / 1,100 MB
3.0.60	8.40 / 8.93 / 10.72 s	338 / 320 / 299 MB

Peak RSS is ~3.3x, in line with your 376 → 1,074 MB. Compile time is roughly +2.5 s (+25–30%) rather than 4x, which is consistent with your scaling estimate: we have about a third of your documented columns at under half the description length, so the added bridge cost is small relative to our ~8.5 s baseline (1,618 actions plus includes/*.js).

Our production runs are on the hosted service and show no change, so for us this is limited to local CLI compiles (pre-commit hook, dry runs).

[kolina]

Thank you so much for the investigation and fix!

You're correct: the reason it didn't affect GCP Dataform is that we don't use vm2 there. Also we're working on removing vm2 dependency for CLI as well

[ashish10alex]

Thanks @kolina , I tested the migration from vm2 PR and Im seeing ~2x compilation time performance improvement for my project. Excited for that release !