Cursor/bigquery universe domain clean 3190 by ybelleguic · Pull Request #2323 · dataform-co/dataform

ybelleguic · 2026-09-18T12:59:52Z

What

Adds support for configuring the universeDomain used by the BigQuery and
data-lineage clients, so the Dataform CLI can connect to a non-default universe
(for example a Trusted Partner Cloud / TPC deployment) instead of the Google
Default Universe (googleapis.com).

Why

@google-cloud/bigquery already accepts universeDomain on its constructor
options, but there was no way to set it from Dataform: the BigQuery client is
built in createBigQueryClientProvider() (cli/api/dbadapters/bigquery.ts)
purely from the credentials, and the credentials shape is defined by the
BigQuery proto. Since .df-credentials.json is strictly validated against
that proto (verifyObjectMatchesProto rejects unknown fields), users targeting
a non-default universe had no supported way to specify it.

Changes

protos/profiles.proto: add an optional string universe_domain = 5; field
to the BigQuery credentials message.
cli/api/dbadapters/bigquery.ts: forward universeDomain to
new BigQuery({ ... }) when set.
cli/api/lineage/emitter.ts: forward universeDomain to
new LineageClient({ ... }) when set (google-gax ClientOptions supports it),
so lineage emission targets the same universe.
cli/credentials.ts: prompt for an optional universe domain during
dataform init-creds (leaving it blank keeps the default).
contributing.md: document the new optional field in the credentials schema.
Tests: cli/api/commands/credentials_test.ts (new) covers credential parsing
including rejection of unknown fields; cli/api/dbadapters/bigquery_test.ts
verifies the value reaches the client and defaults to googleapis.com when
unset.

The field is optional and empty by default, so existing credentials files and
callers are completely unaffected.

Testing

bazel test //cli/api:commands/credentials_test //cli/api:dbadapters/bigquery_test — pass.
./scripts/run_tests (eslint + full Bazel suite) — pass.
Manual: ./scripts/run run <project> --dry-run with a .df-credentials.json
containing universeDomain is accepted and the BigQuery request targets the
derived endpoint (https://bigquery.<universeDomain>/...).

google-cla · 2026-09-18T13:00:15Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Add an optional universe_domain field to the BigQuery credentials proto and plumb it through to the BigQuery and data-lineage clients so the CLI can target non-default universes (e.g. Trusted Partner Cloud). The value is read from .df-credentials.json and can be entered during 'dataform init-creds'. When unset, clients keep defaulting to the Google Default Universe. Adds unit tests covering credential parsing and client construction.

apilaskowski · 2026-09-18T20:34:41Z

          apiEndpoint: endpoint,
          credentials: credentials.credentials && JSON.parse(credentials.credentials),
          libName: DATAFORM_CLI_LIB_NAME,
-          libVersion: version
+          libVersion: version,
+          ...(credentials.universeDomain ? { universeDomain: credentials.universeDomain } : {})


This doesn't take effect: LineageClient resolves servicePath = opts.servicePath || opts.apiEndpoint || 'datalineage.' + universeDomain, and we always pass apiEndpoint from LineageEndpointRouter, which hardcodes googleapis.com. So lineage keeps targeting GDU hosts.

It's also a bit worse than a no-op: gax compares the configured universe against the credential's before each call, so setting it here makes that check pass while we carry on dialing a GDU endpoint — a loud failure becomes a silent one.

Could the endpoint router become universe-aware, or this hunk drop out for now? Either way it needs coverage; emitter_test.ts doesn't touch it.

apilaskowski · 2026-09-18T20:34:41Z

+  const universeDomain = question(
+    "Enter the universe domain to connect to, or leave blank to use the default " +
+      "('googleapis.com'). Set this only when targeting a non-default universe such as a " +
+      "Trusted Partner Cloud (TPC):"
+  ).trim();


Could we avoid prompting every user for this? Nearly all of them are on GDU, and the proto change by itself already lets the rare TPC user set the field by hand in .df-credentials.json.

If it does stay interactive, it's currently untested — cli/index_init_test.ts only covers the failure path.

apilaskowski · 2026-09-18T20:34:41Z

    credentials: fs.readFileSync(cloudCredentialsPath, "utf8"),
-    location
+    location,
+    ...(universeDomain ? { universeDomain } : {})


If someone supplies a TPC service-account key but leaves the prompt blank, we write no universeDomain: BigQuery then defaults to googleapis.com while google-auth derives the key's own universe, and they disagree at query time. Worth falling back to cloudCredentials.universe_domain here.

apilaskowski · 2026-09-18T20:34:41Z

          location: credentials.location,
-          credentials: credentials.credentials && JSON.parse(credentials.credentials)
+          credentials: credentials.credentials && JSON.parse(credentials.credentials),
+          ...(credentials.universeDomain ? { universeDomain: credentials.universeDomain } : {})


Nit: redundant here — @google-cloud/bigquery already falsy-checks options.universeDomain (bigquery.js:120), so universeDomain: credentials.universeDomain || undefined would do.

The guard in emitter.ts is needed though, since gax uses ?? — might be worth a comment there noting the asymmetry.

apilaskowski · 2026-09-18T20:34:41Z

+    expect(credentials.universeDomain).to.satisfy(
+      (value: string) => value === "" || value === undefined
+    );


read() returns a protobufjs message, so an unset string is deterministically "" — this can just be expect(credentials.universeDomain).to.equal(""). As written it would still pass if the value became undefined, which is the distinction the guards in bigquery.ts and emitter.ts depend on.

apilaskowski · 2026-09-18T20:34:41Z

    await adapter.setMetadata(action);
  });
+
+  suite("createBigQueryClientProvider", () => {


Nit: this suite doesn't exercise BigQueryDbAdapter — could it be a sibling top-level suite rather than nested inside it?

apilaskowski · 2026-09-18T20:34:41Z

  string credentials = 3;
  // Options are listed here: https://cloud.google.com/bigquery/docs/locations
  string location = 4;
+  // The universe domain to connect to (e.g. "googleapis.com"). Leave unset to


Nit: (e.g. "googleapis.com") next to "leave unset to use the default" is a little contradictory, since that's the one value you'd never set here. A non-default universe would be a clearer example.

apilaskowski · 2026-09-18T20:34:41Z

   - `projectId`: your GCP project id
   - `credentials`: the entire content of your GCP service account key JSON file as a single string (you can generate it with `jq -Rsa < path/to/key.json`).
   - `location`: location to use in your project
+   - `universeDomain` (optional): the universe domain to connect to (e.g. `googleapis.com`). Leave unset to use the default Google Default Universe (GDU). Set this only when targeting a non-default universe such as a Trusted Partner Cloud (TPC).


This covers the integration-test credentials only — is a matching update to the user-facing Cloud docs tracked anywhere?

apilaskowski · 2026-09-18T20:41:25Z

Thanks for this — the BigQuery half looks good.

Two things I'd like to sort out before this might be considered:

The lineage hunk doesn't actually take effect, and it suppresses gax's universe-mismatch check.
init-creds now asks every user a TPC question.

One question: EXTRA_GOOGLE_SCOPES hardcodes the Drive scope — is that grantable in TPC? And did your manual test cover the ADC path as well as the JSON-key one?

(FYI the CLA check is failing, and the title is still the generated branch name.)

ybelleguic requested a review from a team as a code owner September 18, 2026 12:59

ybelleguic requested review from apilaskowski and removed request for a team September 18, 2026 12:59

Yohan Belleguic added 2 commits September 18, 2026 15:03

Document optional universeDomain credentials field

2f09b5c

ybelleguic force-pushed the cursor/bigquery-universe-domain-clean-3190 branch from 821f551 to 2f09b5c Compare September 18, 2026 13:06

apilaskowski requested changes Sep 18, 2026

View reviewed changes

Navigation Menu

Cursor/bigquery universe domain clean 3190 - #2323

Cursor/bigquery universe domain clean 3190#2323
ybelleguic wants to merge 2 commits into
dataform-co:mainfrom
ybelleguic:cursor/bigquery-universe-domain-clean-3190

ybelleguic commented Sep 18, 2026

Uh oh!

google-cla Bot commented Sep 18, 2026

Uh oh!

apilaskowski Sep 18, 2026

Uh oh!

apilaskowski Sep 18, 2026

Uh oh!

apilaskowski Sep 18, 2026

Uh oh!

apilaskowski Sep 18, 2026

Uh oh!

apilaskowski Sep 18, 2026

Uh oh!

apilaskowski Sep 18, 2026

Uh oh!

apilaskowski Sep 18, 2026

Uh oh!

apilaskowski Sep 18, 2026

Uh oh!

apilaskowski commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ybelleguic commented Sep 18, 2026

What

Why

Changes

Testing

Uh oh!

google-cla Bot commented Sep 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

apilaskowski commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants