Skip to content

fix(operator): support custom cluster domain#183

Merged
GatewayJ merged 2 commits into
rustfs:mainfrom
GatewayJ:fix/custom-cluster-domain
Jul 12, 2026
Merged

fix(operator): support custom cluster domain#183
GatewayJ merged 2 commits into
rustfs:mainfrom
GatewayJ:fix/custom-cluster-domain

Conversation

@GatewayJ

Copy link
Copy Markdown
Member

Type of Change

  • New Feature
  • Bug Fix
  • Documentation
  • Performance Improvement
  • Test/CI
  • Refactor
  • Other:

Related Issues

Closes #152

Summary of Changes

This PR adds first-class support for Kubernetes clusters whose DNS domain is not cluster.local.

Root cause: the operator rendered Tenant peer URLs, generated TLS SANs, and some operator-side TLS clients with a hardcoded cluster.local assumption. In TLS-enabled deployments this also made SAN validation require DNS names that did not match the actual headless-service certificate shape used by custom-domain clusters.

The fix:

  • adds a validated OPERATOR_CLUSTER_DOMAIN runtime setting and a Helm clusterDomain value
  • threads the configured domain through Tenant StatefulSet rendering, TLS reconciliation, pool lifecycle checks, provisioning, tenant monitoring, and STS
  • renders RUSTFS_VOLUMES peer URLs with the configured cluster domain
  • validates required internode TLS names against the actual headless Service and pod DNS names, while allowing a one-label wildcard such as *.tenant-hl.ns.svc.example.cluster
  • makes operator TLS admin clients use the Tenant headless Service FQDN so custom headless certificates work end-to-end
  • updates e2e config and tests to exercise custom cluster-domain rendering

Checklist

  • I have read and followed the CONTRIBUTING.md guidelines
  • Passed make pre-commit (fmt-check + clippy + test + console-lint + console-fmt-check)
  • Added/updated necessary tests
  • Documentation updated (if needed)
  • CHANGELOG.md updated under [Unreleased] (if user-visible change) - N/A: this repository does not currently contain a CHANGELOG.md
  • CI/CD passed (if applicable)

Impact

  • Breaking change (CRD/API compatibility)
  • Requires doc/config/deployment update
  • Other impact:

Verification

make pre-commit
git diff --check
jq empty deploy/rustfs-operator/values.schema.json

Additional Notes

The Helm chart now owns OPERATOR_CLUSTER_DOMAIN; users should set clusterDomain instead of adding OPERATOR_CLUSTER_DOMAIN through operator.env.

@GatewayJ GatewayJ marked this pull request as ready for review July 12, 2026 13:00
@GatewayJ GatewayJ added this pull request to the merge queue Jul 12, 2026
Merged via the queue into rustfs:main with commit 360345a Jul 12, 2026
3 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9906b8f84d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/sts/rustfs_client.rs
Comment on lines +427 to +429
let service_fqdn =
cluster_dns::service_fqdn(&tenant.headless_service_name(), &namespace, cluster_domain);
Ok(format!("https://{service_fqdn}:9000"))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep TLS admin clients on a covered Service DNS name

This switches all operator-side TLS admin calls to the headless Service FQDN, but SAN validation only requires the headless/pod DNS names when enableInternodeHttps=true (san_validation_dns_names falls back to configured names when includeGeneratedDnsNames=false). For a TLS tenant with enableInternodeHttps=false and includeGeneratedDnsNames=false whose certificate explicitly covers the previously used *-io.<ns>.svc endpoint, reconciliation can still mark TLS ready, then STS/provisioning/monitor clients fail hostname verification against *-hl.<ns>.svc.<domain>. Please either require this FQDN in validation whenever this client is used or connect to a Service name that is already validated.

Useful? React with 👍 / 👎.

Comment thread src/reconcile/tls.rs
Comment on lines +1234 to +1237
if entry.default && config.enable_internode_https {
names.extend(required_tls_dns_names(tenant, namespace, cluster_domain));
} else if include_generated_dns_names(entry) {
names.extend(generated_dns_names(tenant, namespace, cluster_domain));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep IO Service names in default TLS SAN checks

When enableInternodeHttps=true, this branch now validates only the headless Service and pod DNS names, so an external TLS Secret with the default includeGeneratedDnsNames=true can pass requireSanMatch while missing tenant-io.<ns>.svc[.<domain>]. The Tenant is still exposed through the *-io Service, so HTTPS clients using that Service get hostname verification failures even though the operator reports TLS ready; include the generated IO Service names in this validation in addition to the internode names.

Useful? React with 👍 / 👎.

Comment thread src/sts/tls.rs
let generated = generate_sts_tls_material(
&config.namespace,
&config.service_name,
&config.cluster_domain,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Regenerate managed STS certs when the cluster domain changes

This applies the configured cluster domain only when a new managed sts-tls Secret is generated; load_or_create_sts_tls_material still accepts an existing managed Secret as long as the keys parse. Upgrading an install that already has the auto-generated cluster.local STS certificate and then setting clusterDomain leaves the served certificate without the new *.svc.<domain> SAN, so clients using the advertised FQDN fail TLS until the Secret is manually deleted. Please validate the existing managed Secret SANs against the current service_dns_names and regenerate when they no longer match.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Allow to override Kubernetes cluster name from cluster.local to a custom domain

1 participant