Skip to content

Configuration to use Depot on GitLab Ci to avoid rate limiting with Maven Central. - #12222

Closed
AlexeyKuznetsov-DD wants to merge 3 commits into
masterfrom
alexeyk/use-depot-to-avoid-rate-limiting
Closed

Configuration to use Depot on GitLab Ci to avoid rate limiting with Maven Central.#12222
AlexeyKuznetsov-DD wants to merge 3 commits into
masterfrom
alexeyk/use-depot-to-avoid-rate-limiting

Conversation

@AlexeyKuznetsov-DD

@AlexeyKuznetsov-DD AlexeyKuznetsov-DD commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

Makes the configured Maven repository proxy authoritative for general dependency resolution:

  • Gradle project repositories use mavenRepositoryProxy when configured and use mavenCentral() only when the proxy property is absent.
  • Muzzle uses only MAVEN_REPOSITORY_PROXY when configured instead of querying both the proxy and Maven Central.

GitLab CI already supplies these proxy settings, so CI resolves these dependencies through Depot. Local and external builds without the settings continue to use Maven Central.

Motivation

The main source of direct Maven Central traffic was the latest-dependency configurations that use dynamic versions such as 2.+, 2.7.+, and 2.3.20+.

For a dynamic selector, Gradle must discover all matching versions. Gradle documentation explains that Gradle checks every defined repository for version metadata rather than stopping after the first repository responds. Consequently, declaring Depot first and Maven Central second did not make Central a passive fallback: Gradle requested maven-metadata.xml from both repositories for each dynamic dependency.

Under parallel CI load, those direct requests can accumulate behind shared egress and eventually result in Maven Central rate limiting. A representative failure was:

Could not resolve com.typesafe.play:play-java_2.13:2.7.+.
Could not GET .../maven-metadata.xml.
Received status code 429 from server: Too Many Requests

Muzzle had the same issue for its version-range scans because its default repository list contained both the proxy and Maven Central.

This change keeps Depot as the general remote repository whenever the proxy is configured, so dynamic-version and Muzzle range lookups no longer query Maven Central alongside Depot through the general repository list.

Additional Notes

Validation:

  • Resolved org.glassfish.grizzly:grizzly-http-server:2.3.20+ for latestDepTestRuntimeClasspath with the proxy configured; the resolver log showed Depot serving the metadata and no request to Maven Central.
  • ./gradlew :buildSrc:test --tests datadog.gradle.plugin.muzzle.MuzzleMavenRepoUtilsTest — 4 tests passed.
  • ./gradlew :buildSrc:spotlessCheck

Specialized repositories with explicit content filters are unchanged.

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@AlexeyKuznetsov-DD AlexeyKuznetsov-DD self-assigned this Aug 16, 2026
@AlexeyKuznetsov-DD
AlexeyKuznetsov-DD requested a review from a team as a code owner August 16, 2026 23:33
@AlexeyKuznetsov-DD AlexeyKuznetsov-DD added the tag: do not merge Do not merge changes label Aug 16, 2026
@AlexeyKuznetsov-DD AlexeyKuznetsov-DD added tag: no release notes Changes to exclude from release notes tag: dependencies Dependencies related changes comp: tooling Build & Tooling labels Aug 16, 2026
@AlexeyKuznetsov-DD
AlexeyKuznetsov-DD marked this pull request as draft August 16, 2026 23:33

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

Proxy-disabled builds retain Maven Central, while proxy-enabled Muzzle directives still append their explicit framework repositories and Gradle builds retain the dedicated snapshot, Confluent, Spring, and Gradle repositories. The changed fallback behavior is consistent with making Depot authoritative for Central artifacts.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Datadog Autotest · Commit e3069e7 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

@datadog-prod-us1-5

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.87 s 14.86 s [-1.2%; +1.4%] (no difference)
startup:insecure-bank:tracing:Agent 13.63 s 13.74 s [-1.5%; -0.1%] (maybe better)
startup:petclinic:appsec:Agent 17.44 s 17.32 s [-0.1%; +1.4%] (no difference)
startup:petclinic:iast:Agent 17.43 s 16.97 s [-1.7%; +7.2%] (no difference)
startup:petclinic:profiling:Agent 17.41 s 17.39 s [-1.2%; +1.4%] (no difference)
startup:petclinic:sca:Agent 17.51 s 16.52 s [+1.6%; +10.3%] (significantly worse)
startup:petclinic:tracing:Agent 16.59 s 16.68 s [-1.5%; +0.5%] (no difference)

Commit: 357e8a8a · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@AlexeyKuznetsov-DD
AlexeyKuznetsov-DD marked this pull request as ready for review August 18, 2026 00:00

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

A present-but-blank proxy value is now treated as authoritative, removing Maven Central from general Gradle and Muzzle resolution. Shell-based CI can materialize missing optional variables as empty strings, turning a clean build into an unrecoverable dependency-resolution failure.

Open Bits AI session

🤖 Datadog Autotest · Commit 357e8a8 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

} else {
val proxy = RemoteRepository.Builder("central-proxy", "default", mavenProxyUrl).build()
listOf(proxy, central)
listOf(proxy)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Treat blank proxy values as absent

CI or external builds that export an unset optional proxy variable as an empty string can fail all general dependency resolution before tests run.

Assertion details
  • Input: MAVEN_REPOSITORY_PROXY or ORG_GRADLE_PROJECT_mavenRepositoryProxy is present but empty or whitespace-only, such as when shell setup exports an undefined variable.
  • Expected: Normalize both proxy values before repository selection and treat blank or whitespace-only values as absent, retaining Maven Central. This requires coordinated changes in both changed files and coverage for blank environment/property values, so no single inline suggestion is complete.
  • Actual: Both changed paths use presence-only checks. An empty MAVEN_REPOSITORY_PROXY now produces only an empty proxy repository, while a present-but-empty Gradle property suppresses mavenCentral(); clean dependency resolution therefore has no usable general repository. Existing tests cover non-empty fixture proxies but not blank default-repository selection.

Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

@AlexeyKuznetsov-DD

Copy link
Copy Markdown
Contributor Author

Closing this, since infra fixed network configuration that should prevent rate-limiting on Maven Central.
Several latest pipelines passed on master and PRs for last 3 days without rate limiting.

@AlexeyKuznetsov-DD
AlexeyKuznetsov-DD deleted the alexeyk/use-depot-to-avoid-rate-limiting branch August 19, 2026 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: tooling Build & Tooling tag: dependencies Dependencies related changes tag: do not merge Do not merge changes tag: no release notes Changes to exclude from release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants