Deduplicate GitLab test job templates #12199
+51
−122
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -685,35 +685,18 @@ muzzle-dep-report: | |
| .restore_pretest_env: &restore_pretest_env | ||
| - source pretest.env | ||
|
|
||
| .test_job_common: | ||
| extends: .gradle_build | ||
| image: ${BUILDER_IMAGE_REPO}:${BUILDER_IMAGE_VERSION_PREFIX}$testJvm | ||
| variables: | ||
| <<: *tier_m_variables | ||
| GRADLE_PARAMS: "-PskipFlakyTests" | ||
| CONTINUE_ON_FAILURE: "false" | ||
| TEST_JVM_ARGS: "" | ||
| TESTCONTAINERS_CHECKS_DISABLE: "true" | ||
| TESTCONTAINERS_RYUK_DISABLED: "true" | ||
| TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX: "registry.ddbuild.io/images/mirror/" | ||
| JETTY_AVAILABLE_PROCESSORS: 4 # Jetty incorrectly calculates processor count in containers | ||
| script: | ||
| - *gitlab_base_ref_params | ||
| - > | ||
|
|
@@ -722,7 +705,7 @@ muzzle-dep-report: | |
| export PROFILER_COMMAND="-XX:StartFlightRecording=settings=profile,filename=/tmp/${CI_JOB_NAME_SLUG}.jfr,dumponexit=true"; | ||
| fi | ||
| - *prepare_test_env | ||
| - export GRADLE_OPTS="-Dorg.gradle.jvmargs='$TEST_JVM_ARGS -Xms$GRADLE_MEMORY_MIN -Xmx$GRADLE_MEMORY_MAX $PROFILER_COMMAND -XX:ErrorFile=/tmp/hs_err_pid%p.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djava.util.prefs.userRoot=/tmp/.java/.userPrefs-${CI_JOB_ID}' -Ddatadog.forkedMinHeapSize=128M -Ddatadog.forkedMaxHeapSize=1024M" | ||
| - ./gradlew --version | ||
| - ./gradlew $GRADLE_TARGET $GRADLE_PARAMS -PtestJvm=$testJvm -Pslot=$CI_NODE_INDEX/$CI_NODE_TOTAL $GRADLE_ARGS --continue || $CONTINUE_ON_FAILURE | ||
| after_script: | ||
|
|
@@ -767,24 +750,44 @@ muzzle-dep-report: | |
| - scheduler_failure | ||
| - data_integrity_failure | ||
|
|
||
| .test_job: | ||
| extends: .test_job_common | ||
| tags: [ "docker-in-docker:amd64" ] # use docker-in-docker runner for testcontainers | ||
| needs: [ build_tests ] | ||
| stage: tests | ||
| rules: | ||
| # Protected branches (master/mq/gh-readonly): all JVMs run unconditionally | ||
| - if: '$CI_COMMIT_BRANCH == "master"' | ||
| when: on_success | ||
| - if: '$CI_COMMIT_BRANCH =~ /^mq-working-branch-/' | ||
| when: on_success | ||
| - if: '$CI_COMMIT_BRANCH =~ /^gh-readonly-queue/' | ||
| when: on_success | ||
| # Enable for default test JVMs or for NON_DEFAULT_JVMS | ||
| - if: '$NON_DEFAULT_JVMS == "true"' | ||
| when: on_success | ||
| - if: '$CI_COMMIT_MESSAGE =~ /\[ci: NON_DEFAULT_JVMS\]/' | ||
| when: on_success | ||
| - if: '$testJvm =~ $DEFAULT_TEST_JVMS' | ||
| when: on_success | ||
|
|
||
| .test_job_arm64: | ||
| extends: .test_job_common | ||
| tags: [ "docker-in-docker:arm64" ] | ||
| stage: tests-arm64 | ||
| # Use the amd64 build only as a compilation gate. Do not download its platform-specific artifacts. | ||
| needs: | ||
| - job: build | ||
| artifacts: false | ||
| variables: | ||
| DEFAULT_TEST_JVMS: /^(8|11|17|21|25|27|tip)$/ # Java 27 TODO: remove 27 after GA (tip will move to 27) | ||
| TEST_JVM_ARGS: "-Xshare:off" | ||
|
Contributor
There was a problem hiding this comment. suggestion: It might be a good thing to comment why this is needed. Honestly I didn't dived into why this was needed, I know it disables class data dharing (CDS) which is used to speed up JVM botstrap, that said this mechanism evolved overt JVM releases so I'm not sure what's at play regarding ARM64 in more recent versions.
If issues arose, I wonder if Linux memory pages are at play when running on ARM64, unusually these systems may use 4 KiB, 16 KiB, or 64 KiB memory pages. And I wonder if the CDS archive has some issues with memory alignment on unusual page size, that said I can be completely off-track. However about the "trade-off", I would mention slightly slower jvm bootstrap, and maybe a tad higher memory use.
Contributor
There was a problem hiding this comment. thought: It's just a thought, but interestingly, I wonder if the env var can be picked up, by programmatic launches that are using this all over the place, we might be able to drop the That said, maybe, this env var need to be more specific? E.g. |
||
| rules: | ||
| # IBM 8 has no arm64 image published upstream. | ||
| - if: '$testJvm == "ibm8"' | ||
| when: never | ||
| # Oracle 8 is available on arm64 but too flaky to run in CI. | ||
| - if: '$testJvm == "oracle8"' | ||
| when: never | ||
| # arm64 tests are newly introduced to the merge queue and master. Keep them | ||
| # non-blocking (allow_failure) for now so we can collect stability stats and | ||
|
|
@@ -803,6 +806,7 @@ muzzle-dep-report: | |
| - if: '$CI_COMMIT_MESSAGE =~ /\[ci: NON_DEFAULT_JVMS\]/' | ||
| when: on_success | ||
| allow_failure: true | ||
| # Keep the default JVM subset available for manual runs on feature branches/PRs. | ||
| - if: '$testJvm =~ $DEFAULT_TEST_JVMS' | ||
| when: manual | ||
| allow_failure: true | ||
|
|
@@ -820,75 +824,9 @@ muzzle-dep-report: | |
| unprotect: true | ||
| before_script: | ||
| - git config --global --add safe.directory "$CI_PROJECT_DIR" | ||
| - !reference [.gradle_build, before_script] | ||
|
|
||
| .test_job_with_test_agent_common: | ||
|
Contributor
There was a problem hiding this comment. suggestion: I'd make architecture explicit |
||
| variables: | ||
| CI_USE_TEST_AGENT: "true" | ||
| CI_AGENT_HOST: local-agent | ||
|
|
@@ -903,27 +841,18 @@ muzzle-dep-report: | |
| DD_DISABLE_ERROR_RESPONSES: "true" | ||
| ENABLED_CHECKS: "trace_content_length,trace_stall,meta_tracer_version_header,trace_count_header,trace_peer_service,trace_dd_service" | ||
| script: | ||
| - !reference [.test_job_common, script] | ||
| - .gitlab/check_test_agent_results.sh | ||
|
|
||
| .test_job_with_test_agent: | ||
|
Contributor
There was a problem hiding this comment. suggestion: I'd make architecture explicit |
||
| extends: | ||
| - .test_job | ||
| - .test_job_with_test_agent_common | ||
|
|
||
| .test_job_arm64_with_test_agent: | ||
| extends: | ||
| - .test_job_arm64 | ||
| - .test_job_with_test_agent_common | ||
|
|
||
| agent_integration_tests: | ||
| extends: .test_job | ||
|
|
@@ -960,7 +889,7 @@ test_base: | |
| matrix: *test_matrix_4 | ||
| script: | ||
| - if [ "$testJvm" == "8" ]; then export GRADLE_PARAMS="-PskipFlakyTests -PcheckCoverage"; fi | ||
| - !reference [.test_job_common, script] | ||
|
|
||
| test_base_arm64: | ||
| extends: .test_job_arm64 | ||
|
|
@@ -972,7 +901,7 @@ test_base_arm64: | |
| # run coverage only on JVM 8, mirroring the amd64 test_base job | ||
| script: | ||
| - if [ "$testJvm" == "8" ]; then export GRADLE_PARAMS="$GRADLE_PARAMS -PcheckCoverage"; fi | ||
| - !reference [.test_job_common, script] | ||
|
|
||
| test_inst: | ||
| extends: .test_job_with_test_agent | ||
|
|
@@ -1100,7 +1029,7 @@ test_debugger: | |
| # exepected coverage | ||
| script: | ||
| - if [[ "$testJvm" != "semeru8" && "$testJvm" != "semeru11" && "$testJvm" != "semeru17" && "$testJvm" != "ibm8" ]]; then export GRADLE_PARAMS="$GRADLE_PARAMS -PcheckCoverage"; fi | ||
| - !reference [.test_job_common, script] | ||
|
|
||
| test_debugger_arm64: | ||
| extends: .test_job_arm64 | ||
|
|
@@ -1113,7 +1042,7 @@ test_debugger_arm64: | |
| # expected coverage (ibm8/oracle8 never run on arm64) | ||
| script: | ||
| - if [[ "$testJvm" != "semeru8" && "$testJvm" != "semeru11" && "$testJvm" != "semeru17" ]]; then export GRADLE_PARAMS="$GRADLE_PARAMS -PcheckCoverage"; fi | ||
| - !reference [.test_job_common, script] | ||
|
|
||
| test_smoke: | ||
| extends: .test_job_with_test_agent | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
suggestion: I'd make the arcvhitecture explicit for
amd64