fix(testkit): replace the deleted minio/minio fixture image with adobe/s3mock - #324
Merged
Conversation
…e/s3mock minio/minio has been REMOVED from Docker Hub, so all four *CopyIntoS3Spec suites (es6rest, es7rest, es8java, es9java) abort before their first test, in MinioTestKit.start, with ContainerFetchException "... pull access denied for minio/minio, repository does not exist or may require 'docker login'". Measured, so it is neither a rate limit nor a login problem: `docker pull minio/minio:latest` fails that way locally and on the runner, and https://hub.docker.com/v2/repositories/minio/minio/ answers 404 - the repository itself is gone (a throttle is 429, an auth failure 401). Replacement: adobe/s3mock, pinned 5.2.2, driven by a hand-rolled GenericContainer. The trait keeps its name and its whole public surface, so CopyIntoS3IntegrationSpec and the four concrete specs need no change. No production source is touched by this branch. Why not the alternatives, each rejected on measurement: * LocalStack - Pro-gated. localstack/localstack:latest (2026.8.2, built 2026-09-09) exits code 55 with "License activation failed! ... No credentials were found in the environment". It cannot be an unauthenticated CI fixture. * The quay.io MinIO mirror - its newest COMMUNITY tag is RELEASE.2025-09-07T16-13-09Z (pushed 2025-09-07, and `latest` points at the same build); every newer push is a .hotfix.* build. It would pin the suites to a year-old frozen AGPL binary. * The s3mock-testcontainers MODULE (rather than GenericContainer) - 5.2.2 is Java-17 bytecode built against Testcontainers 1.21.3, while this build's default JDK is 11 and it pins Testcontainers 2.0.2. adobe/s3mock declares NO EXPOSE (buildpacks image), so withExposedPorts(9090) is required; it seeds buckets from COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS, replacing the two `mc` execInContainer calls. testcontainers-minio gives way to the core testcontainers artifact in both build files. The swap silently removes two protections, so both are replaced: (a) Bucket seeding is silently fallible - Adobe renamed the property between 4.x and 5.x, and the old name leaves the container healthy, answering 200, with an EMPTY bucket list. start() now probes and fails naming the env var. The obvious probe is VACUOUS and that was measured: fs.exists("s3a://<bucket>/") returned true against a container with no buckets (S3A answers the ROOT path from a synthetic directory status; the fs.s3a.bucket.probe default is 0, so nothing HEADs the bucket). The shipped probe sets fs.s3a.bucket.probe=2 and issues a real listStatus, and it distinguishes "bucket absent" from "cannot reach the container" so a missing hadoop-aws is not misreported. Falsified: with the 4.x env-var name the suite ABORTS with IllegalStateException: S3 bucket 'copy-into-test' does not exist on adobe/s3mock:5.2.2 at http://localhost:62469. The container seeds buckets from the env var 'COM_ADOBE_TESTING_S3MOCK_INITIALBUCKETS' - check that name: an unknown one is ignored silently and leaves the container healthy but empty. (b) s3mock does not verify credentials AT ALL, where MinIO answered 403. MinioTestKit sets AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_ENDPOINT_URL as system properties so the production HadoopConfigurationFactory.s3aConf picks them up via envOrProp, and nothing else in the repo asserted that. Two replacements, because one is not enough: - the bucket probe builds its Configuration from the PRODUCTION forPath, not from minioHadoopConf, so the suites really exercise those three properties. Falsified: misspelling ONE of them (AWS_ENDPOINT_URL -> AWS_ENDPOINT_URI) aborts the suite with "Could not reach the S3 bucket 'copy-into-test' ...". - two LocalPathSpec unit tests pin what the properties must produce. Falsified: narrowing envOrProp to sys.env only reddens exactly 1 of the 2 ("read credentials and endpoint from JVM system properties"); the no-static-key test correctly stays green, which is what makes the pair a discriminator rather than a restatement. Verified: the four *CopyIntoS3Spec suites green on real Elasticsearch 6.8.23 / 7.17.29 / 8.18.3 / 9.0.3 (3 tests each); core/test green; testkit and core compile on 2.13.16 and 2.12.20; CI's lint task list (headerCheck scalafmtSbtCheck scalafmtCheck test:scalafmtCheck) green run cold. Follow-up, deliberately not done here: renaming MinioTestKit to S3TestKit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fupelaqu
marked this pull request as ready for review
September 12, 2026 08:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #323
Why
minio/miniohas been deleted from Docker Hub, so the four*CopyIntoS3Specsuites abort before their first test, insideMinioTestKit.start():Measured, so it is neither a rate limit nor a login problem:
docker pull minio/minio:latestfails the same way on a developer machine as on the runner, andhttps://hub.docker.com/v2/repositories/minio/minio/returns 404 (a throttle returns 429, an auth problem 401). Nothing in this repository changed. CI run that caught it: https://github.com/SOFTNETWORK-APP/SoftClient4ES/actions/runs/34655370845/job/103446486136What changed
The fixture image becomes
adobe/s3mock:5.2.2, driven by a hand-rolled TestcontainersGenericContainer. Only the tracked templatetestkit/.../MinioTestKit.scalais edited; the foures{6,7,8,9}/testkitcopies are generated bycopyTestkit()and gitignored, so they follow automatically. The trait keeps its name, so the four concrete specs andCopyIntoS3IntegrationSpecare untouched; renaming it toS3TestKitis a deliberate follow-up.Rejected on measurement, not preference:
localstack/localstack:latest(2026.8.2) exits code 55 withLicense activation failed. Unusable as an unauthenticated CI fixture.RELEASE.2025-09-07T16-13-09Z; everything newer is a.hotfix.*build. It would pin the suites to a frozen AGPL binary from a vendor that has already closed one of its two channels.s3mock-testcontainersmodule — 5.2.2 is Java-17 bytecode built against Testcontainers 1.21.3, while this build's default JDK is 11 and it pins Testcontainers 2.0.2.GenericContaineravoids both.adobe/s3mock:5.2.2is ~84 MB, serves in about 2 s, declares noEXPOSE(buildpacks image, sowithExposedPorts(9090)is required), and seeds buckets fromCOM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS.Two protections the swap removes, and their replacements
(a) Bucket seeding is silently fallible. Adobe renamed the seeding property between 4.x and 5.x. Passing the old name is invisible: the container starts, answers 200 on
/, and lists no buckets.start()now asserts the bucket exists and names the env var when it does not. The obvious probe would have been vacuous:fs.exists("s3a://<bucket>/")returns true against a container with no buckets, because S3A answers the root path from a synthetic directory status andfs.s3a.bucket.probedefaults to0.(b) s3mock does not verify credentials at all, where MinIO answered 403. The testkit's system properties are what exercise the documented
AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEYcontract (documentation/sql/dml_statements.md:364-375) end to end, and nothing else in the repository asserts that propagation. Replaced by a unit assertion inLocalPathSpecon the resultingfs.s3a.access.key,fs.s3a.secret.key,fs.s3a.endpointandfs.s3a.path.style.access.Both gates were shown failing before passing:
S3 bucket 'copy-into-test' does not exist on adobe/s3mock:5.2.2 ... seeds buckets from the env var '...' - check that nameAWS_ENDPOINT_URLmisspelled instart()Could not reach the S3 bucketenvOrPropnarrowed tosys.envonlyLocalPathSpectests red,read credentials and endpoint from JVM system properties; the no-static-key test correctly stays greenVerification
Four
*CopyIntoS3Specsuites 3/3 each on real Elasticsearch 6.8.23 / 7.17.29 / 8.18.3 / 9.0.3;core/test981/981 (43 suites);LocalPathSpec43/43;++ 2.12.20core test compile and testkit compile;headerCheck,scalafmtSbtCheck,scalafmtCheck,test:scalafmtCheckgreen from a cold cache. An independent reviewer re-ran both mutations and the es8 and es6 suites.Notes
softclient4es{N}-core-testkitartifacts change theminioContainerfield type fromMinIOContainertoMinioTestKit.S3MockContainer, which is binary-incompatible for an external subclass. Nothing in this repository or in softclient4es-jdbc, softclient4es-arrow or softclient4es-extensions subclasses it.dml_statements.md:370documents the credential fallback as an AWS SDK v1 class name, which cannot load from this build's own classpath but remains functional for a consumer supplying the v1 SDK, sincehadoop-awsis aprovideddependency. Documentation-versus-implementation mismatch on a secondary path; the documented primary path works and is what these suites exercise.uploadToMinioleaks a cached FileSystem iffs.createthrows.🤖 Generated with Claude Code