Skip to content

fix(testkit): replace the deleted minio/minio fixture image with adobe/s3mock - #324

Merged
fupelaqu merged 1 commit into
mainfrom
fix/s3mock-testkit
Sep 12, 2026
Merged

fix(testkit): replace the deleted minio/minio fixture image with adobe/s3mock#324
fupelaqu merged 1 commit into
mainfrom
fix/s3mock-testkit

Conversation

@fupelaqu

@fupelaqu fupelaqu commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Closes #323

Why

minio/minio has been deleted from Docker Hub, so the four *CopyIntoS3Spec suites abort before their first test, inside MinioTestKit.start():

org.testcontainers.containers.ContainerFetchException:
  ... pull access denied for minio/minio, repository does not exist or may require 'docker login'

Measured, so it is neither a rate limit nor a login problem: docker pull minio/minio:latest fails the same way on a developer machine as on the runner, and https://hub.docker.com/v2/repositories/minio/minio/ returns 404 (a throttle returns 429, an auth problem 401). Nothing in this repository changed. CI run that caught it: https://github.com/SOFTNETWORK-APP/SoftClient4ES/actions/runs/34655370845/job/103446486136

What changed

The fixture image becomes adobe/s3mock:5.2.2, driven by a hand-rolled Testcontainers GenericContainer. Only the tracked template testkit/.../MinioTestKit.scala is edited; the four es{6,7,8,9}/testkit copies are generated by copyTestkit() and gitignored, so they follow automatically. The trait keeps its name, so the four concrete specs and CopyIntoS3IntegrationSpec are untouched; renaming it to S3TestKit is a deliberate follow-up.

Rejected on measurement, not preference:

  • LocalStack — the current image is Pro-gated: localstack/localstack:latest (2026.8.2) exits code 55 with License activation failed. Unusable as an unauthenticated CI fixture.
  • The quay.io MinIO mirror — its newest community tag is RELEASE.2025-09-07T16-13-09Z; everything newer is a .hotfix.* build. It would pin the suites to a frozen AGPL binary from a vendor that has already closed one of its two channels.
  • The s3mock-testcontainers module — 5.2.2 is Java-17 bytecode built against Testcontainers 1.21.3, while this build's default JDK is 11 and it pins Testcontainers 2.0.2. GenericContainer avoids both.

adobe/s3mock:5.2.2 is ~84 MB, serves in about 2 s, declares no EXPOSE (buildpacks image, so withExposedPorts(9090) is required), and seeds buckets from COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS.

Two protections the swap removes, and their replacements

(a) Bucket seeding is silently fallible. Adobe renamed the seeding property between 4.x and 5.x. Passing the old name is invisible: the container starts, answers 200 on /, and lists no buckets. start() now asserts the bucket exists and names the env var when it does not. The obvious probe would have been vacuous: fs.exists("s3a://<bucket>/") returns true against a container with no buckets, because S3A answers the root path from a synthetic directory status and fs.s3a.bucket.probe defaults to 0.

(b) s3mock does not verify credentials at all, where MinIO answered 403. The testkit's system properties are what exercise the documented AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY contract (documentation/sql/dml_statements.md:364-375) end to end, and nothing else in the repository asserts that propagation. Replaced by a unit assertion in LocalPathSpec on the resulting fs.s3a.access.key, fs.s3a.secret.key, fs.s3a.endpoint and fs.s3a.path.style.access.

Both gates were shown failing before passing:

gate mutation result
(a) the 4.x env-var name restored suite ABORTS: S3 bucket 'copy-into-test' does not exist on adobe/s3mock:5.2.2 ... seeds buckets from the env var '...' - check that name
(a) AWS_ENDPOINT_URL misspelled in start() suite ABORTS with the other diagnosis, Could not reach the S3 bucket
(b) envOrProp narrowed to sys.env only 1 of the 2 new LocalPathSpec tests red, read credentials and endpoint from JVM system properties; the no-static-key test correctly stays green

Verification

Four *CopyIntoS3Spec suites 3/3 each on real Elasticsearch 6.8.23 / 7.17.29 / 8.18.3 / 9.0.3; core/test 981/981 (43 suites); LocalPathSpec 43/43; ++ 2.12.20 core test compile and testkit compile; headerCheck, scalafmtSbtCheck, scalafmtCheck, test:scalafmtCheck green from a cold cache. An independent reviewer re-ran both mutations and the es8 and es6 suites.

Notes

  • No production source is touched by this PR.
  • The published softclient4es{N}-core-testkit artifacts change the minioContainer field type from MinIOContainer to MinioTestKit.S3MockContainer, which is binary-incompatible for an external subclass. Nothing in this repository or in softclient4es-jdbc, softclient4es-arrow or softclient4es-extensions subclasses it.
  • Recorded in S3 test-fixture image minio/minio was deleted from Docker Hub — all four *CopyIntoS3Spec suites abort #323 and deliberately not changed here: dml_statements.md:370 documents the credential fallback as an AWS SDK v1 class name, which cannot load from this build's own classpath but remains functional for a consumer supplying the v1 SDK, since hadoop-aws is a provided dependency. Documentation-versus-implementation mismatch on a secondary path; the documented primary path works and is what these suites exercise.
  • Pre-existing and untouched: uploadToMinio leaks a cached FileSystem if fs.create throws.

🤖 Generated with Claude Code

…e/s3mock

minio/minio has been REMOVED from Docker Hub, so all four *CopyIntoS3Spec suites
(es6rest, es7rest, es8java, es9java) abort before their first test, in MinioTestKit.start,
with ContainerFetchException "... pull access denied for minio/minio, repository does not
exist or may require 'docker login'". Measured, so it is neither a rate limit nor a login
problem: `docker pull minio/minio:latest` fails that way locally and on the runner, and
https://hub.docker.com/v2/repositories/minio/minio/ answers 404 - the repository itself is
gone (a throttle is 429, an auth failure 401).

Replacement: adobe/s3mock, pinned 5.2.2, driven by a hand-rolled GenericContainer. The
trait keeps its name and its whole public surface, so CopyIntoS3IntegrationSpec and the
four concrete specs need no change. No production source is touched by this branch.

Why not the alternatives, each rejected on measurement:

  * LocalStack - Pro-gated. localstack/localstack:latest (2026.8.2, built 2026-09-09)
    exits code 55 with "License activation failed! ... No credentials were found in the
    environment". It cannot be an unauthenticated CI fixture.
  * The quay.io MinIO mirror - its newest COMMUNITY tag is RELEASE.2025-09-07T16-13-09Z
    (pushed 2025-09-07, and `latest` points at the same build); every newer push is a
    .hotfix.* build. It would pin the suites to a year-old frozen AGPL binary.
  * The s3mock-testcontainers MODULE (rather than GenericContainer) - 5.2.2 is Java-17
    bytecode built against Testcontainers 1.21.3, while this build's default JDK is 11 and
    it pins Testcontainers 2.0.2.

adobe/s3mock declares NO EXPOSE (buildpacks image), so withExposedPorts(9090) is required;
it seeds buckets from COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS, replacing the two
`mc` execInContainer calls. testcontainers-minio gives way to the core testcontainers
artifact in both build files.

The swap silently removes two protections, so both are replaced:

(a) Bucket seeding is silently fallible - Adobe renamed the property between 4.x and 5.x,
    and the old name leaves the container healthy, answering 200, with an EMPTY bucket
    list. start() now probes and fails naming the env var. The obvious probe is VACUOUS
    and that was measured: fs.exists("s3a://<bucket>/") returned true against a container
    with no buckets (S3A answers the ROOT path from a synthetic directory status; the
    fs.s3a.bucket.probe default is 0, so nothing HEADs the bucket). The shipped probe sets
    fs.s3a.bucket.probe=2 and issues a real listStatus, and it distinguishes "bucket
    absent" from "cannot reach the container" so a missing hadoop-aws is not misreported.
    Falsified: with the 4.x env-var name the suite ABORTS with
      IllegalStateException: S3 bucket 'copy-into-test' does not exist on adobe/s3mock:5.2.2
      at http://localhost:62469. The container seeds buckets from the env var
      'COM_ADOBE_TESTING_S3MOCK_INITIALBUCKETS' - check that name: an unknown one is ignored
      silently and leaves the container healthy but empty.

(b) s3mock does not verify credentials AT ALL, where MinIO answered 403. MinioTestKit sets
    AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_ENDPOINT_URL as system properties so
    the production HadoopConfigurationFactory.s3aConf picks them up via envOrProp, and
    nothing else in the repo asserted that. Two replacements, because one is not enough:
      - the bucket probe builds its Configuration from the PRODUCTION forPath, not from
        minioHadoopConf, so the suites really exercise those three properties. Falsified:
        misspelling ONE of them (AWS_ENDPOINT_URL -> AWS_ENDPOINT_URI) aborts the suite with
        "Could not reach the S3 bucket 'copy-into-test' ...".
      - two LocalPathSpec unit tests pin what the properties must produce. Falsified:
        narrowing envOrProp to sys.env only reddens exactly 1 of the 2 ("read credentials
        and endpoint from JVM system properties"); the no-static-key test correctly stays
        green, which is what makes the pair a discriminator rather than a restatement.

Verified: the four *CopyIntoS3Spec suites green on real Elasticsearch 6.8.23 / 7.17.29 /
8.18.3 / 9.0.3 (3 tests each); core/test green; testkit and core compile on 2.13.16 and
2.12.20; CI's lint task list (headerCheck scalafmtSbtCheck scalafmtCheck
test:scalafmtCheck) green run cold.

Follow-up, deliberately not done here: renaming MinioTestKit to S3TestKit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@fupelaqu
fupelaqu marked this pull request as ready for review September 12, 2026 08:37
@fupelaqu
fupelaqu merged commit 464344f into main Sep 12, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

S3 test-fixture image minio/minio was deleted from Docker Hub — all four *CopyIntoS3Spec suites abort

1 participant