Skip to content

feat: Spark custom credential providers for native scans#4309

Draft
mbutrovich wants to merge 14 commits into
apache:mainfrom
mbutrovich:credential_provider
Draft

feat: Spark custom credential providers for native scans#4309
mbutrovich wants to merge 14 commits into
apache:mainfrom
mbutrovich:credential_provider

Conversation

@mbutrovich
Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #.

Rationale for this change

Comet's native scan paths bypass Spark's Hadoop S3A credential infrastructure, so vendors with custom credential mechanisms (per-path STS, vended REST creds, etc.) can't use Comet on those paths. This PR adds a Java SPI vendors implement once, plus the JNI plumbing to call it from native code. Full justification in docs/source/contributor-guide/cloud-credential-providers.md.

What changes are included in this PR?

  • new Java SPI CometCloudCredentialProvider + POJO + ServiceLoader-driven dispatcher in common
  • new Rust JNI bridge implementing object_store::CredentialProvider and reqsign_core::ProvideCredential for the same struct
  • wire bridge into s3.rs::create_store (Parquet/object_store path) and iceberg_scan.rs (Iceberg/opendal path)
  • bump iceberg-rust pin to 83b4595 to pick up reqsign-core 3.0 / modern CustomAwsCredentialLoader API
  • bump testcontainers to 1.21.4 + override docker-java to 3.7.1 (modern daemons need API ≥ 1.40)
  • new contributor + user docs pages

How are these changes tested?

  • JUnit tests for the dispatcher
  • end-to-end Scala suite reading Parquet and Iceberg from Minio through the bridge, asserting the SPI was actually invoked
  • existing ParquetReadFromS3Suite / IcebergReadFromS3Suite updated to pass with the bridge SPI globally registered on the test classpath
  • new suite added to dev/ci/check-suites.py ignore_list (manual, like the existing S3 suites)

@mbutrovich mbutrovich changed the title Credential provider feat: Spark custom credential providers for native scans May 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

2 participants