Skip to content

Add production KEK providers#1113

Open
bootjp wants to merge 13 commits into
design/encryption-stage7-projectionfrom
feature/encryption-stage9-kek-providers
Open

Add production KEK providers#1113
bootjp wants to merge 13 commits into
design/encryption-stage7-projectionfrom
feature/encryption-stage9-kek-providers

Conversation

@bootjp

@bootjp bootjp commented Jul 18, 2026

Copy link
Copy Markdown
Owner

Summary

  • add AWS KMS, GCP KMS, and Vault Transit KEK wrappers
  • add the test/CI environment KEK source with immediate environment removal
  • enforce mutually exclusive file, URI, and environment sources
  • gate startup and encryption mutators on the wrapper that actually loaded
  • document Stage 9B completion

Safety and compatibility

  • no envelope, sidecar, Raft opcode, or snapshot format changes
  • AWS requests bind a fixed encryption context
  • GCP requests bind AAD and verify CRC32C in both directions
  • Vault accepts only versioned Transit ciphertext and validates decoded DEK length
  • all providers reject empty, malformed, or non-32-byte responses before sidecar mutation

Caller audit

  • KEK source resolution has one production caller before startup guards
  • KEKConfigured now derives from the loaded wrapper and reaches both startup refusal and the EncryptionAdmin mutator gate
  • existing bootstrap, rotation, hydration, and startup-rotation Wrap/Unwrap callers continue through the unchanged wrapper interface

Verification

  • go test ./... -count=1 -timeout=20m
  • go test -race ./internal/encryption/kek . -run ... -count=1 -timeout=15m
  • golangci-lint --allow-parallel-runners --config=.golangci.yaml run ./... --timeout=10m (0 issues)
  • environment KEK Bootstrap -> storage cutover -> encrypted write -> snapshot -> restore -> plaintext readback integration
  • git diff --check

Risk

Remote provider availability and credentials are now startup dependencies when those providers are selected. Each remote operation has a 30-second deadline and startup fails closed when existing wrapped DEKs cannot be unwrapped.

Author: bootjp

Summary by CodeRabbit

  • 新機能

    • AWS KMS、GCP KMS、Vault Transit、環境変数によるKEKプロバイダーに対応しました。
    • KEKの入力元としてファイル、URI、環境変数を利用できるようになりました。
    • 複数の入力元が設定された場合は起動を拒否し、安全に失敗します。
    • 暗号化管理機能は、KEKが正常に読み込まれた場合のみ利用可能になりました。
  • 改善

    • KEK設定エラーの案内を拡充しました。
    • 対応状況と残作業に関する設計ドキュメントを更新しました。

@bootjp

bootjp commented Jul 18, 2026

Copy link
Copy Markdown
Owner Author

Stage 9B verification completed on signed head 35ae4873f789b2fb0b73fea2c6d211ca92012d64.

  • full Go suite passed, including adapter, raftengine, and store integration suites
  • focused provider/startup race tests passed
  • full repository lint reports 0 issues
  • environment KEK production wiring completed Bootstrap -> storage cutover -> encrypted write -> snapshot -> restore -> readback
  • source, startup guard, mutator gate, and all Wrap/Unwrap callers were audited
  • no persisted format or wire protocol changed

@codex review
@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 18, 2026

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c5712dae-022c-4841-8c90-a463592773e1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Stage 9BとしてAWS KMS、GCP KMS、Vault Transit、環境変数のKEKプロバイダとソース選択を追加し、起動時のKEKロード状態を暗号化ミューテータのゲートへ接続しました。関連する依存関係、テスト、設計文書も更新されています。

Changes

KEKプロバイダとソース解決

Layer / File(s) Summary
共通契約とソース解決
internal/encryption/kek/provider.go, internal/encryption/kek/env.go, internal/encryption/kek/source.go, internal/encryption/kek/*_test.go
DEK検証、要求タイムアウト、AES-GCM環境プロバイダ、ファイル・URI・環境変数の相互排他、URI別プロバイダ生成を追加しました。
リモートKEKプロバイダ
internal/encryption/kek/aws_kms.go, internal/encryption/kek/gcp_kms.go, internal/encryption/kek/vault.go, internal/encryption/kek/*_test.go, go.mod
AWS KMS、GCP KMS、Vault TransitによるWrap/Unwrap、暗号化コンテキスト、AAD、CRC32C、応答形式の検証を追加しました。
起動ロードとミューテータゲート
main.go, main_encryption_admin.go, internal/encryption/errors.go, internal/encryption/startup.go, main_encryption_kek_source_test.go
ロード済みKEKラッパを起動ガードとEncryptionAdminミューテータ判定へ伝播し、環境KEKのブートストラップ・ローテーション・スナップショット復元を検証しました。
Stage 9B設計文書
docs/design/*
Stage 9Bの完了項目、KEKソース仕様、相互排他、ゲート条件、残作業を記載しました。

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Startup
  participant KEKSourceLoader
  participant KEKProvider
  participant EncryptionAdmin
  Startup->>KEKSourceLoader: file、URI、環境変数からKEKをロード
  KEKSourceLoader->>KEKProvider: provider wrapperを生成
  KEKProvider-->>KEKSourceLoader: ロード済みwrapper
  KEKSourceLoader-->>Startup: kekConfiguredを返す
  Startup->>EncryptionAdmin: loaded KEK状態でmutator gateを設定
Loading

Possibly related PRs

  • bootjp/elastickv#719: KEK Wrapperインターフェースと暗号化基盤を提供しています。
  • bootjp/elastickv#760: EncryptionAdminミューテータの有効化経路を変更しています。
  • bootjp/elastickv#768: KEKラッパとソースロードを利用する暗号化Applierを実装しています。
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 23.08% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed AWS KMS、GCP KMS、Vault Transit などの本番向け KEK プロバイダ追加を要約しており、変更内容と一致しています。

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai

coderabbitai Bot commented Jul 18, 2026

Copy link
Copy Markdown

@bootjp Thanks for the detailed Stage 9B verification report. I’ll review the changes on the signed head, including the provider implementations, source-selection invariants, startup/mutator gates, and compatibility boundaries.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements Stage 9B of the data-at-rest encryption design, adding support for AWS KMS, GCP KMS, Vault Transit, and a test/CI-only environment-backed KEK provider. It updates the startup wiring and mutator gates to use the loaded KEK wrapper state instead of relying solely on the legacy file flag. Feedback is provided regarding a performance improvement in the GCP KMS wrapper, suggesting that the Castagnoli CRC32 table be defined as a package-level global variable to avoid recreating it on every checksum calculation.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread internal/encryption/kek/gcp_kms.go
@bootjp

bootjp commented Jul 18, 2026

Copy link
Copy Markdown
Owner Author

Latest signed head 68bcf150 addresses the current-head GCP KMS CRC32C allocation finding by reusing one immutable Castagnoli table. Focused provider tests passed and the commit lint hook reports 0 issues. No semantic or wire-format behavior changed. @codex review @coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 18, 2026

Copy link
Copy Markdown

@bootjp, I’ll review the latest signed head, with particular attention to the GCP KMS CRC32C table reuse and preservation of provider behavior.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
internal/encryption/kek/vault_test.go (1)

52-56: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

サブテスト(t.Run)を使用してテーブル駆動テストの可読性を向上させる

コーディングガイドラインに従い、テーブル駆動テストの反復処理を t.Run でラップすることをお勧めします。

♻️ 提案するリファクタリング
 	for _, target := range []string{"", "transit", "transit//key", "../key", "transit/../key"} {
-		_, _, err := parseVaultTarget(target)
-		require.ErrorIsf(t, err, ErrInvalidKEKURI, "target=%q", target)
+		t.Run(target, func(t *testing.T) {
+			_, _, err := parseVaultTarget(target)
+			require.ErrorIsf(t, err, ErrInvalidKEKURI, "target=%q", target)
+		})
 	}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/encryption/kek/vault_test.go` around lines 52 - 56, Wrap each
invalid target iteration in the table-driven test around parseVaultTarget with
t.Run, using the target value to identify the subtest. Keep the existing
ErrInvalidKEKURI assertion and target-specific failure context unchanged.

Source: Coding guidelines

internal/encryption/kek/source_test.go (1)

48-56: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

サブテスト(t.Run)を使用してテーブル駆動テストの可読性を向上させる

コーディングガイドラインに従い、テーブル駆動テストの各ケースを t.Run でラップすることで、テスト失敗時の出力が明確になり、個別のケースを独立して実行できるようになります。

♻️ 提案するリファクタリング
 	for _, uri := range []string{
 		"unknown://key",
 		"aws-kms://not-an-arn",
 		"gcp-kms://projects/incomplete",
 		"vault-transit://transit",
 	} {
-		_, err := newURIWrapper(context.Background(), uri)
-		require.ErrorIsf(t, err, ErrInvalidKEKURI, "uri=%q", uri)
+		t.Run(uri, func(t *testing.T) {
+			_, err := newURIWrapper(context.Background(), uri)
+			require.ErrorIsf(t, err, ErrInvalidKEKURI, "uri=%q", uri)
+		})
 	}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/encryption/kek/source_test.go` around lines 48 - 56, Wrap each URI
iteration in the table-driven test with t.Run, using the URI as the subtest
name, and move the newURIWrapper and ErrInvalidKEKURI assertion into that
subtest so failures identify the specific invalid URI and cases can run
independently.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/encryption/kek/aws_kms.go`:
- Around line 33-36:
更新処理のARN検証で、parsed.Resourceが「key/」または「alias/」だけの場合もErrInvalidKEKURIを返すよう、各接頭辞の後に空でない識別子が存在することを確認してください。既存のAWS
KMS ARN検証条件は維持し、aws_kms_test.goの検証ケースに空のkey/およびalias/リソースを追加してください。

---

Nitpick comments:
In `@internal/encryption/kek/source_test.go`:
- Around line 48-56: Wrap each URI iteration in the table-driven test with
t.Run, using the URI as the subtest name, and move the newURIWrapper and
ErrInvalidKEKURI assertion into that subtest so failures identify the specific
invalid URI and cases can run independently.

In `@internal/encryption/kek/vault_test.go`:
- Around line 52-56: Wrap each invalid target iteration in the table-driven test
around parseVaultTarget with t.Run, using the target value to identify the
subtest. Keep the existing ErrInvalidKEKURI assertion and target-specific
failure context unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 7cb05f6a-f289-4958-92a5-ab6484aef3a4

📥 Commits

Reviewing files that changed from the base of the PR and between 679d83f and 35ae487.

⛔ Files ignored due to path filters (1)
  • go.sum is excluded by !**/*.sum
📒 Files selected for processing (20)
  • docs/design/2026_04_29_partial_data_at_rest_encryption.md
  • docs/design/2026_07_18_implemented_9b_kek_providers.md
  • go.mod
  • internal/encryption/errors.go
  • internal/encryption/kek/aws_kms.go
  • internal/encryption/kek/aws_kms_test.go
  • internal/encryption/kek/env.go
  • internal/encryption/kek/env_test.go
  • internal/encryption/kek/gcp_kms.go
  • internal/encryption/kek/gcp_kms_test.go
  • internal/encryption/kek/kek.go
  • internal/encryption/kek/provider.go
  • internal/encryption/kek/source.go
  • internal/encryption/kek/source_test.go
  • internal/encryption/kek/vault.go
  • internal/encryption/kek/vault_test.go
  • internal/encryption/startup.go
  • main.go
  • main_encryption_admin.go
  • main_encryption_kek_source_test.go

Comment thread internal/encryption/kek/aws_kms.go

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 68bcf1500b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread main.go
Comment thread internal/encryption/kek/aws_kms.go Outdated
Comment thread internal/encryption/kek/vault.go Outdated
Base automatically changed from feature/encryption-stage9-compression to design/encryption-stage7-projection July 19, 2026 06:04
@bootjp

bootjp commented Jul 19, 2026

Copy link
Copy Markdown
Owner Author

Signed head 6d59ccc addresses all current review findings. KEK providers now receive a real startup wrap and unwrap preflight before encryption mutators are exposed; AWS accepts only immutable non-empty key ARNs; Vault validates a positive version and non-empty ciphertext payload; and the two table-driven test loops now use named subtests. Full go test ./..., race tests for the encryption packages and root package, targeted golangci-lint, the full commit lint hook, and git diff --check passed. Startup, mutator-gate, Wrap, and Unwrap callers were audited; no persisted or wire format changed. @codex review @coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 19, 2026

Copy link
Copy Markdown

@bootjp, I’ll review signed head 6d59ccc6, with particular attention to the startup Wrap/Unwrap preflight, AWS ARN and Vault ciphertext validation, mutator exposure gating, and the stated format-compatibility boundary.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 6d59ccc6ce

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@bootjp

bootjp commented Jul 19, 2026

Copy link
Copy Markdown
Owner Author

Signed head 78f24a95 fixes the failing Go race CI without changing the production cache bound. TestLuaNegativeTypeCache_BoundedSize now injects a four-entry test limit while production and zero-value contexts retain the 10,000-entry default. The focused race test completed in 3.9s, targeted adapter lint reports 0 issues, git diff --check passed, and all newLuaScriptContext construction paths were audited. @codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 78f24a9504

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@bootjp
bootjp force-pushed the design/encryption-stage7-projection branch from b3acb83 to 39142ba Compare July 19, 2026 11:59
@bootjp
bootjp force-pushed the feature/encryption-stage9-kek-providers branch from 78f24a9 to 011aeae Compare July 19, 2026 12:38
@bootjp

bootjp commented Jul 19, 2026

Copy link
Copy Markdown
Owner Author

Rebased the four Stage 9B commits onto the sanitized #1058 head 39142ba0 and removed the conflict.

Evidence:

  • git range-diff 679d83fd..78f24a95 39142ba0..011aeaeb maps all four commits in order; commits 2-4 are patch-identical, and commit 1 differs only in current-base dependency/toolchain versions plus the current-base recovery wiring context
  • GitHub compare reports all four PR-visible commits as verified=true with author and committer bootjp <contact@bootjp.me>
  • go test ./internal/encryption/kek -count=1 -timeout=10m
  • focused root/adapter/provider tests passed
  • golangci-lint run . ./internal/encryption/kek ./adapter --timeout=10m --allow-parallel-runners: 0 issues
  • PR is MERGEABLE; latest-head CI is running

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 011aeaebad

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/encryption/kek/gcp_kms.go
@bootjp

bootjp commented Jul 19, 2026

Copy link
Copy Markdown
Owner Author

Latest signed head 011aeaeb remains unchanged. The sole current-head P2 was invalid for the generated GCP KMS v1 API: kmspb.DecryptResponse has no request-verification flags, while RPC errors and returned plaintext CRC mismatches already fail closed. The root has the protobuf/schema and caller-audit evidence and is resolved. All CI checks are green and the PR is CLEAN/MERGEABLE. @codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 011aeaebad

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/encryption/kek/vault.go Outdated
Comment thread internal/encryption/kek/vault.go Outdated
Comment thread internal/encryption/kek/vault.go Outdated
Comment thread docs/design/2026_04_29_partial_data_at_rest_encryption.md Outdated
@bootjp

bootjp commented Jul 19, 2026

Copy link
Copy Markdown
Owner Author

Signed head 0eb8c70f addresses all current-head findings. Vault Transit now verifies that the configured key exists before encrypt, preserves nested mount paths by splitting at the final slash, and binds encrypt/decrypt to fixed AAD. The GCP CryptoKey URI example now matches the parser. Provider/root tests, provider race tests, full lint, the commit hook, and diff checks passed; Wrap/source/startup callers were audited and failures remain before Raft proposal. GitHub reports the commit signature valid for bootjp contact@bootjp.me. @codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 0eb8c70fd5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@bootjp
bootjp force-pushed the design/encryption-stage7-projection branch from 39142ba to 04bfde5 Compare July 19, 2026 14:01
@bootjp
bootjp force-pushed the feature/encryption-stage9-kek-providers branch from 0eb8c70 to bda3501 Compare July 19, 2026 14:29
@bootjp

bootjp commented Jul 19, 2026

Copy link
Copy Markdown
Owner Author

Rebased the five signed Stage 9B commits onto current #1058 head 04bfde58. git range-diff 39142ba0..backup/pr1113-0eb8c70f 04bfde58..bda35013 maps all five commits as patch-identical. Focused provider/root/adapter tests passed, targeted lint reports 0 issues, go mod tidy -diff and diff checks are clean. GitHub compare reports all five PR-visible commits with valid signatures and author/committer bootjp contact@bootjp.me. @codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: bda35013b1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@bootjp
bootjp force-pushed the design/encryption-stage7-projection branch from 04bfde5 to e34b4de Compare July 19, 2026 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant