[Feature] KFP pipeline artifacts remain on in-cluster SeaweedFS while syncing to external AWS S3 or Azure Blob in the background by shay79il · Pull Request #311 · mlrun/ce

shay79il · 2026-08-18T14:08:21Z

📝 Description

Adds SeaweedFS remote gateway support so KFP pipeline artifacts remain on in-cluster SeaweedFS (pipelines.storage.mode: local) while syncing to external AWS S3 or Azure Blob in the background.

This replaces the previous model where KFP talked directly to external object storage via pipelines.storage.mode: s3 / azure-blob. MLRun and Jupyter storage (storage.mode) is unchanged and independent.

🛠️ Changes Made

New SeaweedFS remote components (seaweedfs.remote.enabled):
- seaweedfs-remote-secret.yaml — cloud credentials (S3 keys from storage.s3.*, Azure from storage.azure.*)
- seaweedfs-remote-config-job.yaml — Helm hook Job that runs remote.configure + remote.mount on the filer
- seaweedfs-remote-gateway-deployment.yaml — filer.remote.gateway syncs local bucket → remote backend
New example overlays: examples/seaweedfs-remote-s3-overlay.yaml, examples/seaweedfs-remote-azure-overlay.yaml
Storage validation (storage-validation.yaml):
- Requires pipelines.storage.mode: local (rejects direct external KFP modes)
- Validates seaweedfs.remote.* (provider, bucket, remote name charset, S3 endpoint, credentials)
Helpers refactor (_helpers.tpl):
- mlrun-ce.pipelines.storage.mode hardcoded to local
- Pipeline S3 settings delegate to in-cluster SeaweedFS
- New remote-gateway helpers
KFP templates simplified for local-only object store:
- Removed external providers block from kfp-launcher.yaml
- Removed conditional OBJECTSTORECONFIG_REGION from ml-pipeline.yaml
- Renamed secret template: mlpipeline-seaweedfs-artifact.yaml → mlpipeline-artifact-secret.yaml
MySQL (mysql.yaml): auth plugin flag selected by image tag — --default-authentication-plugin for 8.0.x, --mysql-native-password=ON for 8.4+
Jupyter (deployment.yaml + values.yaml): added startupProbe (/api); readiness probe path changed from /lab to /api for slow cold starts
Values: new seaweedfs.remote block (default enabled: false); added to values.yaml and all three install-mode values files
Chart version: 0.12.0-rc.11

✅ Checklist

🧪 Testing

Please see

charts/mlrun-ce/examples/seaweedfs-remote-azure-overlay.yaml
charts/mlrun-ce/examples/seaweedfs-remote-s3-overlay.yaml

GiladShapira94 · 2026-09-08T07:51:32Z

+    {{- include "mlrun-ce.pipelines.labels" . | nindent 4 }}
+type: Opaque
+stringData:
+  accesskey: {{ include "mlrun-ce.seaweedfs.s3.accessKey" . | quote }}


From the helper, it looks like it is using MLRun configuration. Can you please verify that you can point MLRun to local but pipelines to remote

@shay79il this one still needs to be resolved - you are pointing to mlrun conf and not pipelines creds

{{- define "mlrun-ce.seaweedfs.s3.accessKey" -}} {{- .Values.storage.local.accessKey -}} {{- end -}}

Please try running your code and store MLRun to S3 bucket x with x creds and pipelines to S3 bucket y with different creds.
Other then that look good

royischoss

LGTM

…dation for SeaweedFS remote configuration

… directory

…d, detailing local and remote storage options with configuration examples

… configuration; enhance validation checks for Azure provider

…etails; refactor static credentials usage in templates

… disable remote storage

…guration details; add nonempty option in values.yaml for handling existing pipeline data during upgrades

…structions; update values.yaml for Azure storage configuration; enhance validation checks for Azure provider in templates

…rom values files and templates; update README.md to clarify SeaweedFS usage for pipeline artifacts.

…nfiguration; enhance validation checks for SeaweedFS remote provider in templates.

… by breaking down the command into multiple lines for better readability and maintainability.

… configuration; enhance Azure Blob example and clarify usage in templates.

…te overlay examples for Azure and S3; streamline configuration for clarity and consistency.

… remote overlay example for improved compatibility with S3 storage.

… configuration; clarify usage of mount options and enhance Azure Blob example with command syntax.

…nd S3; clarify usage of emptyDir for lab/dev testing and provide verification steps for remote sync.

Copilot

Copilot review overview

🟡 Changes recommended

Critical IAM credential and mountExisting: false paths can fail, with additional validation gaps.

Review effort: Lite
Findings: 4 · 1

Open (5)

Support IAM mode without requiring S3 credentials · New
Handle gateway startup when mountExisting is false · New
Make gateway init mount wait conditional · New
Allow IAM credentials without required S3 keys · New
Update version matrix to Pipelines 2.16.0 · New

What changed in this PR

Adds optional SeaweedFS remote-gateway synchronization for KFP artifacts stored locally while syncing to AWS S3 or Azure Blob.

Changes:

Adds remote gateway configuration, credentials, mounting, deployment, and examples.
Simplifies KFP storage around in-cluster SeaweedFS and updates component configuration.
Adds storage validation, MySQL compatibility handling, documentation, and a chart version bump.

File	Summary
`charts/mlrun-ce/values.yaml`	Adds remote gateway settings and updated KFP values.
`charts/mlrun-ce/templates/seaweedfs/seaweedfs-remote-secret.yaml`	Creates remote credential Secret.
`charts/mlrun-ce/templates/seaweedfs/seaweedfs-remote-gateway-deployment.yaml`	Runs background remote synchronization.
`charts/mlrun-ce/templates/seaweedfs/seaweedfs-remote-config-job.yaml`	Configures and mounts remote storage.
`charts/mlrun-ce/templates/seaweedfs/s3-bucket-init-job.yaml`	Updates bucket initialization behavior.
`charts/mlrun-ce/templates/pipelines/secrets/mlpipeline-seaweedfs-artifact.yaml`	Removes the replaced artifact Secret template.
`charts/mlrun-ce/templates/pipelines/secrets/mlpipeline-artifact-secret.yaml`	Defines the replacement artifact Secret.
`charts/mlrun-ce/templates/pipelines/deployments/mysql.yaml`	Selects MySQL authentication flags by version.
`charts/mlrun-ce/templates/pipelines/deployments/ml-pipeline.yaml`	Configures KFP for local SeaweedFS storage.
`charts/mlrun-ce/templates/pipelines/deployments/ml-pipeline-ui.yaml`	Updates SeaweedFS service references.
`charts/mlrun-ce/templates/pipelines/configmaps/workflow-controller-configmap.yaml`	Points workflow artifacts to SeaweedFS.
`charts/mlrun-ce/templates/pipelines/configmaps/pipeline-install-config.yaml`	Updates local object-store settings.
`charts/mlrun-ce/templates/pipelines/configmaps/kfp-launcher.yaml`	Sets the local pipeline root.
`charts/mlrun-ce/templates/config/storage-validation.yaml`	Validates remote storage configuration.
`charts/mlrun-ce/templates/_helpers.tpl`	Adds storage and gateway helpers.
`charts/mlrun-ce/README.md`	Documents remote artifact storage.
`charts/mlrun-ce/examples/seaweedfs-remote-s3-overlay.yaml`	Adds an AWS S3 overlay example.
`charts/mlrun-ce/examples/seaweedfs-remote-azure-overlay.yaml`	Adds an Azure Blob overlay example.
`charts/mlrun-ce/examples/README.md`	Documents deployment and verification steps.
`charts/mlrun-ce/Chart.yaml`	Bumps the chart version.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+{{- if and (eq (include "mlrun-ce.seaweedfs.remote.enabled" . | toString) "true") (eq .Values.seaweedfs.remote.provider "s3") (not .Values.global.infrastructure.aws.s3NonAnonymous) (not .Values.storage.s3.accessKey) }}
+{{ fail "seaweedfs.remote with provider \"s3\" requires storage.s3.accessKey." }}
+{{- end }}
+{{- if and (eq (include "mlrun-ce.seaweedfs.remote.enabled" . | toString) "true") (eq .Values.seaweedfs.remote.provider "s3") (not .Values.global.infrastructure.aws.s3NonAnonymous) (not .Values.storage.s3.secretKey) }}
+{{ fail "seaweedfs.remote with provider \"s3\" requires storage.s3.secretKey." }}
+{{- end }}


+                if [ "${MOUNT_EXISTING}" = "true" ]; then
+                  MOUNT_CMD="remote.mount -dir=${MOUNT_DIR} -remote=${REMOTE_PATH}"
+                  if [ "${MOUNT_NONEMPTY}" = "true" ]; then
+                    MOUNT_CMD="${MOUNT_CMD} -nonempty"
+                  fi
+                  echo "${MOUNT_CMD}"
+                fi


+        - name: wait-for-remote-mount
+          image: {{ include "mlrun-ce.seaweedfs.image" . | quote }}
+          imagePullPolicy: IfNotPresent
+          command:
+            - /bin/sh
+            - -ec
+            - |
+              FILER="{{ include "mlrun-ce.seaweedfs.filerAddress" . }}"
+              MASTER="{{ include "mlrun-ce.seaweedfs.masterAddress" . }}"
+              MOUNT_DIR="/buckets/{{ include "mlrun-ce.seaweedfs.remote.localBucket" . }}"
+              until printf 'remote.mount\n' | weed shell -filer="${FILER}" -master="${MASTER}" 2>&1 | grep -Fq "${MOUNT_DIR}"; do
+                echo "waiting for remote mount at ${MOUNT_DIR}..."
+                sleep 5
+              done


+  AWS_ACCESS_KEY_ID: {{ required "storage.s3.accessKey is required for seaweedfs.remote provider s3" .Values.storage.s3.accessKey | quote }}
+  AWS_SECRET_ACCESS_KEY: {{ required "storage.s3.secretKey is required for seaweedfs.remote provider s3" .Values.storage.s3.secretKey | quote }}


+
+| MLRun CE         | MLRun       | Nuclio  | Jupyter     | MPI Operator | SeaweedFS | Spark Operator | Pipelines | Kube-Prometheus-Stack | OpenTelemetry Operator |
+| ---------------- | ----------- | ------- | ----------- | ------------ | --------- | -------------- | --------- | --------------------- | ---------------------- |
+| **0.11.0**       | 1.11.0      | 1.15.27 | 4.5.0       | 0.2.3        | 4.17.0    | 2.1.0          | 2.15.0    | 72.1.1                | 0.78.1                 |


shay79il requested review from GiladShapira94 and tal-haim August 18, 2026 14:08

github-actions Bot added the feature label Aug 18, 2026

shay79il changed the title ~~[Feature] Bump chart version to 0.12.0-rc.11 and enhance storage validation for SeaweedFS remote configuration~~ [Feature] KFP pipeline artifacts remain on in-cluster SeaweedFS while syncing to external AWS S3 or Azure Blob in the background Aug 18, 2026

shay79il force-pushed the CEML-713 branch from 3ae7044 to 3448c80 Compare August 18, 2026 15:13

GiladShapira94 requested changes Aug 20, 2026

View reviewed changes

shay79il force-pushed the CEML-713 branch 3 times, most recently from 521a6bc to d9ecae2 Compare August 26, 2026 19:50

shay79il requested a review from GiladShapira94 September 1, 2026 08:26

GiladShapira94 requested changes Sep 1, 2026

View reviewed changes

GiladShapira94 reviewed Sep 1, 2026

View reviewed changes

Comment thread charts/mlrun-ce/templates/seaweedfs/seaweedfs-remote-gateway-deployment.yaml

shay79il requested a review from GiladShapira94 September 2, 2026 07:51

GiladShapira94 reviewed Sep 2, 2026

View reviewed changes

Comment thread charts/mlrun-ce/examples/README.md Outdated

GiladShapira94 reviewed Sep 2, 2026

View reviewed changes

Comment thread .claude/skills/pr/SKILL.md Outdated

shay79il force-pushed the CEML-713 branch 2 times, most recently from f65589e to 660543b Compare September 7, 2026 11:03

shay79il requested a review from GiladShapira94 September 7, 2026 12:01

GiladShapira94 reviewed Sep 8, 2026

View reviewed changes

shay79il requested a review from GiladShapira94 September 22, 2026 12:10

royischoss approved these changes Sep 23, 2026

View reviewed changes

shay79il added 9 commits September 23, 2026 18:33

[Feature] Bump chart version to 0.12.0-rc.11 and enhance storage vali…

f605d3d

…dation for SeaweedFS remote configuration

[Cleanup] Remove obsolete Nuclio chart version 0.21.27 from tmpcharts…

24af9af

… directory

[Documentation] Add KFP Pipeline Artifact Storage section to README.m…

2721a85

…d, detailing local and remote storage options with configuration examples

[Documentation] Update README.md and values.yaml for SeaweedFS remote…

474f69c

… configuration; enhance validation checks for Azure provider

[Documentation] Update README.md for SeaweedFS remote configuration d…

6414dce

…etails; refactor static credentials usage in templates

[Feature] Add remote configuration option in values.yaml to enable or…

f65fa5d

… disable remote storage

[Documentation] Enhance README.md with updated SeaweedFS remote confi…

3211a00

…guration details; add nonempty option in values.yaml for handling existing pipeline data during upgrades

[Documentation] Revise README.md to remove local SeaweedFS overlay in…

03861f7

…structions; update values.yaml for Azure storage configuration; enhance validation checks for Azure provider in templates

[Cleanup] Remove local storage mode configuration for KFP pipelines f…

8e3980b

…rom values files and templates; update README.md to clarify SeaweedFS usage for pipeline artifacts.

shay79il added 7 commits September 23, 2026 18:33

[Documentation] Update README.md and values.yaml for Azure storage co…

b73f4b0

…nfiguration; enhance validation checks for SeaweedFS remote provider in templates.

[Refactor] Improve SeaweedFS remote configuration script in templates…

b8d290d

… by breaking down the command into multiple lines for better readability and maintainability.

[Documentation] Update README.md and values.yaml for SeaweedFS remote…

0c6bb6c

… configuration; enhance Azure Blob example and clarify usage in templates.

[Cleanup] Remove local storage mode configuration from SeaweedFS remo…

0a5ea66

…te overlay examples for Azure and S3; streamline configuration for clarity and consistency.

[Update] Add 'forcePathStyle' option to SeaweedFS S3 configuration in…

85d26f3

… remote overlay example for improved compatibility with S3 storage.

[Documentation] Update README.md and values.yaml for SeaweedFS remote…

d485002

… configuration; clarify usage of mount options and enhance Azure Blob example with command syntax.

[Documentation] Enhance SeaweedFS remote overlay examples for Azure a…

498967b

…nd S3; clarify usage of emptyDir for lab/dev testing and provide verification steps for remote sync.

shay79il force-pushed the CEML-713 branch from 8f65a4d to 498967b Compare September 23, 2026 15:33

GiladShapira94 requested a lite review from Copilot September 27, 2026 08:44

Copilot started reviewing on behalf of GiladShapira94 September 27, 2026 08:45 View session

Copilot AI reviewed Sep 27, 2026

View reviewed changes

		AWS_ACCESS_KEY_ID: {{ required "storage.s3.accessKey is required for seaweedfs.remote provider s3" .Values.storage.s3.accessKey \| quote }}
		AWS_SECRET_ACCESS_KEY: {{ required "storage.s3.secretKey is required for seaweedfs.remote provider s3" .Values.storage.s3.secretKey \| quote }}

Conversation

shay79il commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Description

🛠️ Changes Made

✅ Checklist

🧪 Testing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GiladShapira94 Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

GiladShapira94 Sep 27, 2026

Choose a reason for hiding this comment

Uh oh!

royischoss left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shay79il commented Aug 18, 2026 •

edited

Loading