Problem with /tmp file

[monsad]

Describe the bug
Flux fills up /tmp

To Reproduce
Flux copies the repository at certain time intervals but does not delete files from the tmp directory, causing it to become full.

Expected behavior
Fluxcd should probably clean-up after itself?

Logs
no space left on device"

Flux version: 2.4.0

[stefanprodan]

Fluxcd should probably clean-up after itself?

It does so probably something else is blocking it on the node. Look at source-controller logs for any errors.

[monsad]

No, I checked source-controller logs and I found only 'info' level logs, any errors. So I looks that /tmp is not clean up

[acondrat]

We noticed the same issue. We run the source controller in kubernetes with the /tmp mounted on an emtyDir volume (see spec in this chart).

As far as I can tell every time we run flux reconcile source git mygitrepo a new folder is created in /tmp that looks like /tmp/gitrepository-flux-system-mygitrepo-someid. This folder exists for a brief period (I assume some checksums are compared) and then it gets removed.

The problem is that this removal doesn't always happen. I can't consistently reproduce it but I see some leftover folders that were never deleted. In our case it eventually leads to increased memory usage and the container gets OOMKilled.

I checked the logs but all I have is "info" level messages and no indication of a problem whatsoever.

$ flux  version 
flux: v2.5.1
distribution: flux-2.5.1
helm-controller: v1.2.0
image-automation-controller: v0.40.0
image-reflector-controller: v0.34.0
kustomize-controller: v1.5.1
notification-controller: v1.5.0
source-controller: v1.5.0

[stefanprodan]

I can't explain how would the tmp cleanup fail without having the error logged:

source-controller/internal/controller/gitrepository_controller.go

Lines 279 to 293 in 4aa31dc

	// Create temp dir for Git clone
	tmpDir, err := util.TempDirForObj("", obj)
	if err != nil {
	e := serror.NewGeneric(
	fmt.Errorf("failed to create temporary working directory: %w", err),
	sourcev1.DirCreationFailedReason,
	)
	conditions.MarkTrue(obj, sourcev1.StorageOperationFailedCondition, e.Reason, "%s", e)
	return sreconcile.ResultEmpty, e
	}
	defer func() {
	if err = os.RemoveAll(tmpDir); err != nil {
	ctrl.LoggerFrom(ctx).Error(err, "failed to remove temporary working directory")
	}
	}()

No matter what happens during the reconciliation, we remove the tmp dir and if it fails, we log an error.

[acondrat]

I suspect the cleanup fails if the container gets OOMKilled in the middle of it. After this happens a few times and more garbage is accumulated in tmp the container constantly gets OOMKilled on startup. The workaround for us was to delete the pod so both /tmp and /data (emptyDir) are wiped.
We also allocated more memory and it no longer seems to happen.

[andrewb3000]

We hit this on edge clusters running v1.9. The cleanup is a defer, so it never runs when the process exits abruptly: on a lost leader lease (os.Exit(1)) or an OOM kill. The emptyDir survives the container restart, so each crash leaves the in-flight clones behind. With a few hundred restarts a month, /tmp reached 7.5 GB on one of our nodes and pods around it were evicted for disk pressure.

This is the same leak kustomize-controller had in fluxcd/kustomize-controller#1723, fixed there by purging its tmp dirs at startup (fluxcd/kustomize-controller#1729). I've opened #2176 doing the equivalent here.