feat: full layer deletion flow (MAPCO-7285) by almog8k · Pull Request #107 · MapColonies/overseer

almog8k · 2026-07-08T12:50:48Z

Question	Answer
Bug fix	✖
New feature	✔
Breaking change	✖
Deprecations	✖
Documentation	✖
Tests added	✔
Chore	✖

Related issues: MAPCO-7285

Overview

Implements the full-layer deletion flow in the overseer. A Delete_Layer job with a delete task drives ordered removal of a layer's metadata across every system that references it, then hands tile (and, later, artifact) cleanup off to the Cleaner via a downstream tiles-deletion task.

What this adds

Per-client deletion methods

CatalogClient.deleteRecord — remove the catalog record.
GeoserverClient: unpublishLayer → performs the geoserver unpublish.
PolygonPartsMangerClient.deleteEntities — drop the polygon-parts entities.
MapproxyApiClient.removeLayer — remove the layer from mapproxy (treats a 404 as already-removed / idempotent).

Storage / cache plumbing

Split the S3 bucket config into artifactsBucket and tilesBucket (helm + config), guarded by tilesStorageProvider.
MapproxyApiClient.getLayerCache returns the layer's cache object (directory + bucket) for the configured cache type, or undefined on 404.

DeleteLayerHandler (Delete_Layer / delete)

Runs the four metadata-deletion steps in order, persisting each step's completion flag immediately so a redelivered task resumes from where it failed (does not repeat completed steps).
Resolves the tiles location from the mapproxy cache before any deletion step and persists it into the task params — the deleteFromMapproxy step destroys the cache, so a redelivered task reads the location from its own params instead of re-querying a layer that no longer exists.
Derives the Cleaner path from the mapproxy cache directory (not the catalogId): S3 drops the leading slash (object keys are lstripped by mapproxy), FS drops the first segment (the mapproxy PVC mount) so the Cleaner rejoins its own base path. S3 also carries the bucket (cache bucket_name, falling back to configured tilesBucket).
Creates the downstream tiles-deletion cleaner task once all metadata steps complete, with an idempotency guard against duplicate creation on redelivery.

Wiring

Registered the Delete_Layer job and delete task in the polling config and DI container.
configUtil: extracted per-domain polling-task-type selection into getPollingTaskTypes.

Notes / follow-ups

Artifacts-deletion task is deferred (TODO in the handler): mapproxy's GET /layer does not expose artifacts, so the existence source-of-truth and params shape are unresolved.

Tests

Unit coverage for every new client method, the bucket-config split, getLayerCache, and the full DeleteLayerHandler flow (step ordering + per-step persistence, redelivery skip, FS/S3 path + bucket resolution, cleaner-task idempotency, and unresolvable-cache rejection).
Full unit suite green (228 tests); tsc --noEmit, eslint, and prettier clean.

…PCO-7285)

… (MAPCO-7285)

…ngTaskTypes (MAPCO-7285)

…ace (MAPCO-7285)

…O-7285)

…PCO-7285)

…nd package-lock.json

… handling

…-7285) The tiles-deletion cleaner task now carries the layer's storage path from the mapproxy cache directory instead of the catalogId: - MapproxyApiClient.getS3CacheBucketName -> getLayerCache, returning the full cache (directory + bucket) for the configured cache type, undefined on 404. - DeleteLayerHandler resolves the tiles location from the cache and persists it into the task params before any deletion step, so a redelivered task survives the deleteFromMapproxy step that destroys the cache. S3 drops the leading slash (object keys are lstripped); FS drops the first segment (the mapproxy PVC mount) so the Cleaner can rejoin its own base path. - Refactor the handler for readability: data-driven deletion-step loop, single telemetry object, and extracted toRelativeTilesPath / resolveTilesBucket.

…n tasks

…eoserverClient

CL-SHLOMIKONCHA · 2026-07-13T05:31:13Z

+        activeSpan?.setStatus({ code: SpanStatusCode.OK, message: 'Catalog record deleted successfully' });
+      } catch (err) {
+        if (err instanceof NotFoundError) {
+          this.logger.warn({ msg: 'catalog record not found, treating as already deleted', catalogId });


Suggested change

this.logger.warn({ msg: 'catalog record not found, treating as already deleted', catalogId });

this.logger.warn({ msg: 'catalog record not found, skipping, catalogId });

CL-SHLOMIKONCHA · 2026-07-13T05:32:16Z

+      } catch (err) {
+        if (err instanceof NotFoundError) {
+          this.logger.warn({ msg: 'catalog record not found, treating as already deleted', catalogId });
+          activeSpan?.setStatus({ code: SpanStatusCode.OK, message: 'Catalog record already absent' });


use the same message from above warn log

CL-SHLOMIKONCHA · 2026-07-13T05:42:19Z

+      } catch (err) {
+        if (err instanceof NotFoundError) {
+          // already gone — unpublish is idempotent, a 404 on (re)run is success (§6)
+          this.logger.warn({ msg: 'Layer feature type not found in geoserver, treating as already unpublished', layerName });


Suggested change

this.logger.warn({ msg: 'Layer feature type not found in geoserver, treating as already unpublished', layerName });

this.logger.warn({ msg: 'Layer feature type not found in geoserver, skipping', layerName });

CL-SHLOMIKONCHA · 2026-07-13T05:42:36Z

+        if (err instanceof NotFoundError) {
+          // already gone — unpublish is idempotent, a 404 on (re)run is success (§6)
+          this.logger.warn({ msg: 'Layer feature type not found in geoserver, treating as already unpublished', layerName });
+          activeSpan?.setStatus({ code: SpanStatusCode.OK, message: 'Layer already absent in geoserver' });


use same message from above warn log

CL-SHLOMIKONCHA · 2026-07-13T05:47:51Z

+        activeSpan?.setStatus({ code: SpanStatusCode.OK, message: 'Layer removed successfully from mapproxy' });
+      } catch (err) {
+        if (err instanceof NotFoundError) {
+          this.logger.warn({ msg: 'layer not found in mapproxy, treating as already removed', layerName });


Suggested change

this.logger.warn({ msg: 'layer not found in mapproxy, treating as already removed', layerName });

this.logger.warn({ msg: 'layer not found in mapproxy, skipping', layerName });

CL-SHLOMIKONCHA · 2026-07-13T06:33:19Z

+          activeSpan?.setStatus({ code: SpanStatusCode.OK, message: 'Layer already absent in mapproxy' });
+          return;
+        }
+        if (err instanceof DeleteLayerError) {


those two block of error condition seems to have a similar behavior can we merge it into a single error throw?

CL-SHLOMIKONCHA · 2026-07-13T06:39:11Z

call the getSteps() earlier to reuse it instead duplicated calls

CL-SHLOMIKONCHA · 2026-07-13T06:43:23Z

+  private async resolveTilesLocation(layerName: LayerName): Promise<TilesLocation> {
+    const cache = await this.mapproxyClient.getLayerCache(layerName);
+    const directory = cache?.cache.directory;
+    if (directory === undefined) {


why are we depends on the "directory" to throw LayerCacheNotFoundError?
instead we can check if the cache const is undefined as fetchLayerCache returns undefined in case of no NotFound error from mapproxy-api

CL-SHLOMIKONCHA · 2026-07-13T06:44:25Z

+      throw new LayerCacheNotFoundError(layerName, this.tilesStorageProvider);
+    }
+
+    const path = this.toRelativeTilesPath(directory);


above change will make it irrelevant

CL-SHLOMIKONCHA · 2026-07-13T06:47:28Z

+        ? { storageProvider: this.tilesStorageProvider, bucket: tilesLocation.bucket ?? this.tilesBucketConfig }
+        : { storageProvider: this.tilesStorageProvider };
+
+    const tilesDeletionTask: ICreateTaskBody<DeleteStoredResourcesParams> = {


set the blockDuplication task to truek and get rid of the "tiles-deletion" task exists check on the top of the function

almog8k added 14 commits June 28, 2026 11:15

feat: implementing layer deletion from geoserver

57ea6d0

refactor: rename deleteLayer to unpublishLayer in GeoserverClient (MA…

ccf6e88

…PCO-7285)

feat: add deleteEntities to PolygonPartsMangerClient (MAPCO-7285)

27a9da0

feat: add deleteRecord to CatalogClient (MAPCO-7285)

e2099fa

feat: add removeLayer to MapproxyApiClient (MAPCO-7285)

7f3b301

refactor: split S3 bucket config into artifactsBucket and tilesBucket…

f6c856b

… (MAPCO-7285)

feat: add getS3CacheBucketName to MapproxyApiClient (MAPCO-7285)

3d668f3

refactor: extract polling task type selection by domain into getPolli…

13f9f92

…ngTaskTypes (MAPCO-7285)

feat: add Delete_Layer job and delete task schemas and handler interf…

4a0724a

…ace (MAPCO-7285)

feat: implement DeleteLayerHandler for full layer deletion flow (MAPC…

3669685

…O-7285)

feat: register Delete_Layer job and delete task in polling and DI (MA…

23ba16b

…PCO-7285)

feat: update raster-shared to version 8.3.0-alpha.0 in package.json a…

e3b062b

…nd package-lock.json

feat: update raster-shared to version 8.3.0-alpha.1, improved storage…

58bad6f

… handling

mapcolonies-devops assigned almog8k Jul 8, 2026

almog8k and others added 6 commits July 8, 2026 15:59

Merge branch 'master' into feat/delete-layer-MAPCO-7285

c823889

feat: update dependencies in package-lock.json for improved stability

00d9851

feat: remove duplicate import of NotFoundError in catalogClient.ts

2d45e1a

feat: refactor layer cache retrieval methods in MapproxyApiClient

2bdb9dc

feat: add delete layer job definitions and update config for ingestio…

a31b872

…n tasks

feat: update unpublishLayer method to support recursive deletion in G…

dc18b37

…eoserverClient

CL-SHLOMIKONCHA requested changes Jul 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: full layer deletion flow (MAPCO-7285)#107

feat: full layer deletion flow (MAPCO-7285)#107
almog8k wants to merge 20 commits into
masterfrom
feat/delete-layer-MAPCO-7285

almog8k commented Jul 8, 2026 •

edited

Loading

Uh oh!

CL-SHLOMIKONCHA Jul 13, 2026

Uh oh!

CL-SHLOMIKONCHA Jul 13, 2026

Uh oh!

CL-SHLOMIKONCHA Jul 13, 2026

Uh oh!

CL-SHLOMIKONCHA Jul 13, 2026

Uh oh!

CL-SHLOMIKONCHA Jul 13, 2026

Uh oh!

CL-SHLOMIKONCHA Jul 13, 2026

Uh oh!

CL-SHLOMIKONCHA Jul 13, 2026

Uh oh!

CL-SHLOMIKONCHA Jul 13, 2026

Uh oh!

CL-SHLOMIKONCHA Jul 13, 2026

Uh oh!

CL-SHLOMIKONCHA Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	this.logger.warn({ msg: 'catalog record not found, treating as already deleted', catalogId });
	this.logger.warn({ msg: 'catalog record not found, skipping, catalogId });

	this.logger.warn({ msg: 'Layer feature type not found in geoserver, treating as already unpublished', layerName });
	this.logger.warn({ msg: 'Layer feature type not found in geoserver, skipping', layerName });

	this.logger.warn({ msg: 'layer not found in mapproxy, treating as already removed', layerName });
	this.logger.warn({ msg: 'layer not found in mapproxy, skipping', layerName });

Uh oh!

Conversation

almog8k commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

What this adds

Notes / follow-ups

Tests

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

almog8k commented Jul 8, 2026 •

edited

Loading