Add first-class ActorSnapshot lifecycle APIs

[EItanya]

Summary

Substrate creates a durable checkpoint whenever it suspends an Actor. Make each successful checkpoint a first-class, immutable ActorSnapshot that callers can get, list, tag, and use to initialize another Actor.

The design changed during implementation. Superseded decisions remain crossed out so the history is visible:

• A caller explicitly promotes a suspended Actor's private checkpoint. Every successful SuspendActor automatically creates an ActorSnapshot and exposes it through Actor.latest_snapshot.
• Tags are globally unique aliases. Tags are owned and addressed by Atespace as atespace/name; the same name may exist in different Atespaces.
• Visibility is mutable state on the snapshot. Reuse policy belongs to each tag: ATESPACE permits local reuse and PUBLISHED permits reuse from any Atespace without changing the tag's address.
• Tags are embedded in ActorSnapshot. ActorSnapshotTag is a separate resource and O(1) index entry.
• Deleting an Atespace cascades to its tags. Atespace deletion is blocked while any Actors or ActorSnapshotTags remain, matching existing non-empty Atespace behavior.
• Expose explicit snapshot deletion and protect tagged snapshots from it. Explicit snapshot deletion and automatic retention are deferred until the GC policy is designed.
• CreateActor(source_snapshot) accepts a canonical snapshot reference or a tag. Explicit reuse accepts only a tag reference, ensuring reuse always has an explicit policy.

Motivation

Control planes need to preserve an Actor at a known state and later create an independent Actor from it. Substrate must own the storage layout, compatibility checks, authorization boundary, and stable references rather than requiring callers to copy private storage fields.

Proposed API

Snapshots remain immutable resources:

message ActorSnapshot {
  ResourceMetadata metadata = 1;
  ObjectRef source_actor = 2;
  string source_actor_uid = 3;
  int64 source_actor_version = 4;
  string actor_template_namespace = 5;
  string actor_template_name = 6;
  string actor_template_uid = 7;
  SnapshotContentScope content_scope = 8;
}

Tags are Atespace-owned aliases and retention pins. ATESPACE is intentionally the zero/default scope:

enum ActorSnapshotTagScope {
  ACTOR_SNAPSHOT_TAG_SCOPE_ATESPACE = 0;
  ACTOR_SNAPSHOT_TAG_SCOPE_PUBLISHED = 1;
}

message ActorSnapshotTag {
  ResourceMetadata metadata = 1;
  ObjectRef snapshot = 2;
  ActorSnapshotTagScope scope = 3;
}

message ActorSnapshotRef {
  oneof reference {
    ObjectRef snapshot = 1;
    ObjectRef tag = 2;
  }
}

TagActorSnapshot creates a tag in the source snapshot's Atespace. UpdateActorSnapshotTag publishes or unpublishes it without changing its atespace/name identity. DeleteActorSnapshotTag removes the tag. Tag lookup uses actor-snapshot-tag:<atespace>:<name> and remains O(1).

Publishing changes who may reuse a tag, not who owns it. There is no separate global tag namespace. A higher-level grouping or tiered Atespaces can be considered later if concrete use cases require it.

Suspend and reuse semantics

• Each successful suspension creates exactly one immutable snapshot for that suspension.
• The source Actor version is captured when suspension starts, so snapshot metadata describes the state actually checkpointed.
• CreateActor(source_snapshot) only accepts a tag reference.
• An ATESPACE tag may initialize Actors only in its owning Atespace.
• A PUBLISHED tag may initialize Actors in any Atespace while retaining its original atespace/name address.
• Full and data snapshots currently require the exact source ActorTemplate UID.
• Cloning a template with external volumes is rejected until CSI/provider snapshot support exists; ordinary suspend/resume behavior is unchanged.

Snapshot data is stored under:

<ActorTemplate snapshotsConfig.location>/snapshots/<opaque-id>

The physical path carries no Actor, Atespace, tag, or reuse-policy semantics. Its flat manifest.json records actor identity and snapshot metadata.

Lifecycle and retention

• Deleting a tag removes its retention pin.
• Deleting an Atespace is blocked while any Actors or ActorSnapshotTags remain.
• Atespace deletion does not delete snapshot metadata or bytes in this change.
• Snapshot metadata is retained so a future GC/retention policy can make deletion decisions.
• There is no explicit DeleteActorSnapshot RPC in this version.

Non-goals

• Automatic snapshot retention or garbage collection.
• Tiered Atespaces or another higher-level grouping concept.
• In-place rollback of an existing Actor.
• Portable reuse across different ActorTemplate UIDs.
• Cloning external volumes; future support should use CSI/provider snapshot APIs.
• Cloning node-local pause snapshots.

Expected errors

Condition	gRPC status
Actor, snapshot, or tag does not exist	NOT_FOUND
Target Actor name or requested tag already exists in its Atespace	ALREADY_EXISTS
Atespace still contains Actors or ActorSnapshotTags	FAILED_PRECONDITION
Canonical snapshot reference is supplied to CreateActor	FAILED_PRECONDITION
Tag is created outside the source snapshot's Atespace	FAILED_PRECONDITION
Tag scope does not permit the target Atespace	FAILED_PRECONDITION
Snapshot uses a different ActorTemplate UID	FAILED_PRECONDITION
Target template has external volumes	FAILED_PRECONDITION
Snapshot data is missing or corrupt	DATA_LOSS

Acceptance criteria

• Every successful suspension creates exactly one independently addressable ActorSnapshot and updates Actor.latest_snapshot.
• Snapshot metadata records the Actor version captured when suspension began.
• Tags are independently stored and addressed as atespace/name with O(1) lookup.
• Identical tag names may exist in different Atespaces.
• Publishing a tag permits cross-Atespace reuse without changing its identity.
• Atespace deletion is rejected while Actors or ActorSnapshotTags remain.
• CreateActor accepts only tag references and enforces tag scope and ActorTemplate UID compatibility.
• Actor creation from snapshots rejects templates with external volumes.
• Existing suspend, resume, pause, create, and external-volume lifecycle workflows continue to work.
• Explicit snapshot deletion, automatic retention/GC, and tiered Atespaces remain deferred.

References

• ateapi.proto
• Storage-layout PR #568
• ActorSnapshot lifecycle PR #570

[krisztianfekete]

A note on telemetry:
A FULL snapshot includes whatever OTel SDK (telemetry) identity the workload had at startup, so an Actor created from another Actor's FULL snapshot will resume with telemetry describing it as the source/original Actor. This is similar to golden template snapshots today.

In OTel, this is what Entities will solve.

If an observer cannot reliably obtain one or more identifying attributes, it MUST NOT emit telemetry using that entity type. Instead, it SHOULD:

• delegate to the observer that can supply the full set and treat that observer as the source of truth, or
• emit a different entity type with a set of identifying attributes it can populate reliably.

A cloned actor can't know its own ate.actor.uid, so we should give restore/create-from-snapshot a defined point where the platform reconciles actor identity instead of inheriting the snapshot one. Happy to open an issue to track this on the telemetry side.

[mini-cookie]

这是来自QQ邮箱的自动回复邮件。您好，已收到您的信息，祝您工作愉快

[EItanya]

External-volume snapshots at Actor scale

Substrate should use the CSI controller API through its volume-provider abstraction. When CreateActorSnapshot is called, the provider snapshots each external volume and returns an opaque handle that Substrate stores privately with the ActorSnapshot. Creating an Actor from that snapshot asks the same provider to create new independent volumes from those handles; deleting the ActorSnapshot deletes them after references are gone.

This uses CSI's standard snapshot and restore semantics without requiring a Kubernetes resource for every snapshot. A CSI adapter can call the driver's controller service directly; provider-specific implementations remain possible. Drivers without snapshot/restore support return FAILED_PRECONDITION. For NFS, the upstream CSI driver can implement this as an archive and restore, although the Actor must remain suspended while the archive is captured.

We prefer this over Kubernetes VolumeSnapshot objects because each snapshot would otherwise become a CRD stored in etcd and processed by Kubernetes controllers. At a potential scale of millions of Actors, that would move Actor-scale state back into the Kubernetes control plane, contrary to Substrate's architecture. Kubernetes objects may still provide the small amount of cluster-level driver and credential configuration, while per-Actor snapshot state remains in Substrate's scalable store and the storage backend.

[dberkov]

External-volume snapshots at Actor scale

Michelle Au (@msau42) could you please take a look on the comment above.

[EItanya]

Updated snapshot design decisions

Following discussion with the broader Substrate team:

1. Every successful suspend will automatically create a snapshot. We expect to add an n-x retention/garbage-collection policy later, but retention policy is out of scope for this change.

2. Snapshot data in GCS will be partitioned by atespace using the layout:

   /root/atespace/<atespace_name>/snapshots/<uuid>
   

3. Snapshots will have an ownership/visibility scope with these values:

   • ACTOR (default)
   • ATESPACE
   • GLOBAL

   This is separate from the existing FULL/DATA content scope, so the API should use distinct names for the two concepts.

4. Tagging a snapshot protects it from garbage collection and makes it explicitly referenceable. Untagged snapshots remain eligible for the future retention policy.

[HavenXia]

I might missed the broader discussion - but should we make different GCS bucket based on snapshot visibility scope (e.g. a global bucket and per-atespace buckets).

Also I proposed in the atespace design doc that tagged snapshots and defualt ACTOR scoped snapshots are both partitioned by atespace but under different subdirectory paths, e.g.

• /atespaces/<atespace_name>/actors/<actor_uuid>/snapshots/<snapshot_id>(an actor may have more than one snapshots?)
• /atespaces/<atespace_name>/tags/<tag_name>/

[EItanya]

I might missed the broader discussion - but should we make different GCS bucket based on snapshot visibility scope (e.g. a global bucket and per-atespace buckets).

Also I proposed in the atespace design doc that tagged snapshots and defualt ACTOR scoped snapshots are both partitioned by atespace but under different subdirectory paths, e.g.

• /atespaces/<atespace_name>/actors/<actor_uuid>/snapshots/<snapshot_id>(an actor may have more than one snapshots?)
• /atespaces/<atespace_name>/tags/<tag_name>/

Personally I don’t feel that snapshots need to be stored by any particular scope. The scope is given by substrate itself and the DB entry. We can also keep metadata here if we also want it collocated. Doing it that way ensures we don’t have to worry about delete semantics or actor + atespace for tagged snapshots. Also, the snapshots are an impl detail of substrate, that’s the security boundary for them, not the bucket.

[HavenXia]

That makes sense. With snapshots as first-class resources in our system, the path doesn't need to carry any semantics — permissions and lifecycle all go through the API anyway.

Snapshot data in GCS will be partitioned by atespace using the layout:
/root/atespace/<atespace_name>/snapshots/<uuid>

One followup on the GC side then: a GLOBAL snapshot would still physically sit under some atespace's prefix? So atespace deletion can't be a blind prefix delete — cleanup has to consult the database.

[EItanya]

One followup on the GC side then: a GLOBAL snapshot would still physically sit under some atespace's prefix? So atespace deletion can't be a blind prefix delete — cleanup has to consult the database.

My plan is just to remove the prefix altogether for that exact reason, the atespace doesn't really mean anything anymore, ownership is strictly semantic.

[HavenXia]

One followup on the GC side then: a GLOBAL snapshot would still physically sit under some atespace's prefix? So atespace deletion can't be a blind prefix delete — cleanup has to consult the database.

My plan is just to remove the prefix altogether for that exact reason, the atespace doesn't really mean anything anymore, ownership is strictly semantic.

LGTM, you might want to update the issue posts to reflect that to avoid future reader confusion.

[EItanya]

LGTM, you might want to update the issue posts to reflect that to avoid future reader confusion.

Updated