improvement: manage tenant databases, so strategy :context needs no tenant_repo of its own

[C-Sinclair]

#224 added the tenant_repo option and no implementation of it. This PR adds the runtime behind it, so that strategy :context needs no tenant repo of your own. That is what you asked for, @zachdaniel: context multitenancy that manages its own storage rather than leaving it to the reader.

children = [
  MyApp.Repo,
  {AshSqlite.MultiTenancy,
   repo: MyApp.Repo,
   dir: "priv/tenants",
   migrations_path: "priv/repo/tenant_migrations"}
]

That is the whole setup. A resource needs only strategy :context. AshSqlite.MultiTenancy.TenantRepo becomes its default tenant_repo, so the compile-time requirement #224 imposes goes away. Naming your own still overrides it, which is what keeps Turso, Litestream or a replica-aware router implementable outside this library.

What's in the box? 📦

• Registry — one connection per tenant, and only one. A second is refused rather than racing.
• Connection — opens the file, migrates it, and refuses to serve until both have happened. A migration that raises stops the connection instead of serving an unmigrated database.
• Manager — activation, an LRU bound on residency that will exceed itself rather than close a tenant with a statement in flight, quarantine for tenants that cannot open, and seal/unseal for draining a node.
• Database — the tenant-to-filename mapping. Escaped rather than sanitised, so it is injective: sanitising maps a:b and a_b onto one file, which quietly puts two tenants in one database. Also why ../../etc/x stays one file inside the directory.
• Migrations — compiles a migration directory once per boot rather than once per tenant.
• TenantRepo — implements with_repo/4 as well as repo/3, so a tenant is held open for the length of each statement and eviction cannot close it underneath one.

Not in this PR

• Documentation. Deliberately none, as in improvement: support strategy :context multitenancy via a tenant repo #224. The guide covering the whole feature is a separate piece of work.
• Tenant migration generation. mix ash_sqlite.generate_migrations has no tenant awareness, so the directory passed as :migrations_path is yours to maintain. AshPostgres routes tenanted resources to priv/repo/tenant_migrations; matching that is not attempted here.
• Mix tasks. No --tenants / --only-tenants / --except-tenants. MultiTenancy.migrate_all/3 is the runtime equivalent.
• Per-tenant rollback. AshPostgres has mix ash_postgres.rollback --tenants. There is no counterpart, by choice. It is forward-only for now.
• Tenant lifecycle DSL. AshPostgres has manage_tenant do ... end. Creation here is implicit, because activation makes the file. rename/3 and delete/2 exist as primitives, but nothing calls them from a resource yet.

[C-Sinclair]

AshPostgres asks the application for this through Repo.all_tenants/0. Here it is derived from the files in :dir, because a tenant is a file.

We also include active residents, because SQLite only creates the file on the first write, so a tenant activated but hasn't written yet would end up being skipped by migrate_all/3.

[C-Sinclair]

Without rename/3, a renamed tenant keeps none of its data: the new name addresses a missing file, and the next request creates an empty one. Both names are marked closing for the whole move, and close/3 returns {:error, :busy} while a statement is in flight, so nothing can write into either file mid-move.

[C-Sinclair]

The sidecars move too, because committed writes sit in the -wal file until a checkpoint. The test "carries the tenant's data to the new name" asserts the -wal file exists before renaming, so it fails if the sidecars are left behind.

[C-Sinclair]

put_dynamic_repo/1 binds one repo module, and the default tenant repo binds the :mutate one. A repo function returning a different module for :read would issue reads unbound, so this fails the compile. A resource naming its own tenant_repo is exempt, because it sees each statement's type and can bind a replica itself.

[zachdaniel]

👋 I feel like this implementation is a lot more complex than we should need.

1. We can start a dynamic supervisor from the repo's child spec based on configuration, so possibly no need for a new supervisor, and read tenants: [dir: "...."] from the repo config so all config is one place.
2. If we just have one process per tenant that owns the lifecycle (i.e a single genserver) then I think we can collapse all of the registry/connection/connection supervisor/binds/manager stuff into a single place? The genserver could stop on idle.
3. If we do that we can get rid of all of the LRU logic. If someone needs a hard limit we can cap it via max_children on the dynamic supervisor.
4. Cut quarantine, as it makes any transient failure prevent a retry from fixing it.
5. Cut seal/unseal logic, I can see a case for it but I think its extra complexity not necessarily needed at least for an implementation, and it could also be implemented at the action level or even more likely at the routing/ingestion layer if you need to drain connections.
6. I'd say we can let them handle deleting, ash_postgres doesn't delete tenant schemas for example. Just too risky to even have that code around. We can have renaming logic on an opt-in basis. And nothing actually calls the rename stuff either, maybe simpler to work the feature set in later?
7. I see we're limiting to a single connection, but with the change to have people raise their repo timeout higher than the busywait, I don't think that is a hard requirement anymore? We could serialize writes and allow parallel reads? Not sure.
8. Couldn't we do something like put @behaviour AshSqlite.TenantRepo on AshSqlite.MultiTenancy. Info.tenant_repo/1 then defaults to {AshSqlite.MultiTenancy, []}
9. Couldn't the registry just be via: {Registry, keys: :unique}
10. MultiTenancy.Database exist only to serve all_tenants which is only for migrate_all. In AshPostgres we provide users a snippet of how to do this themselves considering that its hard to predict what they actually want in practice. If I'm reading the code right, opening a tenant database already migrates it, so can we just get rid of that rigging and let them do it themselves if they need to?
11. MultiTenancy.Migrations could probably be replaced by a few lines of code in the supervisor's init and just stored there.
12. I think a lot of the configuration could be simplified (like an encryption key) by just having the user do something like this in their repo:

defmodule MyApp.Repo do
  use AshSqlite.Repo, otp_app: :my_app

  def init(type, config) do
    if tenant = config[:tenant] do
      super(type, Keyword.put(config, :key, MyApp.Keys.fetch!(tenant)))
    else
      super(type, config)
    end
  end
end

[zachdaniel]

Full transparency, I did not read 100% of this code yet 😓 Mostly dealing w/ the shape of it 😄