Point production at its SQLite database and check the API in health checks

[ale210]

Third of four PRs for #166, after #186 (Cognito import) and #187 (production service). It corrects how the new service is configured. It still takes no traffic — the cutover is the next PR.

Production does not use RDS, and never has

#187 set DATABASE_URL from the homeuniteus-rds-password secret, following the abandoned Terragrunt state, which describes a Postgres database on the shared instance. The service came up and its API died immediately:

FATAL: password authentication failed for user "homeuniteus"
ERROR: Application startup failed. Exiting.

The credential is not stale — the premise was wrong. The live production container's .env points DATABASE_URL at /opt/huu/homeuniteus.db, a SQLite file on the container's own filesystem. Nothing has connected to that Postgres database since January 2025, which is why the password no longer matches anything.

This PR sets DATABASE_URL to the same SQLite path so the new service matches production, and removes the Postgres wiring: the homeuniteus-rds-password data source, the aws_db_instance lookup, and the db_url_prod SSM parameter, which was created holding a value that never worked.

What that means, recorded here deliberately

The production task definition declares no volume and no mount point. So that SQLite file is created empty when a task starts and is destroyed when the task is replaced — by a deploy, a host drain, or an AZ rebalance. The log group holds 743 task generations. Application data has never survived any of them.

It goes unnoticed because Cognito holds the user accounts and Cognito is durable — 96 of them. The site loads, sign-in works, and only the application's own tables reset. It fails in the one way that looks fine from outside.

The current file was inspected before writing this: 53,248 bytes, containing nothing beyond what the application recreates for itself on startup via create_all and the after_create role seed. So there is nothing to migrate, and nothing was lost in arriving at this PR.

This PR reproduces that behaviour rather than fixing it, so that the cutover changes nothing about how the application runs. Making production durable is a real change to a live service and belongs in its own piece of work, not inside a migration. The comment in environment-prod.tf says so at the point where the next reader will meet it.

Health checks now test the API

The target group checked /, which nginx serves whether or not the API is alive — the container runs nginx in the foreground and the API as a background process, so the API dying does not even stop the task. When #187's service came up with a dead API, the target group reported healthy the entire time.

It now checks /api/health/, which is served by the API process and returns "UP". Had this been in place, #187 would have failed visibly instead of looking green.

LOG_LEVEL restored to DEBUG

#187 set INFO; production runs DEBUG. Reverted to match, keeping this migration behaviour-preserving. Whether DEBUG is appropriate for production is a separate question and not settled here.

What to check in the plan

A new task definition revision and a service update to use it, an in-place update to the target group's health check, and the destruction of the unused db_url_prod SSM parameter and its random_password. Nothing else should change, and no other resource should be replaced.

Verification

• terraform validate and terraform fmt pass on 1.16.0.
• After this merges, confirm from the task's CloudWatch logs that the API reaches startup without the SQLAlchemy error — not from the target group, which is what misled us last time. With the new health check path a failed start should also surface as an unhealthy target.
• Production is unaffected either way: rule 16 still routes all traffic to the old service.

[github-actions]

Terraform plan in terraform
With backend config files: terraform/prod.backend.tfvars

Plan: 1 to add, 2 to change, 3 to destroy.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place
-   destroy
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # module.home-unite-us.module.db_url_prod.aws_ssm_parameter.this will be destroyed
  # (because aws_ssm_parameter.this is not in configuration)
-   resource "aws_ssm_parameter" "this" {
-       arn             = "arn:aws:ssm:us-west-2:035866691871:parameter/home-unite-us/fullstack-prod-database_url" -> null
-       data_type       = "text" -> null
-       has_value_wo    = false -> null
-       id              = "/home-unite-us/fullstack-prod-database_url" -> null
-       key_id          = "alias/aws/ssm" -> null
-       name            = "/home-unite-us/fullstack-prod-database_url" -> null
-       region          = "us-west-2" -> null
-       tags            = {
-           "project" = "home-unite-us"
        } -> null
-       tags_all        = {
-           "managed-by" = "terraform-incubator"
-           "project"    = "home-unite-us"
        } -> null
-       tier            = "Standard" -> null
-       type            = "SecureString" -> null
-       value           = (sensitive value) -> null
-       value_wo        = (write-only attribute) -> null
-       version         = 1 -> null
#        (2 unchanged attributes hidden)
    }

  # module.home-unite-us.module.db_url_prod.random_password.password will be destroyed
  # (because random_password.password is not in configuration)
-   resource "random_password" "password" {
-       bcrypt_hash      = (sensitive value) -> null
-       id               = "none" -> null
-       length           = 48 -> null
-       lower            = true -> null
-       min_lower        = 0 -> null
-       min_numeric      = 0 -> null
-       min_special      = 0 -> null
-       min_upper        = 0 -> null
-       number           = true -> null
-       numeric          = true -> null
-       override_special = "!#$-" -> null
-       result           = (sensitive value) -> null
-       special          = true -> null
-       upper            = true -> null
    }

  # module.home-unite-us.module.prod_service.aws_ecs_service.fargate[0] will be updated in-place
!~  resource "aws_ecs_service" "fargate" {
        id                                 = "arn:aws:ecs:us-west-2:035866691871:service/incubator-prod/home-unite-us-fullstack-prod"
        name                               = "home-unite-us-fullstack-prod"
        tags                               = {}
!~      task_definition                    = "arn:aws:ecs:us-west-2:035866691871:task-definition/home-unite-us-fullstack-prod:1" -> (known after apply)
#        (18 unchanged attributes hidden)

#        (7 unchanged blocks hidden)
    }

  # module.home-unite-us.module.prod_service.aws_ecs_task_definition.task must be replaced
-/+ resource "aws_ecs_task_definition" "task" {
!~      arn                      = "arn:aws:ecs:us-west-2:035866691871:task-definition/home-unite-us-fullstack-prod:1" -> (known after apply)
!~      arn_without_revision     = "arn:aws:ecs:us-west-2:035866691871:task-definition/home-unite-us-fullstack-prod" -> (known after apply)
!~      container_definitions    = (sensitive value) # forces replacement
!~      enable_fault_injection   = false -> (known after apply)
!~      id                       = "****************************" -> (known after apply)
!~      revision                 = 1 -> (known after apply)
-       tags                     = {} -> null
#        (13 unchanged attributes hidden)
    }

  # module.home-unite-us.module.prod_service.aws_lb_target_group.this will be updated in-place
!~  resource "aws_lb_target_group" "this" {
        id                                 = "arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/home-unite-us-fullstack-prod-tg/f8cec6f42f04281f"
        name                               = "home-unite-us-fullstack-prod-tg"
        tags                               = {}
#        (20 unchanged attributes hidden)

!~      health_check {
!~          path                = "/" -> "/api/health/"
#            (8 unchanged attributes hidden)
        }

!~      target_health_state {
+           enable_unhealthy_connection_termination = false
+           unhealthy_draining_interval             = 0
        }

#        (3 unchanged blocks hidden)
    }

Plan: 1 to add, 2 to change, 3 to destroy.

✅ Plan applied in Terraform apply (OIDC) #70