Skip to content

CRITICAL BILLING BUG] Infrastructure Deadlock on Disabled Deployment (Account: aurorarecmgmt-cloud #382

Description

@aurorarecmgmt-cloud

Summary:
A critical backend infrastructure deadlock on a private deployment is generating continuous, unauthorized billing charges despite the deployment being Disabled and max_instances explicitly set to 0.

Account & Environment Details:

  • Username: aurorarecmgmt-cloud
  • Affected Resource: Private Deployment (huggy-mochi-generator)
  • Current Status: Disabled, max_instances: 0, API Token Revoked.

Bug Description & Evidence:

  • Infrastructure Deadlock: The deployment metric panel shows SETUP TIME: 100.0%, ACTIVE TIME: 0.0%, and Queue length: 1. The backend hardware pod is stuck in an infinite initialization/cold-start loop and refuses to terminate.
  • Uncontrolled Charges: The outstanding balance jumped from ~$9.00 USD (when the initial support ticket was submitted 18 hours ago) to over $91.46 USD in 18+ hours while completely inactive, with zero user predictions being processed.
  • Deletion Blocked: Attempting to delete the deployment fails due to lingering stuck background instances.

No Support Response: First email was sent to support@replicate.com 18 hours ago when the balance was ~$9.00 USD. No response has been received while costs continue to accumulate rapidly.

Expected Behavior:
Setting max_instances to 0 and disabling the deployment should immediately terminate all underlying hardware pods and stop all billing ticks.

Actual Behavior:
The backend pod remains locked in the setup phase indefinitely, continuously accruing charges on a disabled deployment.

Screenshots Attached:

  • Metric dashboard showing SETUP TIME: 100% / ACTIVE TIME: 0%
  • Billing dashboard showing unpaid balance surging to $91.46 USD
  • Deployment settings showing Disabled and max_instances: 0

Requested Resolution:

  • Force Terminate: Immediately kill all background pods/instances tied to account aurorarecmgmt-cloud.
  • Billing Adjustment: Reset/refund the erroneous charges accumulated during this platform setup deadlock back to $0.00.

(Escalating here for engineering visibility due to zero response on the 18-hour-old support ticket.)

Image Image Image Image

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions