We’re seeing intermittent submission failures on our self-hosted compute worker. Some submissions stay in Running for an unusually long time, then eventually show Failed. Rerunning the same submission later has succeeded.
For affected runs, the worker logs show that after ingestion_program starts, its attempt to update the submission status to Scoring receives HTTP 500. The worker logs “Failed to update submission status to Scoring,” but the Celery task then reports success. We’re unsure whether this failed status update explains the long wait and eventual failure, or whether there is an additional issue.
Could you help us identify the cause of the intermittent HTTP 500s and long-running submissions, and advise how the worker should handle a failed status update?
We’re seeing intermittent submission failures on our self-hosted compute worker. Some submissions stay in Running for an unusually long time, then eventually show Failed. Rerunning the same submission later has succeeded.
For affected runs, the worker logs show that after ingestion_program starts, its attempt to update the submission status to Scoring receives HTTP 500. The worker logs “Failed to update submission status to Scoring,” but the Celery task then reports success. We’re unsure whether this failed status update explains the long wait and eventual failure, or whether there is an additional issue.
Could you help us identify the cause of the intermittent HTTP 500s and long-running submissions, and advise how the worker should handle a failed status update?