Skip to content

Cache a progress report in dscheck.einfo before the PBS walltime - #43

Merged
zaihuaji merged 1 commit into
mainfrom
hua-work-dsquasar
Sep 21, 2026
Merged

zaihuaji merged 1 commit into
mainfrom
hua-work-dsquasar

Conversation

@zaihuaji

Copy link
Copy Markdown
Collaborator

Summary

  • A long -A 3 / -A 4 batch run was killed by PBS before start_actions() reached its email block, so no report was ever sent. The run now caches a progress report into dscheck.einfo an hour before the walltime and keeps working; the dscheck daemon mails it once the record is unlocked (after a kill or a normal finish).
  • A run that finishes in time overwrites the cached progress report with the full report, so exactly one mail goes out.
  • The walltime is read back from PBS (qstat ReqdTime) instead of assumed, since the dscheck daemon caps the requested 24 h to the maximum of the queue the job lands in (only 6 h for default).
  • Guide deck slides 20/23/16 updated to match; the old "stops cleanly at 23 hours" description no longer applied.

Test plan

  • python3 -m py_compile src/rda_python_dsquasar/dsquasar.py
  • Verified get_pbs_info() returns {} on failure and that the ReqdTime key is produced (pg_sig.py:919)
  • Verified build_customized_email() sends first and only caches on failure, so cache_customized_email() is required to park a report (pg_dbi.py:2190/2277)
  • Verified the daemon only acts on einfo when pid = 0 (pg_check.py:436/455), so a cached report is not mailed while the job still runs
  • Verified both check_batch_deadline() callers run in the parent only (children sys.exit() at dsquasar.py ~922/~1014)
  • Deck regenerated and QA'd from the exported PDF

Generated with Claude Code

…p version to 3.0.12

A long -A 3 or -A 4 batch run was killed by PBS before start_actions()
reached its email block, so no report was ever sent. Rather than cut the
work short to make time for the mail, the run now parks a progress report
in dscheck.einfo an hour before the walltime and keeps going: the dscheck
daemon mails that report once the record is unlocked, and a run that does
finish overwrites it with the full report so only one mail goes out.

The walltime is read back from PBS (qstat ReqdTime) instead of assumed,
since the dscheck daemon caps the requested 24 hours to the maximum of
the queue the job lands in.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@zaihuaji
zaihuaji merged commit 862b7d3 into main Sep 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant