fix(par-dsp): handle peaks with no candidate events in evtsel - #51
Open
ggmarshall wants to merge 1 commit into
Open
fix(par-dsp): handle peaks with no candidate events in evtsel#51ggmarshall wants to merge 1 commit into
ggmarshall wants to merge 1 commit into
Conversation
A channel whose DAQ trigger threshold sits above a configured gamma line
yields an empty rough-energy mask for that peak. Nothing is ever read into
the peak's buffer, so `obj_buf` stays at its `None` initialiser, and the
end-of-file flush (`... or file == raw_files[-1]`) then passed `None`
straight to `build_dsp`:
RuntimeError: raw_in was not a file name, Table, or LH5Iterator: None
The second-flush branch already guarded on `obj_buf is not None and
len(obj_buf) > 0`; the first-flush branch did not.
Seen in production on l200-p18-r000 channel V01404A, whose minimum
daqenergy is 1501 across all 49 cal files (~1518 keV after the raw cal
curve), so the 583.191, 727.33 and 860.564 keV peaks have zero candidates
while 1620.5 and 2614.553 are normal.
Extract the pk_dicts construction into `build_peak_dicts`, which skips
peaks with no candidate events and warns, and add a defensive None/empty
guard on the flush so `None` can never reach `build_dsp` regardless. Peaks
with at least one candidate event are unaffected.
Also fail fast in the peak-file consumers: eopt and dplms filter on the
`peak` column with `np.isin` and never checked that the peaks their config
requests are present. A missing peak degraded silently -- in eopt an empty
slice makes `flat_val` NaN, both clamps are skipped because `nan < 1` and
`nan > 4` are each False, and `"nan*us"` is injected into the cusp/zac/etrap
filter parameters -- or blew up later with an IndexError. New
`require_peaks_present` helper alongside `require_config_keys`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens the par-geds DSP calibration pipeline against “configured peak has zero candidate events” cases by skipping such peaks during event selection and by failing fast when downstream consumers request peaks that are absent from the peak file.
Changes:
- Add
build_peak_dictsto drop peaks with no candidate events (and warn), and guard the evtsel flush soNone/empty buffers never reachdspeed.build_dsp. - Add
require_peaks_presentconfig/IO validation helper and use it ineopt.pyanddplms.pyto error out when requested peaks are missing. - Add unit tests covering peak skipping, logging, and missing-peak validation behavior.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_peak_selection.py | New tests for skipping empty peaks, warning logs, and require_peaks_present error semantics. |
| src/legenddataflowscripts/utils/cfgtools.py | Adds require_peaks_present helper for validating required peak labels exist in a peak file. |
| src/legenddataflowscripts/utils/init.py | Re-exports require_peaks_present from legenddataflowscripts.utils. |
| src/legenddataflowscripts/par/geds/dsp/evtsel.py | Introduces build_peak_dicts and adds a defensive flush guard to prevent None buffers from being processed. |
| src/legenddataflowscripts/par/geds/dsp/eopt.py | Validates peak presence before filtering/optimisation to avoid silent NaNs or late failures. |
| src/legenddataflowscripts/par/geds/dsp/dplms.py | Validates peak presence before filtering/processing to avoid silent empty selections. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The p18/r000 production died at
build_pars_evtsel_gedsfor channel V01404A:V01404A has a hard DAQ trigger threshold: its minimum
daqenergyis exactly 1501across all 49 cal files of p18/r000, and the same floor holds in every p18 run
r000–r006. With the raw cal curve
a = 1.01167that is a ~1518 keV threshold, sothe 583.191, 727.33 and 860.564 keV peaks have zero candidate events, while 1620.5
(7315) and 2614.553 (12112) are normal. It is the only channel of 59 with this
signature.
With an empty mask the
len(idx_i[0]) > 0guard skips the read, sopeak_dict["obj_buf"]stays at its
Noneinitialiser. On the last raw file the flush fires unconditionally(
... or file == raw_files[-1]) and the first-flush branch had noNone/emptyguard — unlike the second-flush branch, which already checked
obj_buf is not None and len(obj_buf) > 0. SoNonereachedbuild_dsp.This is a pre-existing latent bug, not a regression from the perf work on this branch.
Changes
evtsel.py— extract thepk_dictsconstruction intobuild_peak_dicts, whichdrops peaks with no candidate events and logs a warning naming each one. Add a
defensive
obj_buf is None or len(...) == 0guard on the flush soNonecan neverreach
build_dspregardless of how the buffer got there; the now-redundant guard onthe second branch collapses to
else.eopt.py,dplms.py— fail fast when a peak the config asks for is absent fromthe peak file. Both filter the
peakcolumn withnp.isinand never checkedmembership. A missing peak degraded silently: in
eopt.pyan empty slice makesflat_valNaN, both clamps are skipped becausenan < 1andnan > 4are eachFalse, and
"nan*us"is injected into the cusp/zac/etrap filter parameters — or itblew up later with an
IndexErroron an empty table.cfgtools.py— newrequire_peaks_presenthelper next torequire_config_keys,same "list everything that is missing" style.
tests/test_peak_selection.py— 7 tests covering the skip, the logging and thehelper.
Behaviour is unchanged for any channel with at least one candidate event in each peak
window.
Verification
pytest: 71 passed.pre-commit: clean.snakemake: exit 0, no traceback, three
skipping this peakwarnings, and the peakfile now holds
{1620: 5797, 2614: 8948}.(
1748.7557954482058 / 1844.971441718002and1099.76778580735 / 1147.7177982685862),confirming no numerical change for peaks that do have data.
Separate pre-existing issue, not fixed here
inputs/dataprod/config/par/geds/dsp/eopt/l200-p01-r%-T%-ICPC-dsp_full_eopt.yamlrequests peak 238.632, which the evtsel peak config never writes. That config will now
fail loudly rather than silently producing nan-filled optimiser output. No shipped p18
config is affected — p18 uses
dsp_eopt.yaml, which requests only 2614.553.Notes
Based on
perf-par-dsp(#44) rather thanmain, sincecfgtools.pydoes not exist onmainyet and the touched files are identical on this branch.Per
AI_POLICY.md: this change was drafted with AI assistance (Claude) and reviewed bythe submitter.