Skip to content

Fix repeatable sonpy SIGABRT crashes; add crash detection and diagnostics - #5

Merged
stevevanhooser merged 6 commits into
mainfrom
claude/ndr-matlab-sonpipe-crashes-o6hnvx
Jul 21, 2026
Merged

Fix repeatable sonpy SIGABRT crashes; add crash detection and diagnostics#5
stevevanhooser merged 6 commits into
mainfrom
claude/ndr-matlab-sonpipe-crashes-o6hnvx

Conversation

@stevevanhooser

@stevevanhooser stevevanhooser commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Summary

Reading certain CED Spike2 files through sonpipe reliably crashed the Python
reader process with SIGABRT (Abort trap: 6) inside CED's compiled sonpy
library — with no error surfaced to MATLAB, just silently missing data and a
macOS crash report. This PR fixes the crash, and adds detection + diagnostics so
any future sonpy misbehavior is loud and locatable instead of silent.

Root cause

sonpy reads the requested samples successfully, but on some files the very
next call into sonpy aborts the process. A breadcrumb log made it obvious:

-> ReadInts args=0,200001,15960000,17960010
<- ReadInts size=200001        ← all samples returned
[abort]                         ← crash on the NEXT sonpy call

That next call was our own: read_waveform fetched GetChannelScale /
GetChannelOffset after the read to convert ADC counts to real units.

The fix

  • Make no sonpy call after the sample read. read_waveform now resolves
    scale/offset before ReadInts/ReadFloats and accepts them from the
    caller; cmd_read passes the values already fetched in channel_info. After
    the data comes back, nothing calls into sonpy, so a good read can't be
    turned into a crash. Verified in the log: <- ReadInts is now followed only
    by del SonFile → done → hard_exit.
  • Clamp over-reads. _wave_tick_range caps nmax to the samples that
    actually exist from the start point to EOF, so fixed-size chunking near the
    end of a file never hands sonpy an over-read (another assert trigger).

Also in this PR (hardening + diagnostics)

  • Hard exit past interpreter shutdown. A separate abort could fire in
    sonpy's static/atexit teardown after a command completed. The entry
    point (sonpipe.cli:run, used by the sonpipe command and python -m sonpipe) flushes all output then calls os._exit(), skipping the teardown.
    main() stays importable (no os._exit), so tests/in-process use are
    unaffected.
  • Explicit file close at the end of each command (context manager), rather
    than relying on garbage collection at shutdown.
  • SONPIPE_LOG breadcrumb logging (debuglog): opt-in, off by default,
    zero overhead when unset. Writes a line before/after every call into sonpy,
    flushed with os.write+fsync so it survives an abort(); the last line
    names the crashing call. Documented in sonpipe --help, the README
    Troubleshooting section, and the module docstring.

MATLAB-side detection (paired PR in NDR-matlab)

The companion NDR-matlab branch adds a matching change so a mid-stream crash is
raised as sonpipe:crash / sonpipe:truncated (with the exact command) instead
of returning truncated data — it relies on the completion sentinel this CLI
already prints. The fake CLI test double now emits that sentinel too.

Tests

  • New tests/test_debuglog.py: log gating, breadcrumb format, a real subprocess
    abort() proving the pre-call breadcrumb survives, and run()'s hard-exit.
  • New tests/test_sonfile.py cases: nmax clamping, scaling with
    caller-supplied scale/offset (asserting sonpy is not consulted after the
    read), and close() / context-manager behavior.
  • Full suite: 60 passed, 5 skipped locally (skips are the real-sonpy integration
    tests, which need CED binaries).

Verification note

Because the crash is inside closed-source sonpy, please spot-check that a known
channel's values/plot look correct on a previously-crashing file, to confirm
sonpy was returning correct data before the post-read call.

🤖 Generated with Claude Code

https://claude.ai/code/session_01C1UYvbvczg9QvqKg8YNr4k

Generated by Claude Code

claude added 5 commits July 19, 2026 19:12
@claude
Catch and diagnose sonpy hard crashes (SIGABRT/abort)
08eab77 
CED's sonpy is a compiled C++ library that, on certain files/channels,
fails an internal assertion and calls abort() (SIGABRT) instead of
raising a catchable Python error. This kills the reader process
mid-stream, which the host (e.g. MATLAB) could previously see as
silently truncated/empty data -- especially on Apple Silicon, where the
'arch -x86_64' wrapper does not reliably propagate a signal death as a
non-zero exit status.

Two complementary changes:

Detection (MATLAB layer). invoke_binary now captures stderr separately
and requires the CLI's completion sentinel ("sonpipe: wrote N ...");
if it is missing, or N does not match the bytes captured, it raises
sonpipe:crash / sonpipe:truncated naming the exact command instead of
returning short data. invoke_text captures stderr to its own file so a
warning or crash text can no longer corrupt the JSON handed to
jsondecode, and surfaces it on failure.

Diagnosis (Python layer). New opt-in breadcrumb logging (debuglog),
enabled via the SONPIPE_LOG environment variable, writes one line
immediately before and after every call into sonpy, flushed with an
unbuffered os.write + fsync so the line survives an abort(). Every sonpy
boundary in sonfile.py (SonFile open, ReadInts, ReadFloats, ReadEvents,
marker reads) and the CLI entry point are instrumented, so the last line
in the log names the exact sonpy call and arguments that crashed.

Adds tests/test_debuglog.py (including a real subprocess abort() that
confirms the pre-call breadcrumb survives) and documents the workflow in
the README Troubleshooting section.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C1UYvbvczg9QvqKg8YNr4k
@claude
Instrument metadata/teardown and close sonpy file explicitly
8b1444f 
Follow-up to the crash-diagnosis work. A breadcrumb log from a real
crash showed the abort happening outside the previously instrumented
read/open calls -- i.e. in the metadata accessors or, most likely, when
sonpy releases the file handle at interpreter shutdown (a teardown-order
assertion that fires after a read has already succeeded).

Two changes:

- Instrument the remaining sonpy boundaries in sonfile.py: GetTimeBase,
  MaxChannels, ChannelType, ChannelDivide, GetMaxTime, GetFileVersion,
  AppID and the _num/_text accessors (ChannelMaxTime, GetChannelScale,
  GetChannelOffset, GetIdealRate, GetChannelTitle/Units/Comment). The log
  now covers every call into sonpy, so a metadata crash is pinpointed the
  same way a read crash is.

- Add SmrxFile.close() plus context-manager support, and have every CLI
  command close the file explicitly (with output flushed first) instead
  of leaving it to garbage collection. Closing while the interpreter is
  healthy makes the release step visible in the log and, in practice,
  avoids the shutdown-order abort() class entirely. A new "done"
  breadcrumb in main() marks a clean finish, so the log's last line
  distinguishes a completed command from a mid-call crash.

Adds close()/context-manager tests and expands the README Troubleshooting
section to explain reading the log tail.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C1UYvbvczg9QvqKg8YNr4k
@claude
Hard-exit past interpreter shutdown to avoid post-read sonpy abort
6f1da8c 
A breadcrumb log from a real crash showed the command running fully to
completion -- read succeeds, file handle released, main() returns rc=0,
'done' logged -- and sonpy then aborting (SIGABRT) during Python's
*interpreter shutdown*, in a static/atexit destructor that runs after all
output has already been delivered. That abort cannot be caught from
Python and does no harm to the result, but it still produces a crash
report and (through the Apple-Silicon arch wrapper) an alarming failure.

Add a process entry point sonpipe.cli:run() that runs main(), flushes
stdout/stderr, and then calls os._exit() -- terminating immediately
without running Python finalizers or C++ static destructors, so the
teardown code where sonpy aborts never executes. Every stream is flushed
first, so no data is lost. main() stays a pure, importable function (no
os._exit), so tests and in-process callers are unaffected.

Wire both entry points to run(): the console script (pyproject
[project.scripts]) and python -m sonpipe (__main__.py). The MATLAB-test
fakecli.py now also calls run() so the double exercises the same exit
path. A 'hard_exit' breadcrumb marks the bypass in the log.

Adds unit + end-to-end subprocess tests for run() and documents the
workaround in the README.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C1UYvbvczg9QvqKg8YNr4k
@claude
Document SONPIPE_LOG in the CLI --help output
498df8e 
Add an epilog to the argument parser describing the SONPIPE_LOG
environment variable (breadcrumb logging for diagnosing a hard sonpy
crash), so it is discoverable from `sonpipe --help` / `python -m sonpipe
--help`, not only in the README.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C1UYvbvczg9QvqKg8YNr4k
@claude
Make no sonpy call after the sample read; clamp over-reads
dd61d56 
Breadcrumb logs from a crashing file showed two signatures: (1) a read
returns all its samples (`<- ReadInts size=200001`) and then aborts on
the very next call into sonpy -- the post-read GetChannelScale/
GetChannelOffset lookups used for scaling; and (2) metadata opens abort
in the first sonpy call after SonFile.

Address the read path (the common, recoverable case):

- read_waveform now resolves scale/offset BEFORE the sample read and
  accepts them from the caller. cmd_read passes the values already
  fetched in channel_info, so after ReadInts/ReadFloats the code makes
  *no* further call into sonpy -- a good read can no longer be turned
  into a crash by a follow-up sonpy call on a problematic file.

- _wave_tick_range clamps nmax to the samples actually available from
  tfrom to the channel end, so a request that runs past the last sample
  (e.g. NDR's fixed-size chunking near EOF) is never handed to sonpy as
  an over-read -- another known way to trip its internal assertions.

- Instrument the last un-logged sonpy call (GetOpenError) so a metadata
  crash is named in the log like every other call.

Adds tests for the clamp and for scaling with caller-supplied
scale/offset (verifying sonpy is not consulted after the read).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C1UYvbvczg9QvqKg8YNr4k
@stevevanhooser stevevanhooser mentioned this pull request Jul 19, 2026
Merged
@claude
Make abort() subprocess test cross-platform (Windows CI)
a8f65ac 
test_breadcrumb_survives_an_abort asserted returncode == -SIGABRT, which
is POSIX-specific. On Windows there are no real signals; a process that
aborts exits with a positive code. Assert abnormal termination
(returncode != 0) instead; the meaningful checks (the pre-call breadcrumb
survives and no completion line was written) are unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C1UYvbvczg9QvqKg8YNr4k
@stevevanhooser
stevevanhooser merged commit 79132d6 into main Jul 21, 2026
9 checks passed
@stevevanhooser
stevevanhooser deleted the claude/ndr-matlab-sonpipe-crashes-o6hnvx branch July 21, 2026 21:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@stevevanhooser @claude