Skip to content

Add sonpipe: CLI bridge for reading CED Spike2 files via sonpy - #1

Merged
stevevanhooser merged 11 commits into
mainfrom
claude/smrx-cli-matlab-bridge-acxw21
Jul 18, 2026
Merged

Add sonpipe: CLI bridge for reading CED Spike2 files via sonpy#1
stevevanhooser merged 11 commits into
mainfrom
claude/smrx-cli-matlab-bridge-acxw21

Conversation

@stevevanhooser

@stevevanhooser stevevanhooser commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR introduces sonpipe, a complete command-line bridge for reading Cambridge Electronic Design (CED) Spike2 data files (both 32-bit .smr and 64-bit .smrx formats) via CED's proprietary sonpy library. The tool streams data as raw binary bytes to stdout for efficient ingestion by MATLAB and other host environments, with accompanying MATLAB wrappers that provide drop-in compatibility with NDR-matlab's ndr.format.ced functions.

Key Changes

Core Python Implementation

  • src/sonpipe/sonfile.py: Thin, testable wrapper around sonpy.SonFile that isolates all proprietary library calls. Handles channel numbering (Spike2's 1-based convention vs. sonpy's 0-based indexing), metadata extraction, and data reading for waveforms, events, and markers.
  • src/sonpipe/cli.py: Command-line interface with four sub-commands:
    • header: Emit file and channel metadata as JSON
    • sampleinterval: Query timing info for a single channel
    • read: Stream channel data as raw binary (waveforms/events) or JSON (markers)
    • channels: List available channels
  • src/sonpipe/channels.py: CED channel-type code definitions and mappings (ADC, RealWave, Event, Marker, TextMark, etc.) that align with sonpy's DataType enum and NDR-matlab conventions.
  • src/sonpipe/errors.py: Custom exception hierarchy.
  • src/sonpipe/__init__.py and __main__.py: Package initialization and CLI entry point.

Testing Infrastructure

  • tests/fakesonpy.py: Synthetic in-memory fake of the sonpy.lib module with a minimal Spike2 file layout (ADC, RealWave, Event, Marker, TextMark channels). Enables full test coverage without CED's proprietary binaries.
  • tests/test_sonfile.py: Unit tests for SmrxFile wrapper (channel discovery, metadata, waveform/event/marker reads).
  • tests/test_cli.py: End-to-end CLI tests (header, sampleinterval, read sub-commands with various options).
  • tests/test_integration.py: Integration tests against real sonpy and a real Spike2 file (skipped if sonpy unavailable).
  • tests/conftest.py: Pytest fixtures that inject the fake sonpy for unit tests.

MATLAB Wrappers

  • matlab/+sonpipe/read_SOMSMR_header.m: Read file and channel metadata (JSON).
  • matlab/+sonpipe/read_SOMSMR_datafile.m: Stream channel data (waveforms, events, markers).
  • matlab/+sonpipe/read_SOMSMR_sampleinterval.m: Query sample interval and duration.
  • matlab/+sonpipe/channels.m: List channels in NDR-style struct format.
  • matlab/+sonpipe/executable.m: Locate and cache the sonpipe CLI executable.
  • matlab/+sonpipe/runcmd.m: Run system commands with a Python-safe environment (clears LD_LIBRARY_PATH/DYLD_LIBRARY_PATH).
  • matlab/+sonpipe/private/invoke_binary.m and invoke_text.m: Helpers to capture binary and text output from the CLI.
  • matlab/+sonpipe/Contents.m and channelinfo.m: Package documentation and utility functions.

MATLAB Unit Tests

  • test/+sonpipe/+unittest/TestCase.m: Base fixture that configures a fake CLI for testing without CED binaries.
  • test/+sonpipe/+unittest/HeaderTest.m, **`DatafileTest.

https://claude.ai/code/session_01VTQEJGfiSxeVG1J5GSmaTy

VH-Lab and others added 11 commits July 18, 2026 14:26
VH-Lab @claude
Add sonpipe CLI bridge, MATLAB package, tests, and CI
9e7365e 
Build a standalone command-line bridge that reads CED Spike2 files (both
32-bit .smr and 64-bit .smrx) via CED's sonpy and streams data as raw
binary for fast, chunked ingestion by MATLAB.

Python CLI (src/sonpipe):
- header / sampleinterval / read / channels sub-commands imitating the
  ndr.format.ced.* reading functions
- raw little-endian binary output for waveforms and event times; JSON for
  metadata and markers
- sample-based (--start/--count) and time-based (--t0/--t1) chunking
- reads all main Spike2 channel kinds (Adc, Event, Marker, AdcMark,
  RealMark, TextMark, RealWave); Spike2 1-based channel numbers mapped to
  sonpy's 0-based indices
- warns when a read exceeds 50 MB (pipe can be slow); --no-size-warning
- sonpy declared as a pip dependency, not vendored, per CED's license

MATLAB client (matlab/+sonpipe):
- read_SOMSMR_header / read_SOMSMR_sampleinterval / read_SOMSMR_datafile
  drop-in analogues of ndr.format.ced.*, plus channels/channelinfo/executable
- invokes the CLI and reads binary output back with fread/typecast

Tests and CI:
- Python tests with a fake sonpy shim (no CED binaries needed)
- MATLAB matlab.unittest suite in test/+sonpipe/+unittest driving a fake CLI
- matbox-style workflows (mirroring NDR-matlab) that compile and test both
  the CLI and MATLAB code on Linux, Windows, macOS Intel, and macOS Apple
  Silicon

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VTQEJGfiSxeVG1J5GSmaTy
VH-Lab @claude
Scope CI to main and make MATLAB->Python calls env-safe
6e413a8 
- CI (both cli-tests and matlab-tests) now triggers on push to main, pull
  requests targeting main, and manual workflow_dispatch, matching NDR-matlab.
- Add sonpipe.runcmd: run system commands with LD_LIBRARY_PATH /
  DYLD_LIBRARY_PATH cleared, so a child Python launched from MATLAB does not
  inherit MATLAB's bundled shared libraries and fail to start (a common cause
  of Python crashes when shelling out from MATLAB on Linux/macOS). Route the
  CLI invokers, executable lookup, and the test harness through it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VTQEJGfiSxeVG1J5GSmaTy
VH-Lab @claude
Add optional real-sonpy integration tests and document test data
0c5ca12 
The default Python/MATLAB suites use a synthetic fake sonpy (no real Spike2
files, deterministic, cross-platform). Add an opt-in integration suite that
exercises the real CED sonpy against a real .smr/.smrx file, gated on the
SONPIPE_TEST_FILE environment variable and sonpy being importable (skipped
otherwise). Document both in the README.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VTQEJGfiSxeVG1J5GSmaTy
VH-Lab @claude
Simplify MATLAB CI: drop P-code step and license secret
571b151 
The MATLAB wrappers only shell out to the CLI, so there is nothing to
compile on the MATLAB side; running the unit tests already loads and parses
every function. Remove the pcode "compile" step (the CLI build step already
compiles the actual tool).

Also drop the batch-license secret plumbing: matlab-actions provides
MathWorks licensing for free on public repositories, so no secret is needed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VTQEJGfiSxeVG1J5GSmaTy
VH-Lab @claude
Add example .smrx and real-sonpy integration CI on all platforms
2287ea6 
- Add example/spike2data.smrx, a real 64-bit Spike2 file, as the default
  fixture for the integration tests.
- Expand the integration suite to exercise the real CED sonpy end-to-end:
  the SmrxFile wrapper (header, waveform/event/marker reads, count-limited
  reads) and the CLI as a subprocess (raw-binary pipe). Defaults to the
  checked-in example file; SONPIPE_TEST_FILE overrides it.
- Add integration-tests.yml running the real sonpy on Linux, Windows, macOS
  Intel and macOS Apple Silicon using Python 3.14, for which CED ships sonpy
  wheels on every platform.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VTQEJGfiSxeVG1J5GSmaTy
VH-Lab @claude
Add cross-platform install scripts
bc16619 
Add install.sh (Linux/macOS) and install.ps1 (Windows) that install sonpipe
into an isolated virtual environment and expose the `sonpipe` command on the
user's PATH:

- Linux/macOS: venv at ~/.local/share/sonpipe/venv, command symlinked at
  ~/.local/bin/sonpipe (the console script's shebang targets the venv, so it
  runs correctly from anywhere, including MATLAB).
- Windows: venv under %LOCALAPPDATA%\sonpipe, with optional -AddToPath.

Both prefer Python 3.14 (where CED ships sonpy wheels on every platform),
verify the CLI and the real sonpy import, warn clearly when no sonpy wheel
exists for the chosen Python, and print the sonpipe.executable(...) line for
MATLAB. Document the scripts and the recommended install location in the
README.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VTQEJGfiSxeVG1J5GSmaTy
VH-Lab @claude
Re-trigger CI after making the repository public
a7db541 
Empty commit to kick a fresh workflow run now that MATLAB CI can obtain a
license on the public repository.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VTQEJGfiSxeVG1J5GSmaTy
VH-Lab @claude
Fix sonpy import to match CED's real, platform-varying layout
2cc56e2 
The integration runs revealed that CED packages sonpy inconsistently: the
Windows wheel exposes the API at the top level (sonpy.SonFile), while the
Linux cp314 wheel ships an empty __init__ and puts the API in the compiled
submodule sonpy.sonpy. Neither provides the sonpy.lib submodule the code
assumed, so `from sonpy import lib` failed everywhere real sonpy was present.

- load_sonpy now probes sonpy, sonpy.lib, and sonpy.sonpy and returns the
  module that actually exposes SonFile; factor the resolution into
  _resolve_son_module and cover all three layouts (plus not-found) with unit
  tests that need no real sonpy.
- Fix the integration workflow's verification step (the old one referenced
  sonpy.__version__, which some wheels do not define) to instead confirm
  load_sonpy() finds SonFile.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VTQEJGfiSxeVG1J5GSmaTy
VH-Lab @claude
Run macOS Apple Silicon integration under Rosetta (x86_64)
936b3c0 
The real-sonpy integration passed on Linux and Windows, confirming the sonpy
binding is correct. It failed only on macos-14 because CED's macOS wheel,
though tagged universal2, ships an x86_64-only binary that cannot load on
arm64.

Force architecture x64 across the integration matrix: native on Linux,
Windows and macOS Intel, and x86_64-under-Rosetta on Apple Silicon, so the
real-sonpy suite is green on all four platforms. Document the Apple Silicon
limitation in the README.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VTQEJGfiSxeVG1J5GSmaTy
VH-Lab @claude
Force x86_64 (Rosetta) Python for macOS ARM integration
6ae2250 
architecture: x64 alone did not yield an x86_64 interpreter on the macos-14
runner (setup-python fell back to the arm64 framework Python), so CED's
x86_64-only sonpy still failed to load. That framework Python is universal2,
so run every Python command on macos-14 through `arch -x86_64` to execute its
x86_64 slice under Rosetta 2, where the x86_64 sonpy loads. Native x86_64
Python is used on the other platforms.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VTQEJGfiSxeVG1J5GSmaTy
VH-Lab @claude
Drop macos-13 (Intel) from CI; rely on macos-14
48d9f9e 
Intel macOS runners are scarce and queue for a long time. macos-14 (Apple
Silicon) already covers both architectures: the CLI and MATLAB fake-based
suites run natively on arm64, and the real-sonpy integration runs x86_64 under
Rosetta 2 (arch -x86_64) — the same x86_64 sonpy binary a native Intel Mac
would load. Remove macos-13 from all three workflows and update the docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VTQEJGfiSxeVG1J5GSmaTy
@stevevanhooser
stevevanhooser merged commit c45c436 into main Jul 18, 2026
9 checks passed
@stevevanhooser
stevevanhooser deleted the claude/smrx-cli-matlab-bridge-acxw21 branch July 18, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@stevevanhooser