fix: detach from the interpreter while creating a list stream - #787
Merged
Merged
Conversation
`list()` created the stream while still attached. `MemoryStore` takes its read lock at that point, and a concurrent `put()` or `delete()` releases the replaced Python buffer while holding the write lock, which needs to re-attach. On GIL builds the lister holds the GIL while waiting for the lock; on free-threaded builds it blocks a pending stop-the-world. Either way both threads wait on each other forever. Create the stream inside `py.detach`, as `list_with_delimiter()` already does. The regression test runs in a subprocess because on failure the stuck lister holds the GIL and would hang the test process itself. Closes #785 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
This PR was written by Claude (Claude Code), not by @kylebarron.
obs.list()deadlocks when another thread callsput()ordelete()on the sameMemoryStore. This PR creates the list stream insidepy.detach, aslist_with_delimiter()already does.Closes #785
Cause
list()calledstore.list(...)while still attached to the interpreter, andInMemory::listtakes the store's read lock right away. Meanwhile,put()keeps the caller's buffer zero-copy, andInMemory::put_optsdrops the value it replaces while holding the write lock. Dropping thatPyBufferhas to re-attach so it can release the buffer.A stack sample of the hung process shows both sides:
_obstore::list::list→InMemory::list→RwLock::read, parked while holding the GILInMemory::put_opts→PyUntypedBuffer::drop→Python::try_attach→take_gil, parked while holding the write lockOn free-threaded builds, the attached lister instead blocks a pending stop-the-world (for example,
gc.collect()), and the writer's attach waits on that stop-the-world.list()was the only top-level function that called into the store while attached. Every other sync entry point already runs its store call insidepy.detach.Verification
Local results (macOS arm64, debug build) with the issue's reproduction script (20k iterations):
🤖 Written by Claude Code