feat: add mock MemoryMappedFile support in a new companion package#1054
Open
vbreuss wants to merge 16 commits into
Open
feat: add mock MemoryMappedFile support in a new companion package#1054vbreuss wants to merge 16 commits into
MemoryMappedFile support in a new companion package#1054vbreuss wants to merge 16 commits into
Conversation
Test Results 122 files 122 suites 2h 22m 43s ⏱️ Results for commit d786dda. ♻️ This comment has been updated with latest results. |
vbreuss
force-pushed
the
feat/memory-mapped-files
branch
from
2ff63c7 to
c34523f
Compare
vbreuss
force-pushed
the
feat/memory-mapped-files
branch
5 times, most recently
from
b83d56a to
078fda9
Compare
Adds a new `Testably.Abstractions.MemoryMappedFiles` companion package that
abstracts `System.IO.MemoryMappedFiles.MemoryMappedFile` behind `IFileSystem`,
so memory-mapped-file code can be tested against the in-memory `MockFileSystem`.
## Usage
The abstraction is exposed as the extension property `fileSystem.MemoryMappedFile`
(note: no parentheses), returning an `IMemoryMappedFileFactory`:
IFileSystem fileSystem; // injected
using IMemoryMappedFile mappedFile = fileSystem.MemoryMappedFile
.CreateFromFile("data.bin");
using IMemoryMappedViewAccessor accessor = mappedFile.CreateViewAccessor();
accessor.Write(0, 42);
int value = accessor.ReadInt32(0);
On the real file system every call forwards to the underlying base class library
(BCL) implementation; on the mock the views are built directly over the
in-memory file bytes.
## Public surface
- `IMemoryMappedFileFactory` - the `CreateFromFile` overloads (path-based and
`FileSystemStream`-based, so they work against both the real and the mocked
file system), plus `CreateNew` / `CreateOrOpen` / `OpenExisting`.
- `IMemoryMappedFile` - the `CreateViewAccessor` / `CreateViewStream` overloads.
- `IMemoryMappedViewAccessor` - the full read/write surface of
`MemoryMappedViewAccessor`.
- `MemoryMappedFileSystemViewStream` - a dedicated `Stream`-derived type
returned by `CreateViewStream`, mirroring the `FileSystemStream` pattern:
every `Stream` member is delegated to the wrapped stream, and the
view-stream-specific `PointerOffset` and `Capacity` are added on top.
Parity tests assert the abstraction matches the BCL surface, excluding only the
unsafe handle / security members (`SafeMemoryMappedFileHandle`,
`SafeMemoryMappedViewHandle`, `MemoryMappedFileSecurity` and the
`SafeFileHandle`-based `CreateFromFile` overload), mirroring how the
`SafeFileHandle`-based `FileStream` construction is excluded elsewhere.
## Intentionally not supported on the mock
`CreateNew`, `CreateOrOpen` and `OpenExisting` operate on operating-system shared
memory (named or anonymous) that is not backed by a file. They forward normally
on the real file system but throw `NotSupportedException` on the mock.
## BCL-faithful behavior on the mock
- Capacity handling, bounds checks and thrown exceptions mirror the real
`MemoryMappedViewAccessor` / `MemoryMappedViewStream`.
- `CopyOnWrite` views operate on a private copy of the mapped bytes: writes are
visible within the view but are never persisted to the underlying file, nor
visible to other views.
- Read-only access with a capacity larger than the file throws
`ArgumentException`, matching the BCL.
Two details tied to operating-system memory mapping are intentionally simplified:
a view created without an explicit size has a `Capacity` of exactly the remaining
bytes (the real file system rounds up to the system page size), and
`PointerOffset` is always `0`.
Targets net6.0, net8.0, net9.0, net10.0, netstandard2.1 and netstandard2.0.
Addresses several parity gaps in the mock memory-mapped file support: - Default view access now matches the BCL (ReadWrite) and view access is validated against the mapping's access, so a write view on a read-only mapping throws UnauthorizedAccessException instead of silently succeeding. - The memory-mapped file and its views now have independent lifetimes: a view stays usable after the file is disposed (reference-counted backing). - WriteArray now validates capacity up-front and fails atomically without a partial write, matching UnmanagedMemoryAccessor. - View bounds validation no longer overflows on absurd (near long.MaxValue) offset/size inputs. - Add parity test for MemoryMappedViewStream (renaming the Seek parameter to `loc` to match UnmanagedMemoryStream) and exclude its pointer/handle APIs. - Correct the Microsoft.Bcl.Memory reference comment (security advisory).
…ings - Re-declare `Capacity` on `MemoryMappedFileSystemViewStream`: the uninitialized `UnmanagedMemoryStream` base threw `ObjectDisposedException` on live streams - Reject writable views on `CopyOnWrite` mappings with `UnauthorizedAccessException` - Reject `FileMode.Truncate`/`Append` and `MemoryMappedFileAccess.Write` in `CreateFromFile` before touching the file, matching the BCL exceptions - Delete a file created by a failed `CreateFromFile` call, matching BCL cleanup - Validate `mapName`: empty string throws `ArgumentException`; non-null names on the mock fail fast with `NotSupportedException` instead of being ignored - Throw `ArgumentOutOfRangeException` when an accessor position is at or beyond the capacity; `IOException` when a CoW capacity exceeds the file length or `offset + size` of a view overflows - Route all view I/O through a shared `MemoryMappedViewBacking` with positional access under a lock, so views no longer move the position of a caller-owned stream (`leaveOpen: true`) and can be used concurrently - Batch `ReadArray`/`WriteArray` into single stream operations instead of per-element round-trips; delegate typed overloads to the generic path - Docs: expand "base class library (BCL)" on first use, document the named mapping limitation, and correct the `Capacity` inheritance claim
All divergences were verified against the real BCL `MemoryMappedFile` before fixing: - Emulate the lazy page privatization of copy-on-write views: reads pass through to the shared backing until the view writes a 4096-byte page, which is copied into private memory at that moment (`CopyOnWriteViewBacking`); previously the whole mapping was snapshotted eagerly at view creation - Validate `capacity < 0` before opening the file, so a `FileMode.Create` call with a negative capacity no longer truncates an existing file - Throw `ArgumentOutOfRangeException` from `ReadArray`/`WriteArray` when the position is at or beyond the capacity (independent of the count), and use the parameter-less BCL message for a partially fitting `WriteArray` - Throw `ObjectDisposedException` when creating views on a disposed memory-mapped file - Throw `UnauthorizedAccessException` when the stream of the `FileSystemStream`-based `CreateFromFile` overload does not support the requested mapping access - Range-check the view access enum in `CreateViewAccessor`/`CreateViewStream` - Restructure the view backing into `MemoryMappedViewBacking` / `StreamViewBacking` (lock owned by the class) / `CopyOnWriteViewBacking`, remove dead zero-size branches and share the negative-argument and access-range validation helpers - Restore the UTF-8 byte-order mark on package files and share the `CreateMappedFile` test helper - Docs: describe the page privatization, the Windows-flavoured exception messages, and the base-typed `UnmanagedMemoryStream` member limitation
- Rename the Seek parameter to `origin`, matching the `Stream` declaration - Remove two redundant int casts in `CopyOnWriteViewBacking` - Convert the test structs to record structs, which implement `IEquatable<T>` - Suppress S2325 on the `MemoryMappedFile` extension property: a C# extension property cannot be static (same false positive as the suppressed CA1822) - Suppress S3874 on `Read<T>`/`Write<T>` of `IMemoryMappedViewAccessor`: the out/ref modifiers deliberately mirror the BCL `UnmanagedMemoryAccessor` signatures, so user code works unchanged against the real and mocked API
SonarCloud and Codacy run different versions of the same rule and resolve the base class declaration differently: the direct base `UnmanagedMemoryStream.Seek` kept the historical `loc` parameter name, while the root `Stream.Seek` declares `origin`. Keep `origin` (the root contract, used by all other streams in this repository) and suppress the rule at this single call site.
The parity test `MemoryMappedFileSystemViewStream_EnsureParityWith_MemoryMappedViewStream` compares parameter names and failed on every platform because the mock's `Seek` deliberately uses `origin` (the root `Stream.Seek` contract used by all other streams here) while the BCL's `UnmanagedMemoryStream.Seek` kept the historical `loc`. Exclude `Seek` from this parity check, consistent with how other BCL divergences are handled. `CreateViewAccessor_WithOffsetBeyondCapacity_ShouldThrowUnauthorizedAccessException` and `CreateFromFile_WithReadOnlyStream_AndReadWriteAccess_ShouldThrowUnauthorizedAccessException` encode Windows-specific exception semantics that the mock mirrors, so they failed against the real file system on Linux and macOS. Guard both with `Skip.IfNot(FileSystem is MockFileSystem || Test.RunsOnWindows, ...)`, matching the existing pattern for platform-specific behavior.
All fixes were verified by running each scenario against the real BCL MemoryMappedFile side by side with the mock: - Reject structs containing object references in Read<T>/Write<T>/ReadArray/WriteArray with the BCL ArgumentException, instead of reinterpreting raw file bytes as object references (undefined behavior) - Stride array elements by the aligned size (sizes other than 1 and 2 rounded up to a multiple of 4) in ReadArray/WriteArray, matching the BCL layout, element count (spaceLeft / alignedSize), bounds check (count * alignedSize), and untouched padding bytes between elements - Track disposed state on view accessors and view streams: operations on a disposed view now throw ObjectDisposedException with the BCL messages and CanRead/CanWrite/CanSeek turn false, instead of silently reading and writing through the still-open shared backing - Allow CopyOnWrite mappings over a read-only stream in the FileSystemStream-based CreateFromFile: a copy-on-write mapping never writes to the file, so the BCL only requires read access - Validate the inheritability argument of the FileSystemStream-based CreateFromFile on the mock path (ArgumentOutOfRangeException) and throw ArgumentNullException instead of NullReferenceException for a null fileStream - Flush view writes to the underlying file on Flush() and view dispose instead of on every write: flushing the MockFileSystem stream copies the complete file content, so the per-write flush made every single write cost O(fileLength); views share one stream, so coherence between views is unaffected, and the new visibility timing is documented - Document the leaveOpen stream-lifetime simplification: disposing a caller-owned stream while views are open makes the mock views unusable, whereas real views keep operating on the mapped pages - Skip CopyOnWriteView_BeforeOwnWrite_ShouldSeeWritesFromOtherViews on the real file system on macOS: POSIX leaves it unspecified whether a MAP_PRIVATE view sees later writes made through other views; macOS snapshots the pages eagerly, while Windows and Linux (which the mock mirrors) privatize them lazily
- Suppress S927 on the span-based Read/Write overrides: the base declarations conflict, UnmanagedMemoryStream kept the historical `destination`/`source` parameter names while the root `Stream` contract declares `buffer`, which all other streams of this repository use (same resolution as for `Seek`) - Move the CA1822/S2325 pragma suppression directly onto the `MemoryMappedFile` extension property; an extension property cannot be static, so the rule is a false positive - Use `Array.Empty<int>()` instead of `new int[0]` in the zero-count WriteArray test
vbreuss
force-pushed
the
feat/memory-mapped-files
branch
from
d6a20d8 to
1ab25e8
Compare
The fallback for RuntimeHelpers.IsReferenceOrContainsReferences on netstandard2.0/2.1 must inspect non-public instance fields, because the object references of a struct usually sit in private fields; the reflection is read-only metadata inspection and mirrors exactly what the runtime helper checks on the modern frameworks.
- Persist a changed length in FileStreamMock.SetLength, so growing a mapping to a larger capacity reaches the file even when no view ever writes - Keep disposing views safe after a caller-owned stream (leaveOpen: true) was disposed: the flush is skipped and the reference to the shared backing is still released - Reject execute views on mappings created without execute access with the UnauthorizedAccessException of the BCL on Windows - Only special-case MemoryMappedFileAccess.Read with the ArgumentException when the capacity exceeds the file size; ReadExecute now throws UnauthorizedAccessException like the BCL - Detect the real file system via the side-effect-free type-name check used by the other companion packages instead of probing FileInfo.New, which registered a phantom call in the statistics of the mocked file system - Align the argument validation order of both CreateFromFile overloads with the BCL (capacity before access, empty file before inheritability) and drop the unreachable capacity re-check in the mock constructor - Copy contiguous ReadArray/WriteArray ranges with a single bulk copy and read copy-on-write views without private pages in one pass instead of per 4096-byte page - Document the Windows exception semantics and the managed struct sizing as known divergences of the mock - Suppress MA0041 alongside CA1822/S2325 on the MemoryMappedFile extension property (same false positive, deprecated duplicate of CA1822) - Revert the unrelated whitespace-only .editorconfig changes
…les mock - Track the pending write range of FileStreamMock as a contiguous union and flush it before a write to a disjoint position, like the real FileStream flushes its buffer when repositioning; this keeps every unflushed write intact when another stream on the same file flushes, instead of silently reverting all but the most recent one - Clamp the range replayed in OnBytesChanged to the new container content, so another stream truncating the file no longer throws an ArgumentException out of its Flush - Set the content-changed flag in SetLength only after the base call succeeded, so a failed SetLength no longer causes a spurious flush with watcher notification and LastWriteTime update - Open the backing stream of a path-based CreateFromFile with FileShare.None to match the exclusivity of the BCL, and base the flush-visibility test on a caller-owned shared stream instead - Open the file of a copy-on-write mapping read-only when running on .NET Framework, whose BCL only requests read access for CopyOnWrite (e.g. mapping a read-only file succeeds) - Guard the shared-backing reference count against resurrection: once the count drops to zero it is atomically marked as released, so a CreateView racing Dispose either gets a live view or an ObjectDisposedException and the stream is disposed exactly once - Validate position, open state and readability in Read<T>/Write<T> before the reference check of SizeOf<T>, matching the BCL order (ReadArray/WriteArray already did) - Throw a deliberate NotSupportedException for a capacity above 2 GB instead of leaking the ArgumentOutOfRangeException of the internal MemoryStream, and document the limit - Release the factory-created backing stream of the wrapper in a finally block, so it does not stay locked when disposing the real memory-mapped file throws - Deduplicate the view-dispose block, the read/write guards and the empty-file capacity validation into shared helpers - Document that truncating the backing stream under a live mapping is not rejected by the mock, and list the MemoryMappedFiles package in the README, the getting-started docs and the copilot instructions
…ryMappedFiles mock - Track the pending writes of FileStreamMock as exact, merged [start, end) ranges instead of one contiguous union, so the OnBytesChanged replay only covers bytes this stream actually wrote: previously a write while a BeginWrite was still pending (where the disjoint-position flush had nothing to persist yet) merged never-written gaps into the range, and the first write on a FileMode.Create/Truncate stream extended it down to the seeded position 0, so the replay clobbered content other streams had flushed in between - Truncate the local stream to the new container content in OnBytesChanged, so a stream no longer resurrects content another stream truncated away when it flushes afterwards; a pending write beyond the new end of the file zero-fills the gap, like the real file system - Flush SetLength immediately, like the real FileStream, which resizes the file at the time of the call: a pending truncation can no longer be silently reverted by another stream's flush, and growing a file to the capacity of a memory-mapped file is visible to other handles while the mapping is alive instead of only after it is disposed - Zero-fill reads beyond the end of the backing stream in the view backings, so a view stream over a caller-truncated backing returns the documented zeros instead of a premature end of stream, and the per-page path of a copy-on-write view no longer reports bytes it never read while leaving caller-buffer garbage in the truncated range - Open the backing stream of a path-based CreateFromFile with FileShare.Read on modern .NET, matching the BCL, which only uses FileShare.None on the .NET Framework; concurrent readers are now permitted while a mapping is alive on both the real and the mocked file system - Correct the stale project and package counts in the copilot instructions, which still stated 6 although the MemoryMappedFiles package is the 7th
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new
Testably.Abstractions.MemoryMappedFilescompanion package that abstractsSystem.IO.MemoryMappedFiles.MemoryMappedFilebehindIFileSystem, so memory-mapped-file code can be tested against the in-memoryMockFileSystem.Usage
The abstraction is exposed as the extension property
fileSystem.MemoryMappedFile(note: no parentheses), returning anIMemoryMappedFileFactory:On the real file system every call forwards to the underlying base class library (BCL) implementation; on the mock the views are built directly over the in-memory file bytes.
Public surface
IMemoryMappedFileFactory- theCreateFromFileoverloads (path-based andFileSystemStream-based, so they work against both the real and the mocked file system), plusCreateNew/CreateOrOpen/OpenExisting.IMemoryMappedFile- theCreateViewAccessor/CreateViewStreamoverloads.IMemoryMappedViewAccessor- the full read/write surface ofMemoryMappedViewAccessor.MemoryMappedFileSystemViewStream- a dedicatedStream-derived type returned byCreateViewStream, mirroring theFileSystemStreampattern: everyStreammember is delegated to the wrapped stream, and the view-stream-specificPointerOffsetandCapacityare added on top.Parity tests assert the abstraction matches the BCL surface, excluding only the unsafe handle / security members (
SafeMemoryMappedFileHandle,SafeMemoryMappedViewHandle,MemoryMappedFileSecurityand theSafeFileHandle-basedCreateFromFileoverload), mirroring how theSafeFileHandle-basedFileStreamconstruction is excluded elsewhere.Intentionally not supported on the mock
CreateNew,CreateOrOpenandOpenExistingoperate on operating-system shared memory (named or anonymous) that is not backed by a file. They forward normally on the real file system but throwNotSupportedExceptionon the mock.BCL-faithful behavior on the mock
MemoryMappedViewAccessor/MemoryMappedViewStream.CopyOnWriteviews operate on a private copy of the mapped bytes: writes are visible within the view but are never persisted to the underlying file, nor visible to other views.ArgumentException, matching the BCL.System.IO.MemoryMappedFiles#1042