[Backport 1.16] memory hotplug, vsock and CPU template fixes (#6076, #6100, #6120) - #6182
Merged
ilstam merged 10 commits intoSep 3, 2026
Merged
Conversation
ilstam
added
the
Status: Awaiting review
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## firecracker-v1.16 #6182 +/- ##
=====================================================
+ Coverage 83.04% 83.07% +0.03%
=====================================================
Files 277 277
Lines 30196 30228 +32
=====================================================
+ Hits 25075 25111 +36
+ Misses 5121 5117 -4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
zulinx86
previously approved these changes
Sep 3, 2026
zulinx86
left a comment
zulinx86
left a comment
Contributor
There was a problem hiding this comment.
approved the CPU template handling fix on 6.18. please obtain an approval from the person who can review the remaining part.
Manciukic
previously approved these changes
Sep 3, 2026
Memory hotplug resize requests used usize, allowing a MiB value to overflow when converted to bytes. Use u32 for requested_size_mib throughout the API, VMM, and virtio-mem update path. The resulting maximum of just under 4 PiB always fits in a u64 byte count, so explicit conversion validation is unnecessary. Document the API range and test the accepted and rejected boundaries. [cherry picked from commit aad7587] Signed-off-by: Jack Thomson <jackabt@amazon.com>
The previous commit bounded requested_size_mib, but the sizes accepted by PUT /hotplug/memory were still usize. mib_to_bytes shifts left by 20 and discards the bits shifted out, so a large enough total_size_mib wrapped to a smaller byte count: 2^44 MiB became a 0-byte region. Use u32 for total_size_mib, block_size_mib and slot_size_mib, matching requested_size_mib and the balloon's amount_mib. Since u32::MAX MiB is just under 4 PiB, every accepted value fits a u64 byte count, so the wrap becomes unrepresentable at deserialization rather than something to check for later. Route the conversions through u32_mib_to_bytes and bytes_to_u32_mib, which widen before shifting, so the MiB conversion is no longer open-coded. Document the ranges in the API spec and record the behaviour change in the changelog. [cherry picked from commit 031a5fd] Signed-off-by: Jack Thomson <jackabt@amazon.com>
The resume kick unconditionally armed `pending_event_ack`, the gate that suppresses RX delivery until the guest acknowledges a TRANSPORT_RESET event. On a bare pause/resume no such event was ever published, so the guest could never acknowledge it, the gate never cleared, and every new host-initiated connection made after the resume hung forever. TX kept working, which made the failure look RX-specific. Persist `pending_event_ack` in the vsock frontend snapshot state and make the resume kick respect the restored value instead of arming it: the event queue is only re-signaled when a reset ack is actually outstanding, and a bare resume leaves RX untouched. [Ilias: This changes the layout of VsockFrontendState, which is a snapshot format change on an already released format. SNAPSHOT_VERSION is therefore bumped to 11.0.0 here as well. Snapshots created by 1.16.2 can not be loaded by 1.16.0 or 1.16.1, and snapshots created by those versions can not be loaded by 1.16.2.] [cherry picked from commit a34d51c] Signed-off-by: Dov Alperin <git@dov.dev>
Add an integration test that pauses and resumes a microVM without snapshotting and checks that fresh host-initiated (RX direction) connections still work afterwards, alongside guest-initiated ones. [Conflicts: context only, on main the following test is pinned with @pin_guest_kernel(ACPI_GUEST_KERNELS), which does not exist on this branch.] [cherry picked from commit 2c46048] Signed-off-by: Dov Alperin <git@dov.dev>
Split CPU template application into independent CPUID and MSR transformations while keeping CpuConfiguration::apply_template() as a compatibility wrapper. This is a preparatory refactor for a follow-up fix. The fix needs to apply CPUID and MSR template modifiers at different points during vCPU configuration. Exposing the transformations independently lets the follow-up commit correct the ordering without duplicating template logic. Add focused tests for both transformations, including exact output values and unsupported-register errors. No functional change is intended. [cherry picked from commit d46a5a2] Signed-off-by: Takahiro Itazuri <itazur@amazon.com>
Extract CPUID setup, MSR setup, and the remaining Linux boot state from KvmVcpu::configure(), and keep configure() as a compatibility wrapper. This is a preparatory refactor for a follow-up fix. The fix needs to install CPUID on every vCPU before reading CPUID-dependent MSRs, such as IA32_ARCH_CAPABILITIES, from vCPU 0. Exposing the operations independently lets the follow-up commit correct the ordering. Return the installed KVM CPUID from configure_cpuid() so MSR snapshot bookkeeping continues to use the exact per-vCPU CPUID and preserves the existing MSR save-list order. No functional change is intended. [cherry picked from commit b851857] Signed-off-by: Takahiro Itazuri <itazur@amazon.com>
KVM's feature MSR series [1] made userspace accesses to
CPUID-dependent MSRs subject to architectural existence checks.
Userspace must install guest CPUID with KVM_SET_CPUID{,2} before
reading such MSRs with KVM_GET_MSRS. If guest CPUID does not advertise
a feature, KVM returns zero when userspace reads the corresponding MSR
that KVM advertised as supported.
The KVM maintainer deferred documenting this behavior until KVM's
internal MSR handling is fully converted to KVM_MSR_RET_UNSUPPORTED [3].
The documentation update has not landed as of Linux 6.18.
This behavior landed in Linux 6.13 through commit [2], and is therefore
present in Firecracker's supported 6.18 host kernel. Firecracker read
CPU template MSRs from vCPU 0 before installing guest CPUID. On T2CL,
KVM_GET_MSRS consequently returned zero for IA32_ARCH_CAPABILITIES. The
template's passthrough mask preserved the zero IBRS_ALL bit, and the
later KVM_SET_MSRS overwrote KVM's post-CPUID value. The guest did not
observe eIBRS support and selected retpoline.
Correct CPU configuration ordering as follows:
1. Apply CPUID modifiers.
2. Install each vCPU's normalized CPUID with KVM_SET_CPUID2.
3. Retrieve CPUID-dependent MSRs from vCPU 0 with KVM_GET_MSRS.
4. Apply MSR modifiers.
5. Add Linux boot MSRs and update CPUID-derived MSR snapshot
bookkeeping.
6. Install each vCPU's MSRs with KVM_SET_MSRS.
[1]: https://lore.kernel.org/all/20240802185511.305849-1-seanjc@google.com/
[2]: torvalds/linux@a5d5638
[3]: https://lore.kernel.org/kvm/Zivh0IaAmHsEOLFc@google.com/
[cherry picked from commit 4681a67]
Signed-off-by: Takahiro Itazuri <itazur@amazon.com>
Remove the CpuConfiguration and KvmVcpu compatibility wrappers left after wiring staged CPUID and MSR configuration into the x86 cold boot path. Update x86 tests to exercise the staged operations directly. CpuConfiguration remains available for CPU configuration dumps, and VcpuConfig remains available for aarch64 boot configuration. [cherry picked from commit d9edec6] Signed-off-by: Takahiro Itazuri <itazur@amazon.com>
Add an integration test that verifies CPU templates preserve passthrough bits from a CPUID-gated MSR. Compare a live IA32_ARCH_CAPABILITIES value from a no-template guest against a custom template that flips bit 0 and passes all other bits through. On Linux 6.13 and later, KVM returns zero when userspace reads IA32_ARCH_CAPABILITIES before guest CPUID advertises it [1]. The test therefore covers the requirement that Firecracker install guest CPUID before reading the base MSR values used for template application. [1]: torvalds/linux@a5d5638 [The test fixture consolidation is not backported, so pin_guest_kernel() and GUEST_KERNEL_DEFAULT do not exist here and the test takes the guest_kernel_default fixture instead. Also, framework.utils_cpu_templates only provides get_cpu_template_name on this branch.] [cherry picked from commit 4aa9bb8] Signed-off-by: Takahiro Itazuri <itazur@amazon.com>
Refresh the T2CL baselines for Cascade Lake and Ice Lake after Firecracker sets each vCPU's CPUID before retrieving MSRs to be modified by CPU templates. The corrected output exposes IA32_SPEC_CTRL and IA32_TSX_CTRL and records the post-CPUID IA32_ARCH_CAPABILITIES values instead of zero across guest kernels 5.10, 6.1, and 6.18. For each CPU model, all three MSR results match the corresponding Linux 6.1 host baselines. This confirms that the ordering fix restores the established T2CL behavior on Linux 6.18 hosts. [Dropped the two *_6.18host_6.18guest.csv baseline updates, since 6.18 guest kernels are not supported on this branch.] [cherry picked from commit 9c493b4] Signed-off-by: Takahiro Itazuri <itazur@amazon.com>
ilstam
force-pushed
the
backport/v1.16-fixes
branch
from
September 3, 2026 11:09
fbe30bc to
d890461
Compare
Manciukic
approved these changes
Sep 3, 2026
ilstam
enabled auto-merge (rebase)
September 3, 2026 11:43
zulinx86
approved these changes
Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Backport to
firecracker-v1.16of three fixes from theUnreleased→Fixedsection of
main's changelog. The three series are independent of each other,and each carries its own changelog entry into the existing
## [1.16.2]section.
#6076 —
memory hotplug sizes wrapping from MiB to bytes
memory-hotplug: bound requested size to u32(clean cherry-pick ofaad75879f)memory-hotplug: bound device config sizes to u32(031a5fd6b) —src/vmm/src/utils/mod.rshas notestsmodule on this branch, since thealign_up!/align_down!macro refactor is not backported, so the newconversion tests are wrapped in a new one.
#6100 — vsock
RX permanently suppressed after a bare pause/resume
fix(vsock): do not arm the TRANSPORT_RESET RX gate on bare resume(
a34d51c93) — additionally bumpsSNAPSHOT_VERSION, see below.test(vsock): cover host-initiated connections after bare resume(
2c4604808, clean)#6120 — guest
CPUID must be installed before reading CPUID-gated MSRs
vmm: x86: cpu_config: split CPU template application stages(d46a5a210,clean)
vmm: x86: vcpu: split boot configuration stages(b85185750, clean)vmm: x86: fix CPUID-dependent MSR read ordering(4681a67f7, clean)vmm: x86: remove obsolete CPU configuration path(d9edec6f4) —EntryPointhas nosetup_headerfield on this branch, since bzImage bootsupport is not backported, so that field is dropped from the updated tests.
tests: x86: verify CPUID-gated MSR passthrough(4aa9bb83c) — the testtakes the
guest_kernel_defaultfixture, sincepin_guest_kernel()andGUEST_KERNEL_DEFAULTonly exist onmain.test: x86: update T2CL MSR baselines(9c493b4da) — dropped the two*_6.18host_6.18guest.csvupdates, since 6.18 guest kernels are notsupported on this branch.
Every adaptation is also noted in the commit message of the commit it applies
to.
Reason
Three bug fixes that are already on
mainand apply to a supported releasebranch:
the MiB sizes accepted by
PUT /hotplug/memoryandPATCH /hotplug/memorywere
usizeand shifted left by 20 to get a byte count, discarding the bitsshifted out. A large enough size therefore wrapped:
2^44MiB was acceptedas a 0-byte region instead of being rejected.
bare pause/resume (
PATCH /vmwithPausedthenResumed, no snapshotinvolved) armed the
TRANSPORT_RESETRX gate even though no reset event hadbeen sent. The guest could never acknowledge it, so every new host-initiated
vsock connection made after the resume hung forever. TX kept working, which
makes the failure look RX-specific.
since Linux 6.13, KVM returns zero when userspace reads a CPUID-gated MSR
before guest CPUID advertises it. On 6.18 hosts Firecracker read the CPU
template MSRs before installing guest CPUID, so passthrough bits preserved
that zero: T2CL guests did not observe eIBRS and selected retpoline.
One consequence to be aware of when taking this into a patch release:
#6100 adds a
field to the persisted vsock device state, and the bitcode encoding used for
the microVM state does not allow backwards compatible changes. The same commit
therefore bumps
SNAPSHOT_VERSIONto 11.0.0, so snapshots do not interoperatebetween 1.16.2 and 1.16.0/1.16.1. This is recorded in the changelog's
Changedsection.
Testing
tools/devtool checkbuild --allpasses (x86_64 and aarch64).tools/devtool checkstylepasses.cargo test -p vmm: the newtest_persist_pending_event_ackand thesnapshot::andutils::tests pass. The failures that remain(
devices::virtio::mem::device, the twovsock::unix::muxermetrics tests,arch::x86_64::vcpu::tests::test_set_tscand thepersist::tests failing onKVM
SetUserMemoryRegion) reproduce identically on pristinefirecracker-v1.16, so they are pre-existing and unrelated.git range-diffagainst the originalmaincommits shows no code driftbeyond the adaptations documented above.
main.License Acceptance
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.PR Checklist
tools/devtool checkbuild --allto verify that the PR passesbuild checks on all supported architectures.
tools/devtool checkstyleto verify that the PR passes theautomated style checks.
how they are solving the problem in a clear and encompassing way.
in the PR.
CHANGELOG.md.Runbook for Firecracker API changes.
integration tests.
TODO.rust-vmm.