[VFIO] Chapter 2: Basic implementation by ShadowCurse · Pull Request #6055 · firecracker-microvm/firecracker

ShadowCurse · 2026-07-23T11:26:46Z

Changes

This is a follow up for the #6042 PR. Here we add basic VFIO implementation and testing infrastructure.

Reason

The original VFIO PR #5870 is quite big and this is the attempt to ease the reviewers burden of giving the review.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.

PR Checklist

This functionality cannot be added in rust-vmm.

codecov · 2026-07-23T11:34:02Z

Codecov Report

❌ Patch coverage is 18.76173% with 433 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.87%. Comparing base (b651dd0) to head (7fcda5c).

Files with missing lines	Patch %	Lines
src/vmm/src/vfio.rs	18.06%	313 Missing ⚠️
src/vmm/src/device_manager/pci_mngr.rs	4.44%	43 Missing ⚠️
src/vmm/src/device_manager/mod.rs	8.00%	23 Missing ⚠️
src/vmm/src/rpc_interface.rs	14.81%	23 Missing ⚠️
src/vmm/src/pci/msix.rs	0.00%	10 Missing ⚠️
src/vmm/src/resources.rs	35.71%	9 Missing ⚠️
src/vmm/src/lib.rs	27.27%	8 Missing ⚠️
src/vmm/src/builder.rs	78.94%	4 Missing ⚠️

Additional details and impacted files

@@               Coverage Diff                @@
##           feature/vfio    #6055      +/-   ##
================================================
- Coverage         82.91%   81.87%   -1.05%     
================================================
  Files               280      280              
  Lines             31932    32328     +396     
================================================
- Hits              26478    26467      -11     
- Misses             5454     5861     +407

Flag	Coverage Δ
5.10-m5n.metal	`81.99% <18.76%> (-1.15%)`	⬇️
5.10-m6a.metal	`81.33% <18.76%> (-1.18%)`	⬇️
5.10-m6g.metal	`78.88% <18.76%> (-1.16%)`	⬇️
5.10-m6i.metal	`81.98% <18.76%> (-1.15%)`	⬇️
5.10-m7a.metal-48xl	`81.33% <18.76%> (-1.18%)`	⬇️
5.10-m7g.metal	`78.88% <18.76%> (-1.16%)`	⬇️
5.10-m7i.metal-24xl	`81.96% <18.76%> (-1.15%)`	⬇️
5.10-m7i.metal-48xl	`81.97% <18.76%> (-1.15%)`	⬇️
5.10-m8g.metal-24xl	`78.87% <18.76%> (-1.16%)`	⬇️
5.10-m8g.metal-48xl	`78.88% <18.76%> (-1.16%)`	⬇️
5.10-m8i.metal-48xl	`81.96% <18.76%> (-1.15%)`	⬇️
5.10-m8i.metal-96xl	`81.96% <18.76%> (-1.15%)`	⬇️
5.10-m9g.metal-48xl	`78.87% <18.76%> (-1.16%)`	⬇️
6.1-m5n.metal	`82.01% <18.76%> (-1.15%)`	⬇️
6.1-m6a.metal	`81.36% <18.76%> (-1.18%)`	⬇️
6.1-m6g.metal	`78.87% <18.76%> (-1.16%)`	⬇️
6.1-m6i.metal	`82.01% <18.76%> (-1.15%)`	⬇️
6.1-m7a.metal-48xl	`81.35% <18.76%> (-1.18%)`	⬇️
6.1-m7g.metal	`78.88% <18.76%> (-1.15%)`	⬇️
6.1-m7i.metal-24xl	`82.02% <18.76%> (-1.16%)`	⬇️
6.1-m7i.metal-48xl	`82.02% <18.76%> (-1.15%)`	⬇️
6.1-m8g.metal-24xl	`78.87% <18.76%> (-1.16%)`	⬇️
6.1-m8g.metal-48xl	`78.87% <18.76%> (-1.16%)`	⬇️
6.1-m8i.metal-48xl	`82.02% <18.76%> (-1.15%)`	⬇️
6.1-m8i.metal-96xl	`82.02% <18.76%> (-1.15%)`	⬇️
6.1-m9g.metal-48xl	`78.87% <18.76%> (-1.16%)`	⬇️
6.18-m5n.metal	`82.01% <18.76%> (-1.14%)`	⬇️
6.18-m6a.metal	`81.36% <18.76%> (-1.18%)`	⬇️
6.18-m6g.metal	`78.98% <18.76%> (-1.15%)`	⬇️
6.18-m6i.metal	`82.01% <18.76%> (-1.15%)`	⬇️
6.18-m7a.metal-48xl	`81.35% <18.76%> (-1.19%)`	⬇️
6.18-m7g.metal	`78.98% <18.76%> (-1.16%)`	⬇️
6.18-m7i.metal-24xl	`82.02% <18.76%> (-1.15%)`	⬇️
6.18-m7i.metal-48xl	`82.02% <18.76%> (-1.16%)`	⬇️
6.18-m8g.metal-24xl	`78.97% <18.76%> (-1.16%)`	⬇️
6.18-m8g.metal-48xl	`78.98% <18.76%> (-1.16%)`	⬇️
6.18-m8i.metal-48xl	`82.02% <18.76%> (-1.15%)`	⬇️
6.18-m8i.metal-96xl	`82.02% <18.76%> (-1.15%)`	⬇️
6.18-m9g.metal-48xl	`78.98% <18.76%> (-1.15%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ilstam · 2026-07-29T10:54:32Z

+    pub id: String,
+    /// Host identifier for the PCI device
+    #[serde(
+        serialize_with = "serialize_sbdf_as_str",


Isn't this type a u32? Why serialize it as a string?

because it is nicer to look at 0000:aa:bb.c string rather than raw u32 value. This also how the sbdf will look when the user specifies it in the first place.

For my understanding, who looks at serialised state?

"Who" is a good question I don't have answer for. But we have GetFullVmConfig call that returns the VM config and there these vfio configs will be used and presented to the user.

ilstam · 2026-07-29T10:54:55Z

+}
+
+impl VfioConfigs {
+    /// Add config to the set. Overwrite existing one if


I know we have this weird "overwrite" logic for virtio devices, but is there really a reason to add this here too?

For consistency I would say yes.

Normally I favour consistency and I agree it's a good reason, but did anyone ever really use this and will anybody even notice that it doesn't work for VFIO?

I don't think anyone uses this "overwrite" feature, and I would like it to be gone, but we cannot do this without a major release unfortunately.

I'm not suggesting that we remove it from existing APIs, I'm suggesting we're not adding it to new ones.

ilstam · 2026-07-29T10:55:56Z

+
+        // Mismatched ids
+        let body = r#"{
+            "id": "bar",


Again, I know we're doing this weird duplication for virtio devices, but do we also need to do it for new endpoints?

what do you mean here? What duplication?

Specifying the id in two places (see line 31)

oh this.. again, it just keeps the API consistent

ilstam · 2026-07-29T10:56:17Z


+  /vfio/{id}:
+    put:
+      summary: Creates or updates a VFIO passthrough device. Pre-boot only.


This still says "Pre-boot only"

because this is correct. There is no hot-plug yet in this PR

ilstam · 2026-07-29T10:56:55Z

 /// Utility functions and struct
 pub mod utils;
+/// VFIO device configuration and emulation
+pub mod vfio;


Is this "vfio: create module" commit a joke? :)

no. I can merge it with the next commit if you want.

ilstam · 2026-08-05T10:45:50Z

    Ok((areas, bar_hole_infos))
 }

+fn vfio_map_bar_mapping(


As discussed offline there's no reason for adding IOMMU mappings for BARs until we plan to support P2P DMA.

The naming was inconsistent from a time the areas were called regions. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Fix remaining places where emulated areas were referred to as `holes`. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

This will help in the next commits with the dispatch logic for the BarDevice trait implementation. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Initially it was added because it is a companion region in the MSIX capability. But on a secondary inspection, this was wrong. Since Firecracker never triggers interrupts for the VFIO device (they are directly wired from the device to the KVM), the PBA that the Firecracker emulates is never changed. This means if guest ever wants to read PBA, it would read all zeroes regardless if the device's PBA actually contains some data. Fortunately it seems guest kernel/drivers hardly ever read PBA in general, so this is not an issue right now. As an additional note, since PBA was treated as an EmulatedArea before, it was checked for collisions with regions reported by SparseMmap capability. As it appears, SparseMmap capability only carves the space for the MSIX table, so that check was incorrect. Also move `arrayvec` back to be an optional dependency only used for `gdb` feature. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Combine all previous steps to create a VfioDevice type. One minor change is the removal of `VfioBarMappings` wrapper. Instead just store the vector of `VfioBarMapping` directly inside `VfioDevice` and unmap them in the `VfioDevice::drop`. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Implement emulation for the BAR accesses of the VfioDevice. The emulation only needs to happen for Msix table/pba, but since they may be smaller than host page region, the code has to handle full host page sized regions. If access does not hit any Msix table/pba, it is forwarded to the device directly. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Implement emulation logic for the guest accesses to the configuration space of the device. Emulation only touches some registers like BARs, Msix, masked registers. Everything else is forwarded to the device. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Rename vfio_dellocate_... to vfio_deallocate_... Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Currently BAR relocation is not implemented for VFIO, but it still can happen due to the changes to the `Bars` type. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

We already ensure that the Msix is present in the device. The `vfio_calculate_bar_areas` was one weird place that still was accepting it as an `Option` Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add utility functions for DMA mapping guest memory to the device. This must be done only once on first device setup/terdown. PciMng will be handling this in the future commits. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add functions for creation of KVM VFIO device and VFIO container. These will need to be create once on the first device init. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add logic to the PciDevices to create new VFIO devices. As an additional step in VFIO device setup, guest RAM regions are mapped into the VFIO container's IOMMU so the device can DMA directly to guest memory. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Since `device-passthrough` API is now connected, we need to not set dummy values for it before starting the VM. Otherwise it the VM startup will fail. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Current device passthrough implementation has some restrictions: - Does not work without PCI since VFIO devices are PCI devices - Does not work with virtio-mem device since we don't update DMA mappings on hot-plug/unplug - Does not work with virtio-balloon since it can `fadvise` on memory In order to prevent VMs being launched with invalid configurations, implement multiple checks for invalid configurations: - At API level, prevent adding of incompatible combinations (VFIO after balloon/mem or in reverse) - At vm creation or snapshot restoraton since they get VmResources from other sources. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Passhtrough device state is opaque to the VMM and cannot be serialized or restored. Add these devices to the list of snapshot-incompatible devices so that snapshot requests are rejected with a clear error instead of producing a corrupt snapshot. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

VFIO devices will use pread64/pwrite64 syscalls (from vfio-ioctls) to interact with BARs during runtime. Add them to the VPU thread syscall lists. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add devtool options for preparing a PCI device for VFIO passthrough testing. `--vfio-nvme-device` accepts a block device path (e.g. /dev/nvme1n1) or a PCI SBDF, resolves it to a PCI device, binds it to vfio-pci, and passes the SBDF and sysfs path to the test container via environment variables. `--first-vfio-nvme-device` is a fallback that searches for the first NVMe device already bound to vfio-pci if the targeted search fail's. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add an integration tests that verify passthrough with a physical NVMe device. Tests are gated behind the `vfio` pytest mark and FC_VFIO_PCI_SBDF environment variable so they only run when a suitable device is available. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

VFIO tests need exclusive access to the passthrough device, so they cannot run in parallel with other tests. Add a separate Buildkite step in the PR pipeline that runs only the vfio-marked tests, similar to the existing performance step. CI instances will have an additional 1GB NVMe device at /dev/nvme1n1 for this purpose. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add docs/device_passthrough.md covering how device passthrough works in Firecracker. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add a changelog entry for the new device passthrough feature. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

ShadowCurse force-pushed the vfio_part_2 branch 2 times, most recently from b4f901d to e08d1c6 Compare July 23, 2026 13:44

ShadowCurse self-assigned this Jul 23, 2026

ShadowCurse marked this pull request as ready for review July 23, 2026 13:54

ShadowCurse added Status: Awaiting review Indicates that a pull request is ready to be reviewed Type: Enhancement Indicates new feature requests labels Jul 23, 2026

ShadowCurse force-pushed the vfio_part_2 branch 4 times, most recently from 8398519 to 60c0824 Compare July 24, 2026 14:56

ilstam reviewed Jul 29, 2026

View reviewed changes

ShadowCurse mentioned this pull request Jul 29, 2026

[VFIO] add basic implementation #5870

Closed

11 tasks

ShadowCurse changed the base branch from main to feature/vfio July 29, 2026 12:38

ShadowCurse force-pushed the vfio_part_2 branch 6 times, most recently from 77db7fc to 537e549 Compare August 4, 2026 15:36

ilstam reviewed Aug 5, 2026

View reviewed changes

ShadowCurse mentioned this pull request Aug 6, 2026

[VFIO] Chapter 2, Section 2: Caps and Bars #6085

Merged

11 tasks

ShadowCurse mentioned this pull request Aug 14, 2026

[VFIO] Chapter 2, Section 3: BAR areas and mappings #6114

Merged

11 tasks

ShadowCurse force-pushed the vfio_part_2 branch 2 times, most recently from 613e210 to 85c8fe0 Compare August 18, 2026 10:33

ShadowCurse force-pushed the vfio_part_2 branch 2 times, most recently from d2fc17d to 8c59b5a Compare September 2, 2026 16:53

ShadowCurse requested review from Manciukic, marco-marangoni and micz010 as code owners September 2, 2026 16:53

ShadowCurse force-pushed the feature/vfio branch from fb33146 to 94812b4 Compare September 2, 2026 16:57

ShadowCurse force-pushed the vfio_part_2 branch from 8c59b5a to e07b5cd Compare September 2, 2026 17:08

ShadowCurse mentioned this pull request Sep 2, 2026

[VFIO] Chapter 2, Section 4: VfioDevice and its traits #6183

Merged

11 tasks

ShadowCurse force-pushed the feature/vfio branch from 94812b4 to b651dd0 Compare September 7, 2026 15:20

ShadowCurse added 3 commits September 7, 2026 16:22

vfio: fmt: rename emulated area flags bitfield

3b261ef

The naming was inconsistent from a time the areas were called regions. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

vfio: fmt: rename holes to emulated area

c24958e

Fix remaining places where emulated areas were referred to as `holes`. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

vfio: add more info to the VfioBarEmulatedArea

1fe28cb

This will help in the next commits with the dispatch logic for the BarDevice trait implementation. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

ShadowCurse force-pushed the vfio_part_2 branch from e07b5cd to a559fe1 Compare September 7, 2026 16:00

ShadowCurse added 19 commits September 11, 2026 18:00

vfio: fmt: fix typo in function name

ab8d9a5

Rename vfio_dellocate_... to vfio_deallocate_... Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

vfio: add a TODO about BAR relocation

6f4d824

Currently BAR relocation is not implemented for VFIO, but it still can happen due to the changes to the `Bars` type. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

vfio: make msix_cap non optional arg in the vfio_calculate_bar_areas

e30bdea

We already ensure that the Msix is present in the device. The `vfio_calculate_bar_areas` was one weird place that still was accepting it as an `Option` Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

vfio: add dma_map/unmap_guest_memory

d70d179

Add utility functions for DMA mapping guest memory to the device. This must be done only once on first device setup/terdown. PciMng will be handling this in the future commits. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

vfio: create vfio kvm device/container

c1f89b5

Add functions for creation of KVM VFIO device and VFIO container. These will need to be create once on the first device init. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

device passthrough: separate runtime API test

f8d1cfd

Since `device-passthrough` API is now connected, we need to not set dummy values for it before starting the VM. Otherwise it the VM startup will fail. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

vfio: seccomp: allow syscalls for VFIO device runtime

1d16b47

VFIO devices will use pread64/pwrite64 syscalls (from vfio-ioctls) to interact with BARs during runtime. Add them to the VPU thread syscall lists. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

device passthrough: doc: add documentation

efc9a37

Add docs/device_passthrough.md covering how device passthrough works in Firecracker. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

changelog: add a note about device passthrough

7fcda5c

Add a changelog entry for the new device passthrough feature. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

ShadowCurse force-pushed the vfio_part_2 branch from a17369b to 7fcda5c Compare September 15, 2026 15:13

Conversation

ShadowCurse commented Jul 23, 2026

Changes

Reason

License Acceptance

PR Checklist

Uh oh!

codecov Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilstam Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilstam Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Jul 23, 2026 •

edited

Loading

ilstam Jul 29, 2026 •

edited

Loading

ilstam Aug 3, 2026 •

edited

Loading