From 64054a0129ec5c88925706aa08ba600ffa608944 Mon Sep 17 00:00:00 2001 From: xwings Date: Tue, 1 Sep 2026 12:15:35 +0800 Subject: [PATCH] update new ARCHITECTURE docs --- ARCHITECTURE.md | 317 +++++++++++++++++++++++++++-------- ARCHITECTURE/arch.md | 4 +- ARCHITECTURE/debugger.md | 8 +- ARCHITECTURE/hw.md | 8 +- ARCHITECTURE/loader.md | 2 +- ARCHITECTURE/os-baremetal.md | 21 ++- ARCHITECTURE/os-base.md | 10 +- ARCHITECTURE/os-windows.md | 4 +- 8 files changed, 284 insertions(+), 90 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 56843aeda..11f4437b3 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -31,13 +31,13 @@ with reverse debugging), and fuzzing integration (AFL++/unicornafl). - **Shape**: pure-Python library (`from qiling import Qiling`), plus the `qltool` CLI and `qltui.py` TUI. Also shipped as a Docker image (`Dockerfile`) and PyPI package. -- **Python**: 3.8+ (`pyproject.toml:30`). Packaging via Poetry; version 1.4.8 - (`pyproject.toml:4`), status Beta, license GPLv2. -- **Core dependencies**: `unicorn ==2.1.3` (hard-pinned CPU emulator), - `capstone` (disassembly), `keystone-engine` (assembly), `pefile`, - `pyelftools`, `python-registry`, `gevent` (multithread emulation), - `pyyaml` (MCU profiles). Extras: `fuzz` → unicornafl/fuzzercorn, - `RE` → r2libr. +- **Python**: 3.10+ (`pyproject.toml:27`). Packaging via Poetry; version + 1.4.11-dev (`pyproject.toml:4`), status Beta, license GPLv2. +- **Core dependencies**: `unicorn ==2.1.3` (hard-pinned CPU emulator, + `pyproject.toml:29`), `capstone` (disassembly), `keystone-engine` + (assembly), `pefile`, `pyelftools`, `python-registry`, `gevent` + (multithread emulation), `pyyaml` (MCU profiles). Extras: `fuzz` → + unicornafl/fuzzercorn, `RE` → r2libr. - **Hosts**: Linux, Windows, macOS. Some test suites are host-gated (Windows PE tests need real system DLLs collected on Windows; see [os-windows.md](ARCHITECTURE/os-windows.md)). @@ -59,7 +59,7 @@ with reverse debugging), and fuzzing integration (AFL++/unicornafl). | `qltui.py` | Terminal UI invoked via `qltool qltui` | | `Dockerfile` | Multi-stage Poetry wheel build on `python:3-slim` | | `pyproject.toml` / `poetry.lock` | Packaging and dependency pins | -| `.github/workflows/` | CI (`build-ci.yml`: Windows + Ubuntu × Python 3.9/3.11), Docker publish, PyPI publish, Gitee mirror sync | +| `.github/workflows/` | CI (`build-ci.yml`: Windows + Ubuntu × Python 3.11/3.13, plus a Docker container job), Docker publish, PyPI publish, Gitee mirror sync | ## Boot / Entry Flow @@ -85,88 +85,269 @@ From CLI to emulated instructions (details live in the module docs): ## Roadmap -Maturity-based — Qiling is a released project in maintenance/beta (v1.4.8). -There are no in-repo milestones; module Status is `done` when its test suite -proves it. Feature requests and the forward-looking wishlist are tracked in +Maturity-based — Qiling is a released project in maintenance/beta +(v1.4.11-dev). There are no in-repo milestones; module Status is `done` when +its test suite proves it. Feature requests and the forward-looking wishlist are tracked in GitHub issue [#333](https://github.com/qilingframework/qiling/issues/333) (the `TODO` file is a pointer to it). Known cross-cutting gaps: -- `ChangeLog` stops at 1.4.6 while `pyproject.toml` says 1.4.8. -- macOS CI job is commented out in `.github/workflows/build-ci.yml`. +- `ChangeLog` stops at 1.4.6 (`ChangeLog:4`) while `pyproject.toml:4` says + 1.4.11-dev. +- macOS is dropped from the CI matrix and the macOS/kext job is commented out + (`.github/workflows/build-ci.yml:12`, `:84`). - `jexamples/` is legacy and unexercised by CI. -## Coding Discipline - -Behavioral guidelines to reduce common LLM coding mistakes. Merge with -project-specific instructions as needed. +## Development Loop -**Tradeoff:** These guidelines bias toward caution over speed. For -trivial tasks, use judgment. +Coding Discipline governs writing; Review Checks govern review. This +loop connects them and defines when work is ready to release. -### 1. Think Before Coding +```text +Frame → Write → Prove → Review → Gate + ▲ findings │ + └────────────────────────┘ +``` -**Don't assume. Don't hide confusion. Surface tradeoffs.** +### The loop + +**1. Frame.** Convert the request into a goal with an observable check. +Inspect the request, code, docs, and repository conventions; record the +narrowest supported assumptions. Ask one focused question only when a +required decision cannot be discovered or safely inferred and guessing +would materially change the result. Once framed, continue without an +approval pause. + +**2. Write.** Make the smallest change that reaches the goal. Add no +unrequested features or abstractions, match local style, touch only +in-scope code, and remove only orphans created by the change. + +**3. Prove.** Run relevant tests and retain observable evidence. + +*Survey the suite before touching it.* Before adding, changing, merging, +or deleting any test, inventory the whole suite: enumerate every test +file and case name, then read in full each test whose subject, fixtures, +or assertions touch this change. Use a subagent for broad inventory when +supported. From that inventory decide the complete set of test edits at +once — what to change, what to add, what to merge, what to remove — each +backed by `file:line`, then execute only that plan. Never write a test +before the survey, and never discover existing coverage afterward. + +The plan obeys four rules: + +- **Reuse or extend first.** Add a case to the test that already owns + the behavior or shares its setup, fixtures, and subject. A new test + function or file is justified only when the survey found no existing + test owning the behavior, or when merging would hide which case + failed. +- **Add only what the goal needs.** A bug fix needs a reproducing + regression test; a new capability needs a test of its claimed + behavior. Nothing further. +- **Retire what this change made obsolete.** Delete tests whose behavior + no longer exists, and merge tests this change turned into duplicates, + citing the surviving test. Leave unrelated pre-existing tests alone; + record suspected redundancy under **Open Gaps / Roadmap**. +- **Never delete to reach green.** A failing test is a finding for + Write. Removal requires evidence that its behavior is gone or is still + covered elsewhere, cited by `file:line`. + +Coverage of claimed behavior must not decrease. A failure returns +directly to Write, never forward to Review. + +**4. Review.** Walk all seven Review Checks as separate passes. Read +whole affected files, not only the diff. Every finding needs `file:line` +evidence. Use an independent agent or isolated pass for Fit, +Dependencies, and Security when available. + +**5. Gate.** Apply the Definition of Done. Any unticked criterion, +`blocker`, or unresolved `major` returns its evidence to Write. All +criteria passing means the change is ready for public or production +release. There is no separate approval or reporting phase. + +### Definition of Done + +**Correctness** + +- The framed goal and its named check pass. +- Tests cover claimed behavior and pass; a bug fix has a regression test. +- The suite was surveyed before any test was written, changed, or + deleted; no added test duplicates coverage another test owns, and no + removal left claimed behavior uncovered. +- The owning module's **How to Test** command passes with evidence. +- The project builds and tests from a fresh clone without local-only + dependencies. + +**Review** + +- All seven Review Checks ran; none was skipped or assumed. +- No `blocker` or unresolved `major` remains. +- Nits were applied or consciously declined. + +**Legibility and contract** + +- A new maintainer can build, test, run, and understand public behavior + from the docs. +- Every changed line serves the goal; no drive-by formatting, debugging + remnants, commented-out code, secrets, tokens, or local paths remain. +- Public names, signatures, errors, and recovery are intelligible. +- Architecture docs and `file:line` references are current. +- Breaking changes, deprecations, dependencies, licenses, and attribution + are handled; commit or PR text explains why. + +### Iterating without thrashing + +- Every pass closes a named finding and touches only what it names. +- Nits alone do not trigger another pass. +- Re-run Prove after every fix. +- Two no-change passes force Gate re-evaluation: release if Done passes; + otherwise return the surviving evidence to Frame. +- Three passes on one finding return automatically to Frame for a new + approach. +- Never widen scope to satisfy a finding. Record out-of-scope work under + **Open Gaps / Roadmap**. -Before implementing: -- State your assumptions explicitly. If uncertain, ask. -- If multiple interpretations exist, present them - don't pick silently. -- If a simpler approach exists, say so. Push back when warranted. -- If something is unclear, stop. Name what's confusing. Ask. +## Coding Discipline -### 2. Simplicity First +### 1. Think Before Coding -**Minimum code that solves the problem. Nothing speculative.** +- Understand the request, code, goal, and repository conventions first. +- Record assumptions and choose the narrowest evidence-backed reading. +- Prefer the simpler approach when it reaches the same verified goal. +- Ask only during planning and only for a required answer that cannot be + discovered or safely inferred. -- No features beyond what was asked. -- No abstractions for single-use code. -- No "flexibility" or "configurability" that wasn't requested. -- No error handling for impossible scenarios. -- If you write 200 lines and it could be 50, rewrite it. +### 2. Simplicity First -Ask yourself: "Would a senior engineer say this is overcomplicated?" If -yes, simplify. +- Implement only what was requested. +- Do not add single-use abstractions, speculative flexibility, or checks + for impossible conditions. +- If the implementation is materially larger than the problem, simplify + it. ### 3. Surgical Changes -**Touch only what you must. Clean up only your own mess.** - -When editing existing code: -- Don't "improve" adjacent code, comments, or formatting. -- Don't refactor things that aren't broken. -- Match existing style, even if you'd do it differently. -- If you notice unrelated dead code, mention it - don't delete it. - -When your changes create orphans: -- Remove imports/variables/functions that YOUR changes made unused. -- Don't remove pre-existing dead code unless asked. - -The test: Every changed line should trace directly to the user's request. +- Do not refactor, reformat, or clean up unrelated code. +- Match the surrounding style. +- Remove imports, variables, and functions made unused by this change; + leave pre-existing dead code alone unless requested. +- Every changed line must trace to the stated goal. ### 4. Goal-Driven Execution -**Define success criteria. Loop until verified.** - -Transform tasks into verifiable goals: -- "Add validation" → "Write tests for invalid inputs, then make them pass" -- "Fix the bug" → "Write a test that reproduces it, then make it pass" -- "Refactor X" → "Ensure tests pass before and after" - -For multi-step tasks, state a brief plan: - -``` -1. [Step] → verify: [check] -2. [Step] → verify: [check] -3. [Step] → verify: [check] -``` +Turn work into verifiable outcomes, then loop until they pass: -Strong success criteria let you loop independently. Weak criteria ("make -it work") require constant clarification. +- Add validation → invalid inputs are rejected by a named passing test. +- Fix a bug → a regression test fails before the fix and passes after. +- Refactor → behavior tests pass before and after. ---- +Give every plan step its own check. Strengthen vague criteria from +repository evidence before implementation. -**These guidelines are working if:** fewer unnecessary changes in diffs, -fewer rewrites due to overcomplication, and clarifying questions come -before implementation rather than after mistakes. +### Project-Specific Deviations + +- Emulation fidelity beats abstraction: syscall, API, and peripheral + implementations mirror the real platform's observable behavior even + when that means repetitive per-OS or per-chip code. Cross-OS + "unification" is a scope increase, not a simplification. +- Coverage is demand-driven by design (see the OS and HW module docs). + Adding an unrequested syscall, Win32 API, or peripheral register is + out of scope; record it under the owning module's **Open Gaps / + Roadmap**. +- `unicorn` is hard-pinned (`pyproject.toml:29`). Changing it, or any + behavior that depends on its version, is a project-wide event and + never an incidental part of another change. + +## Review Checks + +Run every check against every change before merge. Keep checks separate. + +Four rules bind all checks: + +- **Evidence or no finding.** Every finding cites `file:line`. +- **The repository is authoritative.** Demand only conventions visible + in the tree. +- **Read files, not only hunks.** Context can invalidate a finding or + reveal unreachable code, unused parameters, and hidden duplication. +- **Review the change, never the author.** Describe code and impact, not + how or by whom it was produced. + +### 1. Style + +Check indentation and local file conventions. Mixed indentation is +`major`; a consistent new file using the wrong local indent is `nit`. +Leave machine-checkable formatting to existing formatters and linters; +never demand unrelated reformatting. + +### 2. Naming + +Compare new names with nearby precedents before filing a finding. If the +repository is inconsistent, demand nothing. A local mismatch is `nit`; +an inconsistent public name is `major`. + +### 3. Duplication + +Search distinctive constants, errors, fields, and call sequences—not +only symbol names—for code performing the same job. Cite both sites and +the remedy. Cross-layer duplication is `major`; small local repetition +is `nit`. Similar code with meaningfully different branches is not +duplication. + +### 4. Quality + +Require followable control flow, errors handled where they occur, and +abstractions proportional to the problem. Swallowed errors, +inappropriate prints, unexplained magic values, and dead branches are +`major`. Remove unrequested configurability, one-caller wrappers, filler +comments, debugging remnants, and unrelated formatting. Missing tests +belong to Prove, not this check. + +### 5. Fit + +Read `ARCHITECTURE.md` and the owning module doc before the diff. Check +scope, layering, ownership, public-API growth, and performance claims. A +layering violation or unjustified public API is `major`. Architectural or +public-behavior changes must update the relevant docs in the same change. + +### 6. Dependencies + +Check manifests and imports, maintenance, supply-chain risk, advisories, +install-time behavior, license, transitive cost, and whether the standard +library is sufficient. An unjustified top-level dependency is `major`; +a live advisory or abandoned upstream is `blocker`. Incomplete evidence +does not pass. + +### 7. Security + +Check both defects and widened exposure: unsafe memory access, unchecked +sizes or offsets, integer overflow, path traversal, unsafe +deserialization, command construction, committed secrets, and unbounded +untrusted input. Trace input to impact; without a reachable path there is +no finding. A real defect is `major`; a trust-boundary break is `blocker`. +Describe the fix without publishing exploit steps. + +### Project-Specific Deviations + +- **Security, scope.** Qiling *emulates* untrusted binaries; guest code + doing something hostile inside the sandbox is the product working, not + a finding. Findings target the host boundary: rootfs escape via path + handling (`qiling/os/path.py`), unchecked guest-controlled sizes or + offsets reaching host allocations or `struct` unpacking, and parser + input in `qiling/loader/` reachable from an untrusted image. +- **Dependencies.** `pyproject.toml` is the only manifest. A new + top-level runtime dependency is `blocker` absent an explicit request; + optional integrations belong in an extra (`fuzz`, `RE`). + +### Severity and the merge threshold + +| Severity | Effect | +| -------- | ------ | +| `blocker` | Must not merge. | +| `major` | Must be resolved before merge. | +| `nit` | Apply or consciously decline. | +| `info` | Context or a question; no action implied. | + +Merge only with no `blocker` and no unresolved `major`. A check that did +not run does not pass. Findings feed Write and Gate directly; they do not +create a reporting phase. ## Index diff --git a/ARCHITECTURE/arch.md b/ARCHITECTURE/arch.md index 1bf638cd0..75e9d3b08 100644 --- a/ARCHITECTURE/arch.md +++ b/ARCHITECTURE/arch.md @@ -26,7 +26,7 @@ covered by `tests/test_cpu_models.py`. | `qiling/arch/models.py` | CPU model enums (`X86_CPU_MODEL` … `RISCV64_CPU_MODEL`) | | `qiling/arch/msr.py`, `cpr.py`, `cpr64.py` | x86 MSRs, ARM/ARM64 coprocessor registers | | `qiling/arch/utils.py` | `QlArchUtils`: disassembly output for verbose/trace modes | -| `qiling/cc/__init__.py` + `intel.py`, `arm.py`, `mips.py`, `ppc.py`, `riscv.py` | Calling conventions (arg/retval marshalling) consumed by `os/fcall.py` | +| `qiling/cc/__init__.py` + `intel.py`, `arm.py`, `mips.py`, `ppc.py`, `riscv.py` | Calling conventions (arg/retval marshalling) consumed by `qiling/os/fcall.py` | ## Key Types and Entry Points @@ -35,7 +35,7 @@ covered by `tests/test_cpu_models.py`. - `qiling/arch/x86.py:22,53,79,111` - `QlArchIntel` / `QlArchA8086` / `QlArchX86` / `QlArchX8664`. - `qiling/arch/cortex_m.py:67` - `QlArchCORTEX_M(QlArchARM)` - plus `QlInterruptContext` (`:25`) for exception entry/exit in MCU mode. - `qiling/arch/models.py` - CPU model enums selected via the `cputype` kwarg (resolved in `select_arch`, `qiling/utils.py:376`). -- `qiling/cc/__init__.py:9` - `QlCC` - abstract calling convention; `QlCommonBaseCC` (`:104`); e.g. `cc/intel.py` defines `cdecl`/`stdcall`/`ms64`/`macosx64`. +- `qiling/cc/__init__.py:9` - `QlCC` - abstract calling convention; `QlCommonBaseCC` (`:110`); e.g. `qiling/cc/intel.py` defines `cdecl`/`stdcall`/`ms64`/`macosx64`. ## Interactions diff --git a/ARCHITECTURE/debugger.md b/ARCHITECTURE/debugger.md index 88289e3fb..52156ec9c 100644 --- a/ARCHITECTURE/debugger.md +++ b/ARCHITECTURE/debugger.md @@ -17,7 +17,8 @@ reverse debugging. Mature released infrastructure; maturity-based status. | File | Role | | ---- | ---- | | `qiling/debugger/debugger.py` | Base `QlDebugger` | -| `qiling/debugger/gdb/gdb.py` | `QlGdb`: GDB remote-serial-protocol server | +| `qiling/debugger/gdb/gdb.py` | `QlGdb`: GDB remote-serial-protocol server, plus the `GdbSerialConn` transport | +| `qiling/debugger/gdb/utils.py` | `QlGdbUtils`: breakpoint table and the per-instruction `dbg_hook` that services breakpoints, stepping, and async interrupts | | `qiling/debugger/gdb/xmlregs.py`, `gdb/xml/` | Target-description XML per arch for modern GDB clients | | `qiling/debugger/qdb/qdb.py` | `QlQdb`: interactive Cmd-based debugger | | `qiling/debugger/qdb/arch/` | Per-arch Qdb support (arm, intel, mips) | @@ -27,7 +28,10 @@ reverse debugging. Mature released infrastructure; maturity-based status. ## Key Types and Entry Points - `qiling/debugger/debugger.py:13` - `QlDebugger` - base; `run()` starts the session. -- `qiling/debugger/gdb/gdb.py:68` - `QlGdb(QlDebugger)` - listens on ip:port, translates RSP packets to Qiling hook/mem/reg operations. +- `qiling/debugger/gdb/gdb.py:84` - `QlGdb(QlDebugger)` - listens on ip:port, translates RSP packets to Qiling hook/mem/reg operations; `run()` (`:139`) serves the session. +- `qiling/debugger/gdb/gdb.py:817` - `GdbSerialConn` - the socket transport; `poll_interrupt()` (`:856`) is a non-blocking check for a client `\x03`, wired into `QlGdbUtils.check_interrupt` (`qiling/debugger/gdb/gdb.py:145`) so a running guest can be broken into asynchronously. +- `qiling/debugger/gdb/utils.py:16` - `QlGdbUtils` - `dbg_hook` (`:48`) runs per instruction to service breakpoints, single-step, and interrupts; `bp_insert`/`bp_remove` (`:81`/`:94`); `resume_emu` (`:107`). +- Stop replies report `SIGTRAP` (`qiling/debugger/gdb/gdb.py:49`) for both single-step (`:243`) and async-interrupt stops (`:262`). - `qiling/debugger/qdb/qdb.py:59` - `QlQdb(Cmd, QlDebugger)` - CLI loop; `rr` mode enables record/replay reverse debugging. - Activation: set `ql.debugger = True | "gdb" | "gdb:0.0.0.0:9999" | "qdb" | "qdb:rr"` (`qiling/core.py:437`); instantiated lazily in `Qiling.run` via `select_debugger` (`qiling/utils.py:332`). - `qltool` flags: `--gdb` and `--qdb` (see [cli.md](cli.md)). diff --git a/ARCHITECTURE/hw.md b/ARCHITECTURE/hw.md index 346f6c3b3..c1466aaaf 100644 --- a/ARCHITECTURE/hw.md +++ b/ARCHITECTURE/hw.md @@ -26,11 +26,11 @@ firmware images (UART echo, freertos, blink, crc, dma_clock, i2c/spi/lcd). ## Key Types and Entry Points -- `qiling/hw/hw.py:14` - `QlHwManager` - available as `ql.hw`; created by core only when `ql.baremetal` (`qiling/core.py:191`). -- `qiling/hw/hw.py:23` - `QlHwManager.create(label, struct, base)` - instantiates a peripheral from the profile entry and maps its MMIO region. -- `qiling/hw/hw.py:82` - `QlHwManager.step()` - advances every peripheral one tick; called from the MCU run loop. +- `qiling/hw/hw.py:60` - `QlHwManager` - available as `ql.hw`; created by core only when `ql.baremetal` (`qiling/core.py:191`). +- `qiling/hw/hw.py:67` - `QlHwManager.create(label, struct, base)` - instantiates a peripheral from the profile entry and maps its MMIO region. +- `qiling/hw/hw.py:135` - `QlHwManager.step()` - advances every peripheral one tick; called from the MCU run loop. - `qiling/hw/peripheral.py:132` - `QlPeripheral(QlPeripheralUtils)` - base class: a ctypes register struct + read/write handlers on the MMIO region. -- `qiling/extensions/mcu/stm32f4xx/stm32f407.py` (and siblings) - chip `env` dicts consumed at `Qiling(..., env=...)` construction. +- `qiling/extensions/mcu/stm32f4/stm32f407.py` (and siblings) - chip `env` dicts consumed at `Qiling(..., env=...)` construction. ## Interactions diff --git a/ARCHITECTURE/loader.md b/ARCHITECTURE/loader.md index 5cbb3d9f1..da4b71966 100644 --- a/ARCHITECTURE/loader.md +++ b/ARCHITECTURE/loader.md @@ -30,7 +30,7 @@ Test command proves the ELF path end-to-end. - `qiling/loader/loader.py:21` - `QlLoader` - base; `Image` NamedTuple (`:15`), abstract `run()` (`:62`), `skip_exit_check` (`:27`). - `qiling/loader/elf.py:68` - `QlLoaderELF` - loads binary + interpreter, builds stack/auxv (`AUXV` enum `:32`), sets `entry_point`/`elf_entry`. -- `qiling/loader/pe.py:666` - `QlLoaderPE(QlLoader, Process)` - `Process` (`:69`) builds PEB/TEB/LDR; `QlPeCache` (`:41`) caches parsed DLLs behind the `libcache` kwarg (wired in `select_loader`, `qiling/utils.py:300`). +- `qiling/loader/pe.py:817` - `QlLoaderPE(QlLoader, Process)` - `Process` (`:77`) builds PEB/TEB/LDR; `QlPeCache` (`:49`) caches parsed DLLs behind the `libcache` kwarg (wired in `select_loader`, `qiling/utils.py:300`). - `qiling/loader/pe_uefi.py:26` - `QlLoaderPE_UEFI` - loads DXE/SMM modules and installs protocols into the UEFI context. - `qiling/loader/macho.py:70` - `QlLoaderMACHO`. - `qiling/loader/mcu.py:57` - `QlLoaderMCU` - with `IhexParser` (`:15`). diff --git a/ARCHITECTURE/os-baremetal.md b/ARCHITECTURE/os-baremetal.md index 2823b0954..0b7a68a7e 100644 --- a/ARCHITECTURE/os-baremetal.md +++ b/ARCHITECTURE/os-baremetal.md @@ -11,7 +11,8 @@ released infrastructure; maturity-based status. ## Status `done` — MCU covered by `tests/test_mcu.py` (STM32F1/F4, GD32VF1 firmware), -BLOB by `tests/test_blob.py` (u-boot) and `tests/test_edl.py`. +BLOB by `test_blob.BlobTest.test_uboot_arm` and `tests/test_edl.py`. The other +case in `tests/test_blob.py` is blocked on a missing fixture (see Open Gaps). ## Code Structure @@ -23,10 +24,10 @@ BLOB by `tests/test_blob.py` (u-boot) and `tests/test_edl.py`. ## Key Types and Entry Points -- `qiling/os/mcu/mcu.py:37` - `QlOsMcu(QlOs)` - run loop; steps hardware between execution chunks and delivers interrupts. +- `qiling/os/mcu/mcu.py:41` - `QlOsMcu(QlOs)` - run loop; steps hardware between execution chunks and delivers interrupts. - `qiling/os/mcu/mcu.py:17` - `MCUTask(UnicornTask)` - the firmware execution task. - `qiling/extensions/multitask.py:26` / `:152` - `UnicornTask` / `MultiTaskUnicorn(Uc)` - task-switching Unicorn subclass MCU mode runs on. -- `qiling/os/blob/blob.py:12` - `QlOsBlob(QlOs)` - runs `entry_point` → `exit_point` with no OS services. +- `qiling/os/blob/blob.py:14` - `QlOsBlob(QlOs)` - runs `entry_point` → `exit_point` with no OS services. - MCU selection: `QL_OS.MCU` is the only member of `QL_OS_BAREMETAL` (`qiling/const.py:74`); `ql.baremetal` (`qiling/core.py:357`) gates hardware-manager creation. ## Interactions @@ -40,13 +41,21 @@ BLOB by `tests/test_blob.py` (u-boot) and `tests/test_edl.py`. ## How to Test ```sh -cd tests && python3 test_blob.py # pass = unittest "OK", exit 0 +cd tests && python3 -m unittest test_blob.BlobTest.test_uboot_arm # pass = "OK", exit 0 ``` -- MCU (also proves [hw.md](hw.md)): `cd tests && python3 test_mcu.py`. -- Qualcomm EDL loader: `cd tests && python3 test_edl.py`. +- MCU (also proves [hw.md](hw.md)): `cd tests && python3 test_mcu.py` — pass = `Ran 18 tests … OK`. +- Qualcomm EDL loader: `cd tests && python3 test_edl.py` — pass = `Ran 1 test … OK`. +- The whole `test_blob.py` file does **not** pass from a clean checkout; see + Open Gaps. ## Open Gaps / Roadmap - Supported chip families are those with board definitions in `qiling/extensions/mcu/` (STM32F1/F4, GD32VF1, NXP, Atmel, BES); new chips need new peripheral maps. - BLOB mode provides no services by design — targets needing hardware must use MCU mode instead. +- `test_blob.BlobTest.test_blob_raw` (`tests/test_blob.py:85`) errors with + `FileNotFoundError` on a clean checkout: it reads + `examples/rootfs/blob/example_raw.bin` (`tests/test_blob.py:96`), but the + pinned `examples/rootfs` submodule ships only `u-boot.bin.img`. Either the + fixture must be added upstream and the submodule bumped, or the test skipped + when the fixture is absent. diff --git a/ARCHITECTURE/os-base.md b/ARCHITECTURE/os-base.md index 7d71305de..756a0e044 100644 --- a/ARCHITECTURE/os-base.md +++ b/ARCHITECTURE/os-base.md @@ -29,13 +29,13 @@ everything else is exercised by every OS suite. ## Key Types and Entry Points -- `qiling/os/os.py:24` - `QlOs` - composes utils/stats/path/fs-mapper; `user_defined_api` keyed by `QL_INTERCEPT`; abstract `run()` (`:239`). -- `qiling/os/os.py:224` - `QlOs.set_api(target, handler, intercept)` - user override of an emulated API (address- or name-based). -- `qiling/os/memory.py:23` - `QlMemoryManager` - `ql.mem`; instantiated by core *before* the OS (`qiling/core.py:188`). -- `qiling/os/memory.py:658` - `QlMemoryHeap` - heap used by Windows/UEFI APIs and the sanitizers extension. +- `qiling/os/os.py:24` - `QlOs` - composes utils/stats/path/fs-mapper; `user_defined_api` keyed by `QL_INTERCEPT`; abstract `run()` (`:240`). +- `qiling/os/os.py:225` - `QlOs.set_api(target, handler, intercept)` - user override of an emulated API (address- or name-based). +- `qiling/os/memory.py:40` - `QlMemoryManager` - `ql.mem`; instantiated by core *before* the OS (`qiling/core.py:188`). +- `qiling/os/memory.py:678` - `QlMemoryHeap` - heap used by Windows/UEFI APIs and the sanitizers extension. - `qiling/os/fcall.py:21` - `QlFunctionCall` - argument/return marshalling on top of `qiling/cc/`. - `qiling/os/mapper.py:64` - `QlFsMapper` - behind `ql.add_fs_mapper` (`qiling/core.py:701`). -- `qiling/os/path.py:14` - `QlOsPath` - rootfs-confined path resolution. +- `qiling/os/path.py:16` - `QlOsPath` - rootfs-confined path resolution. - `qiling/os/thread.py:11` - `QlThread(Greenlet)` - gevent-based thread base. ## Interactions diff --git a/ARCHITECTURE/os-windows.md b/ARCHITECTURE/os-windows.md index 88a642765..cf4187796 100644 --- a/ARCHITECTURE/os-windows.md +++ b/ARCHITECTURE/os-windows.md @@ -31,14 +31,14 @@ run on Linux. Windows tests are platform-gated locally (need collected DLLs). - `qiling/os/windows/windows.py:33` - `QlOsWindows(QlOs)` - `__setup_components` (`:156`) builds handle manager/registry/clipboard/fiber; `run()` (`:201`). - `qiling/os/windows/fncc.py` - `@winsdkapi` decorator - declares an API's calling convention and typed params; implementations live in `dlls/`. -- API dispatch is address-based: IAT addresses recorded by the PE loader are hooked and marshalled through `QlFunctionCall` — user overrides via `QlOs.set_api` (`qiling/os/os.py:224`). +- API dispatch is address-based: IAT addresses recorded by the PE loader are hooked and marshalled through `QlFunctionCall` — user overrides via `QlOs.set_api` (`qiling/os/os.py:225`). - `qiling/os/uefi/uefi.py:22` - `QlOsUefi(QlOs)` - executes DXE/SMM modules; services in `bs.py`/`rt.py`/`smm.py` are installed as callable tables. - `qiling/os/dos/dos.py:33` - `QlOsDos(QlOs)` - dispatches BIOS/DOS interrupts from `interrupts/`. ## Interactions - All three subclass [os-base.md](os-base.md) `QlOs` and allocate from `QlMemoryHeap`. -- [loader.md](loader.md): `QlLoaderPE` builds PEB/TEB and records IAT hook addresses; `QlLoaderPE_UEFI` installs protocols into `os/uefi/context.py`; `QlLoaderDOS` sets real-mode state. +- [loader.md](loader.md): `QlLoaderPE` builds PEB/TEB and records IAT hook addresses; `QlLoaderPE_UEFI` installs protocols into `qiling/os/uefi/context.py`; `QlLoaderDOS` sets real-mode state. - Argument marshalling uses `QlFunctionCall` + `qiling/cc/intel.py` conventions ([arch.md](arch.md)). - The registry emulation reads hive files from the rootfs via [os-base.md](os-base.md) path services. - `examples/uefi_sanitized_heap.py` pairs UEFI with the heap sanitizer from [extensions.md](extensions.md).