feat: Regression-test overhaul, and the parser/renderer fixes it uncovered

[PeterStaar-IBM]

Reworks how the regression suite selects and stores its data, then fixes the
defects that the sharper suite and the quality benchmark surfaced: pattern
clipping, CMap decoding, word spacing, glyph identity, inline colour spaces,
annotation appearances and image minification.

Across the 444-document regression corpus (page 1 of each, mean absolute error
against pypdfium2 at scale 1):

	mean delta
main	0.01774
this branch	0.01593

182 documents improved, 28 regressed, 234 unchanged, no render failures.

---

Test infrastructure

Page selection is now generated, not hand-maintained. PARSER_PAGE_RESTRICTIONS
was a hand-written map that had to be edited whenever the corpus grew.
tests/tools/select_regression_pages.py now generates
tests/regression_page_selection.py from stated rules: pinned documents win,
pages that once failed are always kept, small documents keep every page, and the
rest are topped up from a generator seeded by the document name — so a sample is
reproducible and adding one document never reshuffles another.

Orphaned groundtruth has somewhere to go. tests/tools/prune_groundtruth.py
moves groundtruth for documents that have left the corpus into
tests/data/groundtruth-legacy, mirroring the layout, so nothing is lost and no
test globs it.

Bitmap groundtruth is one metadata file per page instead of one file per
artifact. The corpus paints tens of thousands of bitmaps across a handful of
distinct decode paths; the per-artifact layout exceeded what Git LFS handles in
one directory. Every artifact is still compared exactly — the metadata carries
raw_sha256 and encoded_sha256 — while only the first artifact of each decode
signature per page keeps its bytes on disk.

scripts/check_rendering_regression.py is now a triage tool, not a second
regression check. It reads artifacts that already exist rather than rendering
anything, and reports the per-page table, the error distribution, and the
observed_limits you would use to set ImageTolerance.

Tests renamed by what they need

Every test file now declares its dependency in its name:

• test_unit_* (24 files, 140 tests) — self-contained. The document under
  test is built in the file via pdf_builder.py or in memory. Runs on a bare
  checkout in 5.5 s with no corpus.
• test_regression_* (6 files, 83 tests) — driven by the corpus under
  tests/data/regression, mostly compared against stored groundtruth.

pytest_sessionstart now skips the Hugging Face download when every selected
path is a test_unit_* file. The predicate is deliberately conservative: a bare
pytest, a directory, or a -k expression all still download, because only an
all-unit selection can prove it does not need the corpus.

Two files sit on the line and are worth a reviewer's opinion:
test_regression_embedded_fonts.py and test_regression_locale_safety.py read
corpus documents but assert inline rather than against groundtruth. They are
grouped as regression because they cannot run without the dataset.

---

Parser and renderer fixes

Each was found from a specific page, and each is grounded in ISO 32000-1 rather
than tuned against an output.

Tiling pattern floods the page (shape.h)

normal_4pages.pdf painted its whole page teal. An Illustrator logo strokes its
gradient mesh through sh inside tiny clip paths; keep_shape() measured
degeneracy as the longest edge between consecutive points, and since
interpolate() flattens every curve into N straight edges, a real 0.09-unit
curve became edges of 0.011 and was discarded. sh paints its entire clip
region (8.7.4.3), so an sh that loses its only clip path floods the canvas.

Degeneracy is now the bounding-box diagonal of the subpath, and
capture_pending_clip() keeps every subpath that has a segment: dropping a clip
path widens it, which is the unsafe direction, and a zero-area clip path
legitimately clips everything away.

0.3432 → 0.0169 on that page.

Predefined CMaps barely parsed (font_cid.h, text.h)

Japanese pages came out as Latin noise. UniJIS-UCS2-HW-H is a 25-line file that
says /UniJIS-UCS2-H usecmap and overrides four ASCII ranges; ignoring usecmap
left the font with ~95 mappings and no codespace. 77 of the 194 bundled CMaps are
overrides like this.

The scanner now follows usecmap (parent first, so the child's entries win, per
9.7.5.2) and reads begincodespacerange and begincidchar as well as
begincidrange. UniJIS-UCS2-HW-H goes from 95 to 9736 mappings.

Byte splitting was a guess — "two bytes if that pair is in the cmap, else one" —
which put the reader one byte out of phase for the rest of the string the first
time a code was missing. It now uses the codespace ranges (9.7.6.2), matched byte
by byte the way Adobe defines it, so Shift-JIS CMaps mixing one- and two-byte
codes in one string split correctly.

Across the whole corpus: CMAP_RESOURCES decode errors ~470 → 0.

Embedded CMap streams (enums.h, page_font.h, font_cid.h)

A Type0 font whose /Encoding is an indirect reference to a CMap stream fell
through to STANDARD, so every two-byte code split into a phantom 0x00 plus
the real code. The phantom took /DW 1000, pushing each glyph one em right of
where the producer put it — which is why map labels sat beside their white
halos instead of on them.

New CMAP_STREAM encoding; init_embedded_cmap() decodes the stream and keeps
its codespace. The CMap scanner became token-driven rather than line-driven,
which was necessary rather than cosmetic: Adobe's files put one entry per line,
but an embedded CMap writes 1 begincidchar <0020> 1 endcidchar on a single
line. The tokenizer skips % comments and reads (...) whole, so the page of
%%Copyright atop every Adobe CMap cannot be mistaken for operators.

Word spacing applied to the wrong thing (text.h)

Tw was charged whenever the decoded text was a space, rather than to the
single-byte character code 32 (9.3.3). A symbolic font whose codes the
decoder cannot resolve yields " " for every glyph, so a Cyrillic line was
stretched by a whole character width and ran over the absolutely-positioned text
beside it.

8 documents improved, 0 regressed.

Glyph identity ignored the declared base encoding (page_font.h, encoding.h)

get_glyph_name() returned a name only for codes listed in /Differences, so
everything else was resolved through the font program's own encoding — and
/WinAnsiEncoding code 0xE1 drew the Standard-encoding glyph at that slot: Æ
where the page says á. The effective encoding of a simple font is the declared
base encoding with /Differences on top, and it takes precedence over the
program's (9.6.6.2). Gated on has_explicit_encoding, because a font dictionary
that declares no base encoding is exactly the case where the program's own
encoding governs.

Inline images naming a colour-space resource (stream.h, bitmap.h, page_colorspace.h)

read_inline_image_header() only canonicalised the abbreviations (/G, /RGB,
/CMYK, /I), so /CS /R50 — a name in the page's /ColorSpace dictionary,
which 8.9.7 Table 93 explicitly allows — reached bitmap.h verbatim and the
image was dropped. On one figure that was 241 of 2316 inline images, and the
flat background they were drawn on showed through.

pdf_resource<PAGE_COLORSPACE> already parsed /Indexed fully, so the new
build_indexed_rgb_palette() runs every index through the existing
map_to_rgb(). A decoder handed that can treat any indexed space as a plain
/DeviceRGB palette, whatever its base actually was — ICCBased, Lab, or a
Separation with a tint transform — with no logic duplicated from the XObject path.

Annotation appearances (page.h, stream.h)

Only /Widget annotations were presented through their appearance stream. Any
annotation carrying an /AP/N now renders through it (12.5.5), honouring /F
bit 2 (Hidden) and bit 6 (NoView) and skipping /Popup.

The placement was also wrong: appearances were shifted by the /Rect origin,
which is only correct when /Matrix is the identity and /BBox starts at the
origin. appearance_matrix() now composes /Matrix with the fit of the
transformed /BBox onto /Rect, and set_base_matrix() seeds the sub-decode
with it so every instruction comes out in page space — rotation and scaling
included, rather than approximated away. This is a latent fix for form fields
too: any widget whose appearance has a non-identity /Matrix was misplaced.

Minified images were point-sampled (blend2d_renderer.h)

Blend2D samples a pattern once per destination pixel. A 16×16 one-bit hatch tile
drawn 4 px wide kept one source row in four, so a floor plan's grid came out as a
field of dots and its diagonal wall hatch vanished. box_downsample() now
area-averages the source by an integer factor before the blit whenever the drawn
extent is at least 2× smaller. The samples are premultiplied, so every channel
including alpha averages linearly.

The single largest win in the branch: 162 improved, 19 regressed, corpus mean
0.01677 → 0.01593. Every regression is ≤ +0.002.

Smaller fixes

• CJK fallback font never found (blend2d_font_resolver.h). The candidate
  list matched Linux filename stems only, so macOS reported "no CJK-capable font
  installed" while carrying twelve. Adding the macOS names was not enough —
  APFS stores ヒラギノ角ゴシック W3.ttc in NFD, so even the Japanese spelling
  does not compare equal. The scan now ranks on the family name from the font's
  name table as well as the stem, which required moving
  cjk_fallback_candidates() after build_font_index().
• Degenerate image bboxes (page_dimension.h). A zero-width or zero-height
  intersection means the image matrix maps the unit square of 8.9.5.2 onto a line
  or a point, so it covers no area; > became >=.
• Invalid Unicode scalars (page_font.h). A CID interpreted as a scalar
  value is already a last resort and is not even possible for surrogates or
  values beyond U+10FFFF.
• Glyph markers instead of leaked code bytes (text.h). Raw code bytes are
  not text and are not always valid UTF-8.
• Thread-safety of the CMap cache (font_cids.h). cids is filled lazily
  while the threaded parser decodes on several workers; unsynchronised inserts
  are a data race, and a second thread must not observe a half-built font_cid.

---

Quality benchmark: 5.6× faster

scripts/benchmarking/run_quality_benchmarking.py was a single serial loop that
wrote a three-panel PNG for every page. Profiling put the reference renderer at
6% of the runtime and the visualization writing at 31%.

• --workers/-j drives a ProcessPoolExecutor. Processes, not threads:
  pdfium is not thread-safe and aborts the interpreter when several threads
  render at once — reproduced, and the reason a reference cache was considered
  instead. Processes sidestep it, and results are bit-identical to serial.
  The visualization is written inside the worker so page images never cross a
  process boundary.
• --render-visualizations defaults to above-tolerance, not all.
• Render and text passes merged into one per-page job, so the docling-parse
  decode behind them happens once. PageParseResult hands out both the image and
  the char cells from the same decode: 162 → 135 ms/page, identical output.
• flatten_on_white short-circuits on an already-RGB image, which the render
  path was paying for twice.

On a 60-page subset, against the original script:

	wall	output
before (serial, all viz)	19.9 s	56 MB
after (defaults)	3.5 s	14 MB

render_quality.csv, text_quality.csv and all 60 visualization PNGs are
byte-identical to the original script's.

[github-actions]

✅ DCO Check Passed

Thanks @PeterStaar-IBM, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?: