FR: extraction override surface is per-field-value only — no hook for payload location or structural mapping

[dmealing]

Affected port(s): Python measured; the surface is cross-port, so this is a
metamodel/API question rather than a Python one.
Package + version: metaobjects 1.0.0 (PyPI)

The gap

ExtractOptions is a genuinely useful override surface, and it is live — on_field,
normalizers and tolerance are all read in render/extract/coerce.py, and each
records a Coercion in the report so an adopter can see what happened. That last part
is the good bit: an override is auditable rather than invisible.

But every one of them is per-field-VALUE: they take a raw string for one field
and return a coerced value. There is no hook above that level. So the moment an
adopter needs to change which text is the payload, or how the reply's structure maps
onto the declared shape, the surface runs out and the only move is to stop using the
generated extract.

That is what I actually observed. Auditing a Python codebase with six hand-rolled JSON
readers, I mapped each one's post-processing onto the declared path:

what the adopter's reader did	expressible today?
uppercase every extracted entity name	✅ normalizers — the adopter simply didn't know
fold a wrapper section ({"Classification": {…}}) into top-level fields	❌ structural key remap
merge two sibling arrays (other_relationships into relationships)	❌ structural array merge
choose which object in a dirty reply is the payload	❌ no hook (and see #363)

The first row is the encouraging one and the reason I'm filing this rather than just
the bug: it was already supported, the adopter reimplemented it in application code,
and the platform lost the audit trail it would have given for free. Discoverability is
part of the ask.

What I'd like

Three things, in the order I'd value them:

1. A document-level locate/pre-parse hook on ExtractOptions. Something like
on_locate(text, format) -> str | None — given the raw reply, return the substring
that is the payload (or None to accept the default strategy). That single seam would
have let every adopter above keep the generated extract instead of abandoning it, and
would let a consumer with an unusual reply envelope adapt without forking. It also
gives #363 an escape hatch for anyone who needs different precedence than whatever
default you settle on.

If a whole-text hook is too blunt, the narrower version is a payload_location
strategy enum (first_object / prefer_fenced / last_object) — less general, but it
covers the observed cases and keeps the behaviour declarative.

2. Structural aliasing as vocabulary. @enumAlias already establishes the idea of
"this incoming token means that declared one", but it is enum-value-only. The same idea
one level up would cover wrapper folding: a declared alias saying the incoming key
Classification (or Rewrite, or ROUTING) contributes its children to this object's
fields. Models really do emit these — they mirror the section headings from the prompt's
own documentation — and today every adopter re-discovers and re-encodes that
independently.

meta types --all finds no alias, normalize or tolerance vocabulary reachable
from template.prompt, and @enumAlias / @normalize are field.enum-scoped, so as
far as I can tell there is nothing existing to reach for here. Happy to be corrected —
I searched before proposing, per the authoring guidance.

3. Array merge is the weakest of the three and I'd honestly defer it. It showed up
once, it is arguably a modelling smell in the prompt rather than a platform gap, and I'd
rather not add vocabulary for a single sighting.

Explicitly NOT asking for

A general "repair malformed JSON" knob. 1.0.0 already recovers a trailing comma at
NORMAL tolerance, which was the only malformation I could actually evidence, and the
adopter's separate repair stage turned out to be largely redundant for a declared
prompt. Adding a repair dial on that evidence would be speculative.

Why the level matters

The framing that made this click: fence handling is not the interesting problem. Six
readers each solved fence handling and each still had a blind spot, because the real
variable is where the payload is and how its structure maps onto the declared shape
— and those are decisions the metadata is better placed to own than any call site. The
per-field surface says the design already accepts "the 80% is declared, the 20% is
overridable"; this is a request to apply the same split one level up, where the
remaining 20% actually lives.

Related

#363 — the concrete defect that exposed this: the locator takes the first object that
parses, so a fenced answer after any brace-bearing prose is silently discarded.

[dmealing]

One more limitation, found while trying to actually adopt the surface rather than
just reading it — a fourth item, and I think the sharpest one.

Neither hook has a reject channel. on_field and normalizers are both gated on
is not None:

hooked = opts.on_field(field_path, raw, spec)
if hooked is not None:
    ...
    return hooked

norm = opts.normalizers.get(field_path) or opts.normalizers.get(spec.name)
if norm is not None:
    normalized = norm(raw)
    if normalized is not None:
        ...
        return normalized

So None means "I have no opinion, fall through to default coercion" — the only
meaning available. A hook cannot say "this value is invalid."

That is fine for a pure transform (uppercase, trim). It quietly breaks the common case
where the per-field rule is a validator as much as a normalizer — a slug
normalizer, an id canonicaliser, a bounded-vocabulary mapper. Verified on 1.0.0:

opts = ExtractOptions(normalizers={"topic": lambda raw: None})   # "reject this"
extract_object(mo, '{"topic":"Not A Slug!","worth_forming":true,"reason":"r"}',
               Format.JSON, opts)
# topic == 'Not A Slug!'   has_lost_required() == False

The rejected value survives unchanged and the extraction reports success. An adopter
who reads None as "reject" — which is what it means in most validation APIs — gets
the opposite of what they asked for, silently. That is the same
plausible-value-instead-of-a-failure shape the extraction contract otherwise works hard
to avoid: @required + has_lost_required() exist precisely so a missing value cannot
masquerade as an answer, and then this path lets an invalid one do exactly that.

This is why I left a per-field slug normalizer in application code rather than moving
it onto the surface — not because the surface was the wrong place, but because moving
it would have converted a hard rejection into a silent pass.

Suggestion, cheapest first: a distinguished sentinel (extract.REJECT) that a hook can
return to mark the field lost, so it lands in lost_required() / the report like any
other missing required field. A sentinel avoids overloading None and stays
backward-compatible — every existing hook keeps its current meaning.

Ordering-wise I would now put this above the structural-aliasing item in the original
post: it is smaller, it is a correctness gap rather than an expressiveness one, and it
does not need new metadata vocabulary.