A driver declares which families it serves and what each family can be asked for —
Provider.families maps a family name to a ModelCapability. It declares nothing about what shape
its answers come back as, and the code assumes one: a text-prompted model returns boxes.
That assumption is only true of the family this distribution happens to ship. Prompt kind and
output geometry are two independent axes:
|
takes points |
takes words |
| returns boxes |
— |
grounding-dino, today |
| returns masks |
the SAM 2 family, today |
a text-promptable segmenter |
The bottom-right cell is empty in this build and is not empty in the world. A model that accepts a
phrase and answers with masks — SAM 3, or a detector and a segmenter composed — declares
text_detect under the current vocabulary, which is correct about the prompt and silent about the
polygon it is going to return.
Where the assumption is written down
The prompt is derived from box classes only. detectable_classes in
src/visionset/inference/prelabel.py keeps a class when GeometryType.BBOX in label_class.geometries. A schema whose classes are polygon-only produces an empty prompt and the
run is refused, even against a model that would have answered with polygons perfectly well.
Suppression only understands boxes, and says why. src/visionset/inference/nms.py: "Only
BboxGeometry participates. A polygon or a polyline has no cheap IoU and no evidence that it needs
one — the duplication this was measured against is a detector's, and a mask-producing model is
prompted per instance." That reasoning is sound for the two families here now and stops being sound
for the empty cell above: a segmenter prompted with words returns many masks from one call and
duplicates exactly the way a detector does. Anything that is not a box currently passes through
untouched, which is the right default for "I cannot compare these" and the wrong outcome for
"these were the same object twice".
Nothing else needs changing. PredictedRegion.geometry is already the domain's own union, so
the port can carry a polygon today; AssetPrediction needs no new shape; and the write gate
already judges a geometry against what the class declares. The gap is that nobody declares which
geometries a driver produces, so no caller can act on it.
The precedent to follow
Exporter already models this exactly. It carries supported_geometries: frozenset[GeometryType]
— "declared over GeometryType, every name the domain can address" — alongside
degraded_geometries for the shapes it accepts but flattens. A provider declaring the geometries
it produces is the same idea pointed the other way, and it belongs on Provider beside
families rather than being inferred from a capability name.
What that unlocks, and what has to be decided
Once a driver says what it returns, detectable_classes can ask for the classes the resolved
model can satisfy rather than the classes a box can, and the pre-label dialog can say which
geometry a run will produce before it starts — which is also the honest answer to a schema of
polygon classes and a detector, where the run is refused today with a sentence about boxes.
Decisions this needs:
- Whether a driver may declare more than one output geometry, and if so whether the caller
chooses or the model does. A segmenter that can emit either a mask or its bounding box is a real
case.
- Whether the geometry is declared per family or per driver. Families already carry the
capability, and a driver serving several may not answer in the same shape for each.
- What suppression does for masks. Either an IoU over polygons, or a declared "this family is
prompted per instance and does not duplicate", so the absence of suppression is a stated property
rather than an omission.
- How a class admitting both a box and a polygon is handled when the model can produce both.
That is the common schema shape, not an exotic one.
The provider contract that introduced families is #498, and batch pre-labeling, which is the
first consumer to depend on the output shape, shipped in #673.
A driver declares which families it serves and what each family can be asked for —
Provider.familiesmaps a family name to aModelCapability. It declares nothing about what shapeits answers come back as, and the code assumes one: a text-prompted model returns boxes.
That assumption is only true of the family this distribution happens to ship. Prompt kind and
output geometry are two independent axes:
The bottom-right cell is empty in this build and is not empty in the world. A model that accepts a
phrase and answers with masks — SAM 3, or a detector and a segmenter composed — declares
text_detectunder the current vocabulary, which is correct about the prompt and silent about thepolygon it is going to return.
Where the assumption is written down
The prompt is derived from box classes only.
detectable_classesinsrc/visionset/inference/prelabel.pykeeps a class whenGeometryType.BBOX in label_class.geometries. A schema whose classes are polygon-only produces an empty prompt and therun is refused, even against a model that would have answered with polygons perfectly well.
Suppression only understands boxes, and says why.
src/visionset/inference/nms.py: "OnlyBboxGeometryparticipates. A polygon or a polyline has no cheap IoU and no evidence that it needsone — the duplication this was measured against is a detector's, and a mask-producing model is
prompted per instance." That reasoning is sound for the two families here now and stops being sound
for the empty cell above: a segmenter prompted with words returns many masks from one call and
duplicates exactly the way a detector does. Anything that is not a box currently passes through
untouched, which is the right default for "I cannot compare these" and the wrong outcome for
"these were the same object twice".
Nothing else needs changing.
PredictedRegion.geometryis already the domain's own union, sothe port can carry a polygon today;
AssetPredictionneeds no new shape; and the write gatealready judges a geometry against what the class declares. The gap is that nobody declares which
geometries a driver produces, so no caller can act on it.
The precedent to follow
Exporteralready models this exactly. It carriessupported_geometries: frozenset[GeometryType]— "declared over
GeometryType, every name the domain can address" — alongsidedegraded_geometriesfor the shapes it accepts but flattens. A provider declaring the geometriesit produces is the same idea pointed the other way, and it belongs on
Providerbesidefamiliesrather than being inferred from a capability name.What that unlocks, and what has to be decided
Once a driver says what it returns,
detectable_classescan ask for the classes the resolvedmodel can satisfy rather than the classes a box can, and the pre-label dialog can say which
geometry a run will produce before it starts — which is also the honest answer to a schema of
polygon classes and a detector, where the run is refused today with a sentence about boxes.
Decisions this needs:
chooses or the model does. A segmenter that can emit either a mask or its bounding box is a real
case.
capability, and a driver serving several may not answer in the same shape for each.
prompted per instance and does not duplicate", so the absence of suppression is a stated property
rather than an omission.
That is the common schema shape, not an exotic one.
The provider contract that introduced
familiesis #498, and batch pre-labeling, which is thefirst consumer to depend on the output shape, shipped in #673.