Skip to content

Add ORDS v0.3 repair data export endpoint - #58

Open
ardelato wants to merge 8 commits into
hermesfrom
feat/ords-endpoint
Open

Add ORDS v0.3 repair data export endpoint#58
ardelato wants to merge 8 commits into
hermesfrom
feat/ords-endpoint

Conversation

@ardelato

@ardelato ardelato commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Description

Adds GET /api/public/v2/repairs, which exports repair records in Open Repair Data Standard v0.3 for the Open Repair Alliance. They asked for this directly since they currently hand-export from the database. No exporter exists in this fork or upstream, so this is the first codified one. Ships behind FEATURE__PUBLIC_REPAIRS_API, off by default.

Changes

  • Endpoint: GET /api/public/v2/repairs under a new repairs:read scope, JSON or CSV, with updated_since, event_start, event_end, powered, page and per_page filters. Per-page ceiling is 1000 rather than the events endpoint's 100 since this is a bulk export. Visibility mirrors PublicEventController: approved events on approved groups, soft-deletes excluded, allowed_network_ids honoured. The network filter is a subquery rather than a join, so a group in several permitted networks doesn't multiply its devices. Files: API/PublicRepairController.php, EnsurePublicRepairsApiEnabled.php.
  • Feature flag split: the flag moves off the shared public/v2 group onto each scope, so repairs can ship dark without taking the live events API down with it. Files: routes/api.php, bootstrap/app.php, config/restarters.php.
  • Column mapping: the fourteen standard columns in spec order. year_of_manufacture isn't stored so it's derived from the event year minus devices.age, country converts alpha-2 to alpha-3, and product_category_id is a name lookup because our idcategories don't match theirs. Files: Services/Ords/OrdsRecordMapper.php, Helpers/Iso3166.php.
  • Vocabulary: config/ords.php follows ORA's published data rather than their tableschema.json, which is stale. The standard collapses our screen-size and laptop-size splits, carries Unknown as a real repair_status, and drops the "the" from the barrier we seed.
  • Redaction: devices.problem is free text with no Purify mutator. ProblemTextScrubber strips HTML and redacts emails, phone numbers, digit runs of 8 or more, and URL query strings, logging counts by type per request. Names aren't pattern-detectable and aren't removed.
  • Config guard: ORDS_ID_PREFIX and ORDS_DATA_PROVIDER have no defaults and the endpoint 503s while either is blank. The id is a stable key ORA upserts on across releases, so a borrowed namespace would overwrite another provider's rows.
  • Docs: docs/public-repairs-api.md, alongside the existing events one.

Output was validated against ORA's published tableschema.json. Column names and order match exactly and every declared constraint passes, except their id regex, which uses a hyphen while all 305,649 of their own published rows use an underscore.

Deferred: updated_since doesn't catch approval flips, so approving an old event leaves it invisible to an incremental consumer. The categories inner join silently drops devices with an orphaned category. Pagination is offset-based so a full crawl is quadratic. Throttling runs after auth, so invalid tokens hit the database unrated.

QA Notes

We'll need to enable the flag on the test cluster and validate a full export against the spec.

Groups store country_code as alpha-2 but the Open Repair Data Standard
requires alpha-3, and Fixometer::getAllCountries only maps alpha-2 to a
localised display name. Checked in as a static map rather than pulling in
league/iso3166 so the exporter stays dependency-free for other instances.
The key set is verified to match lang/instances/base/en/countries.php.
devices.problem is unsanitised free text with no Purify mutator, so it can
carry raw HTML, email addresses, phone numbers and URLs with tracking
parameters. Redaction is a standalone pass rather than logic inside the
mapper so it can be toggled, tested against fixtures and audited on its own,
and it reports per-run counts by type so an export can be checked before
handover.

Every pass fails closed: a regex error yields an empty string rather than
leaking the unredacted original. The phone heuristic uses a nine-digit floor
to keep hyphenated substance such as rpm ranges, part numbers and firmware
versions intact.
Emits the fourteen standard columns in spec order. Most map straight across
because the standard was derived from this schema, but three need work:
year_of_manufacture is not stored so it is derived from the event year minus
the item age, country needs alpha-3, and product_category_id needs a name
lookup because our idcategories do not match the published ones.

Vocabulary handling follows the Open Repair Alliance's published releases
rather than the table schema, which is stale in two places: the standard
collapses our screen-size and laptop-size splits into single categories,
repair_status carries Unknown as a real value, and the barrier wording drops
the "the" we seed.
Holds the instance identity and the vocabulary maps in config rather than a
database table so a change shows up in a diff and is versioned alongside the
standard it tracks.

id_prefix and data_provider are deliberately undefaulted. The identifier is
a stable key the consumer upserts on across releases, so serving under an
unassigned or borrowed namespace would overwrite another provider's records;
the endpoint refuses to serve while either is blank.
Adds GET /api/public/v2/repairs behind a new repairs:read scope, serving
JSON or CSV with filters for update time, event window, powered category and
pagination. The per-page ceiling is 1000 rather than the events endpoint's
100 because this is a bulk export rather than a display API.

Visibility mirrors PublicEventController: approved events on approved
groups, soft-deleted rows excluded, and allowed_network_ids honoured. The
network restriction is a subquery rather than a join so a group belonging to
several permitted networks does not multiply its devices.

The feature flag moves from the shared public/v2 group onto each scope so
the export can ship dark without taking the live events API down with it.

CSV output escapes cells opening with a formula character, which spreadsheet
software would otherwise execute on open, and carries the pagination and
sync metadata in headers since the format has no envelope for it.

Documented in docs/public-repairs-api.md alongside the events API.
repairs:read is the first scope beyond events:read, so the option help now
lists the valid values instead of only showing the default.
Covers the auth surface including an events:read token being refused on
repairs and the reverse, the feature flag in both directions, the config
guards, all fourteen columns against a fixture, CSV column order and
formula escaping, visibility exclusions, network restriction, filters,
pagination and the redaction passes.
Ships every value blank or off. The identity settings are per deployment
rather than defaulted in the chart, and the endpoint refuses to serve until
an instance sets them.
@ardelato
ardelato force-pushed the feat/ords-endpoint branch from 8179e34 to 3618e63 Compare August 4, 2026 15:52
@ardelato

ardelato commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator Author

QA 👍

  1. I seeded the test cluster instance with groups, events, and devices.
  2. Created the API token
  3. Used curl to query the endpoint and confirmed the data retrieved was in the correct format, was the correct schema, stripped PII, and proper escaping.

@probablyian

probablyian commented Aug 6, 2026

Copy link
Copy Markdown
Member

1. The new OPTIONS guard never runs, and preflight stopped 404ing

routes/api.php:52-59 checks both flags and aborts 404. That closure is unreachable. PublicApiCors::handle returns 204 for every OPTIONS request before it calls $next, at PublicApiCors.php#L13-L16, and group middleware merges ahead of the route action, so the request never reaches your check.

Before this PR the group ran publicEventsApiEnabled first and that middleware 404'd OPTIONS while the flag was off. Dropping it from the group list means OPTIONS /api/public/v2/repairs now answers 204 plus Access-Control-Allow-* on an instance with both flags off, where it used to answer 404.

Nothing covers it. grep -in options tests/Feature/Repairs/PublicRepairsApiTest.php is empty. This should fail today:

config([
    'restarters.features.public_events_api' => false,
    'restarters.features.public_repairs_api' => false,
]);
$this->options('/api/public/v2/repairs')->assertStatus(404);

Route-level middleware won't fix it, since group middleware merges in front. The check has to go in the group list ahead of publicApiCors:

->middleware(['publicApiEnabled', 'publicApiCors'])

with publicApiEnabled aborting when neither flag is set, and the closure back to a bare noContent().

2. ?powered= returns only the unpowered dataset

PublicRepairController.php:75-86. filter_var maps an empty value to false, not to null, so FILTER_NULL_ON_FAILURE never fires:

$ php -r 'var_dump(filter_var("", FILTER_VALIDATE_BOOL, FILTER_NULL_ON_FAILURE));'
bool(false)
$ php -r 'var_dump(filter_var(null, FILTER_VALIDATE_BOOL, FILTER_NULL_ON_FAILURE));'
bool(false)

?powered= merges false, clears nullable|boolean, and applyFilters adds where categories.powered = 0. docs/public-repairs-api.md:52 says to omit the param to get both, and an empty param is how plenty of clients spell an unset one. test_powered_filter_accepts_true_and_false_spellings covers true and false but not the empty case.

$raw = $request->input('powered');

if ($raw === null || $raw === '') {
    return;
}

$normalised = filter_var($raw, FILTER_VALIDATE_BOOL, FILTER_NULL_ON_FAILURE);
Four non-blocking notes: CORS headers, the chart's problem-text default, the error body, the categories join

A browser can't read the CSV pagination headers

PublicRepairController.php:148-155 puts X-Total-Count, X-Last-Page and X-Max-Updated-At in headers, and the docs advertise both those headers and CORS. PublicApiCors::addHeaders (PublicApiCors.php:26-32) sets no Access-Control-Expose-Headers, so cross-origin JavaScript gets the body and none of the five headers. A browser consumer can't tell when it has hit the last page. One line fixes it, but it changes the events API too, so maybe its own PR.

A question: the chart ships ORDS_INCLUDE_PROBLEM: "true"

charts/restarters/values.yaml:375. Flipping FEATURE__PUBLIC_REPAIRS_API then also publishes free-text notes, and the scrubber leaves names in, which you say plainly in the docs. Is on the default you want for the shipped chart? Turning the endpoint on and publishing the free-text column feel like two decisions, and one env var covers both.

ORA's error body will read validation.max

Your test asserts it at tests/Feature/Repairs/PublicRepairsApiTest.php:619. There is no English validation file: find lang -path "*en*" -name validation.php returns nothing, while lang/instances/base/{it,nl,ne,de,fr}/validation.php all exist. A partner sending per_page=1001 gets the raw key instead of the ceiling. Pre-existing, not yours to fix here, but worth an issue since ORA reads it.

Put a number on the categories join

You list the inner join under Deferred. One count says whether it matters:

SELECT COUNT(*) FROM devices d
LEFT JOIN categories c ON c.idcategories = d.category
WHERE c.idcategories IS NULL;

Zero and it's a line in the docs. Not zero and the number belongs in the handover note.

test_every_seeded_category_has_a_vocabulary_mapping reading names straight out of the migrations is a good call. That gap would otherwise surface as a null product_category_id in ORA's data months later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants