diff --git a/README.md b/README.md index 40ceb58..c2255d6 100644 --- a/README.md +++ b/README.md @@ -36,4 +36,6 @@ The feed client uses the RSS feed and is similar to the web client except it ove ### ArcGIS REST Client -The ArcGIS REST client uses the ArcGIS REST API to retrieve incidents. This is the most accurate client since it uses the same data source as the LCWC website. This is still a bit of a prototype and may be subject to change. The ArcGIS REST client is the recommended client if you need more granular information such as static identifiers and coordinates. \ No newline at end of file +The ArcGIS REST client uses the ArcGIS REST API to retrieve incidents. This is the most accurate client since it uses the same data source as the LCWC website. This is still a bit of a prototype and may be subject to change. The ArcGIS REST client is the recommended client if you need more granular information such as static identifiers and coordinates. + +Note that `coordinates` is `None` for incidents the service has not geocoded, and that the service does not publish non-emergency EMS activity (`ROUTINE TRANSFER`, `EMS ACTIVITY`) at all. See [ArcGIS live feed service behavior](docs/arcgis-service-behavior.md) for the measured details. \ No newline at end of file diff --git a/docs/arcgis-service-behavior.md b/docs/arcgis-service-behavior.md new file mode 100644 index 0000000..a57f5f2 --- /dev/null +++ b/docs/arcgis-service-behavior.md @@ -0,0 +1,204 @@ +# ArcGIS live feed service behavior + +Notes on how the upstream ArcGIS service actually behaves, why `ArcGISClient` +queries each layer twice, and which gaps are outside the client's control. + +Measured 2026-08-11 over a 90 minute window, 180 samples at 30 second +intervals, comparing `ArcGISClient`, `FeedClient` and `WebClient` side by side. +18 distinct incidents passed through, covering all three layers. + +## The service + +``` +https://utility.arcgis.com/usrsvcs/servers/a1f6aa7faab44b1582029509c46dce86 + /rest/services/Maps/Public_LiveFeeds/MapServer +``` + +Three point layers, `maxRecordCount` 2000, no definition expression, no time +info: + +| Layer | Name | Category | +| ----- | ----------------- | --------- | +| 0 | Fire Incidents | `FIRE` | +| 1 | EMS Incidents | `MEDICAL` | +| 2 | Traffic Incidents | `TRAFFIC` | + +This is the same service the official LCWC live map consumes. The map at +`lcwc911.us/live-incident-map` embeds web app `cc2eb34d14ce4f2ba15e1f1a3ccb6a01`, +whose web map `5b0e126547b046e69e9559ea866bdf34` lists exactly this MapServer as +its `Live Incidents` operational layer with a 30 second refresh. There is no +richer or alternate endpoint to fall back on, which is why gaps in this service +show up identically on the official map. + +## Requesting geometry silently drops rows + +This is the root cause of [#3](https://github.com/NateShoffner/python-lcwc/issues/3). + +The service routinely holds rows whose geometry is null, because the address has +not been geocoded yet or never will be. **Any query that asks for geometry drops +those rows without comment**, and a spatial filter implies one. Same layer, same +instant, only the output flags varying: + +``` +geom=false=4 geom=true=2 geom=true,outSR=4326=2 countOnly=4 +``` + +`returnCountOnly` and `returnGeometry=false` agree on 4. Asking for geometry +returns 2. Identifying the rows that vanish: + +``` +OK oid=4 2608012247 WEST LAMPETER TOWNSHIP geom={'x': -8490111.67, 'y': 4863072.28} +OK oid=5 2608012253 CLAY TOWNSHIP geom={'x': -8487468.24, 'y': 4897429.80} +DROP oid=6 2608012253 CLAY TOWNSHIP +DROP oid=7 2608012270 MANHEIM BOROUGH | E HIGH ST / S HAZEL ST +``` + +The dropped rows carry no geometry at all: `returnExtentOnly=true` on either one +returns `{"xmin": "NaN", "ymin": "NaN", "xmax": "NaN", "ymax": "NaN"}`. They are +unreachable through every output variant tried, including `f=geojson`, +`returnCentroid=true`, `outSR=102100`, `geometryPrecision` and quantization. +Normally a null-geometry feature comes back as `"geometry": null`; this service +omits the row entirely. + +Manheim Borough above had no geocoded twin, so that incident was invisible to any +geometry-bearing query. + +**The spatial envelope was never the problem.** `returnGeometry=true` with no +spatial filter whatsoever still returns the reduced set, so widening the +envelope, splitting the county into quadrants, or dropping the filter changes +nothing. The envelope has been removed because `where=1=1` alone already returns +every row the service holds, and the filter could only ever subtract. + +Geometry also flaps. Clay Township had coordinates at 23:51 and none at 23:55, +so a row that answers a geometry query now may not answer the next one. + +### How the client works around it + +Each layer is queried twice: once with `returnGeometry=false` for the +authoritative row set, once with `returnGeometry=true` purely as a coordinate +lookup keyed on `IncidentNumber`. Incidents come from the first query, so nothing +is lost; coordinates are attached when the second query has them. + +`ArcGISIncident.coordinates` is therefore `Optional[Coordinates]`. Over the +measurement window the client returned coordinates for 741 of 775 incidents +(96%). + +## Duplicate rows + +The service sometimes carries more than one row for a single incident, typically +one geocoded and one not (`oid=5` and `oid=6` above are both incident +2608012253, identical in every attribute). The client dedupes on +`IncidentNumber` per layer. + +Duplicate and null-geometry rows appeared in 26 of 180 samples, clustered in +bursts rather than spread evenly. Across the window a geometry-bearing query +would have dropped 4.7% of rows on average, and 50% in the worst sample. + +## Incidents the service never publishes + +Four incidents in the window never appeared in the service at any point in their +lifetimes, across every sample the feed carried them: + +``` +[NEVER] 150/150 EAST LAMPETER|LANDIS AVE + LINCOLN HWY E ROUTINE TRANSFER-CLASS 3 +[NEVER] 99/99 ELIZABETH|HOPELAND RD + LEE LN ROUTINE TRANSFER-CLASS 3 +[NEVER] 40/40 QUARRYVILLE|PARK AVE + S HESS ST EMS ACTIVITY +[NEVER] 30/30 FULTON|ROBERT FULTON HWY + WARFEL RD ROUTINE TRANSFER-CLASS 3 +``` + +Only two types, `ROUTINE TRANSFER-CLASS 3` and `EMS ACTIVITY`. Every +emergency-type incident reached the service, including fire and traffic. This +looks like a deliberate upstream filter on the public map rather than a defect, +and no client-side change can recover them. Callers who need complete coverage +of non-emergency EMS activity should use `FeedClient` or `WebClient`. + +## Timing and transient dropouts + +Aggregated over the 14 incidents that did reach the service, 790 samples of +lifetime: + +| Behavior | Measurement | +| ----------------------------------------------- | --------------------- | +| Ingest lag before the service picks an incident up | ~19s | +| Trailing presence after the feed clears it | ~13s | +| Mid-lifetime single-sample dropouts | 6 (0.76% of samples) | + +The dropouts are the notable one. An incident the service is already carrying can +vanish for a single poll and return on the next, presumably during the same table +rebuild that produces the empty-layer blips below. Rendering two incident +lifetimes at 30s per character, `F` = feed only, `B` = both, `A` = ArcGIS only: + +``` +LANCASTER|CHESTER ST + S DUKE ST FBFBBBBBBBBBBB...BBBBFBBBBBB...BBBBA +EAST PETERSBURG|SUNDRA CIR FBBBBBBBBBBBBB...BBBBFBBBBBB...BBBBB +``` + +A consumer diffing consecutive polls will see spurious clear/re-open pairs at +roughly 1 sample in 130. Debouncing by one poll suppresses it. The client does +not do this, because a genuine clear-down is indistinguishable from a dropout +within a single sample and the delay would be paid on every incident. + +Separately, the whole layer occasionally returns zero rows mid-refresh: 2 of 151 +samples in an earlier 6 minute run at 2 second intervals, 0 of 180 in the 90 +minute run. A poller can misread that as "all incidents cleared". A retry on +empty would mask it, but zero is also legitimate during quiet hours, so the +client does not retry. + +## Unrelated: the feed client drops MICU units + +Found while cross-checking the three clients, not an ArcGIS issue. + +`MEDICAL_UNIT_NAMES` in `src/lcwc/feed/utils/__init__.py` is +`["AMB", "EMS", "INT", "MEDIC", "QRS"]`. `MICU` is missing, and that one omission +causes two failures, because the list gates both `has_unit_names()` in +`feed/parser.py` and `determine_category()`: + +1. The units segment is not recognized as units, so `unit_names` stays empty and + the MICU unit is discarded entirely. +2. With no units, classification falls through to the description keyword check, + which only matches `MEDICAL`. `EMS ACTIVITY` and `ROUTINE TRANSFER-CLASS 3` + therefore land in `UNKNOWN`. + +Same instant, same incidents: + +``` +FEED: UNKNOWN QUARRYVILLE BOROUGH units=[] EMS ACTIVITY + MEDICAL CLAY TOWNSHIP units=[] MEDICAL EMERGENCY + UNKNOWN FULTON TOWNSHIP units=[] ROUTINE TRANSFER-CLASS 3 +WEB: MEDICAL Quarryville Borough units=['MICU'] EMS ACTIVITY + MEDICAL Clay Township units=['MICU'] MEDICAL EMERGENCY + MEDICAL Fulton Township units=['MICU'] ROUTINE TRANSFER-CLASS 3 +``` + +Clay Township got `MEDICAL` only because its description happened to say so, +which masks the unit loss. The raw feed confirms the units are present upstream: +`QUARRYVILLE BOROUGH; PARK AVE & S HESS ST; MICU 56-5;`. + +`WebClient` is immune because it reads the category from the page's section +header instead of inferring it. `UnitParser` handles `MICU` fine, so the keyword +list is the only gate. This was observed across 169 samples and is not fixed. + +## Feed and web agree + +`FeedClient` and `WebClient` returned identical incident sets in 180 of 180 +samples. The only differences between them are the categories above. + +## Reproducing + +Incidents are matched across sources by normalized location, since the three +sources share no identifier: the feed has a guid, the web page has nothing, and +ArcGIS has `IncidentNumber`. Two genuinely distinct co-located incidents (a +structure fire and its EMS dispatch, filed as separate CAD incidents at one +address) therefore collapse into a single key. This happened in about 10 of 180 +samples and is a limitation of any cross-source comparison, not of the clients. + +The quickest way to observe the core defect directly: + +``` +GET .../MapServer/1/query?f=json&where=1=1&returnCountOnly=true +GET .../MapServer/1/query?f=json&where=1=1&outFields=IncidentNumber&returnGeometry=false +GET .../MapServer/1/query?f=json&where=1=1&outFields=IncidentNumber&returnGeometry=true +``` + +Whenever the third disagrees with the first two, the difference is rows the +service holds but will not hand over with geometry attached. diff --git a/examples/arcgis_example.py b/examples/arcgis_example.py index 8029260..e9ca942 100644 --- a/examples/arcgis_example.py +++ b/examples/arcgis_example.py @@ -21,8 +21,14 @@ async def main(): print(f"Number: {incident.number}") print(f"Priority: {incident.priority}") print(f"Agency: {incident.agency}") + coordinates = incident.coordinates print( - f"Coordinates: {incident.coordinates.latitude}, {incident.coordinates.longitude}" + "Coordinates: " + + ( + f"{coordinates.latitude}, {coordinates.longitude}" + if coordinates + else "None" + ) ) print("-----") diff --git a/pyproject.toml b/pyproject.toml index c477887..43b2248 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "lcwc" -version = "0.13.0" +version = "0.14.0" authors = [ { name="Nate Shoffner", email="nate.shoffner@gmail.com" }, ] diff --git a/src/lcwc/__init__.py b/src/lcwc/__init__.py index ac11253..1f3c7c9 100644 --- a/src/lcwc/__init__.py +++ b/src/lcwc/__init__.py @@ -2,7 +2,7 @@ __author__ = "Nate Shoffner" __license__ = "MIT" __copyright__ = "Copyright 2023-present Nate Shoffner" -__version__ = "0.12.3" +__version__ = "0.14.0" from .agencies.agency import Agency from .agencies.agencyclient import AgencyClient diff --git a/src/lcwc/arcgis/client.py b/src/lcwc/arcgis/client.py index 17001f0..7d6c075 100644 --- a/src/lcwc/arcgis/client.py +++ b/src/lcwc/arcgis/client.py @@ -1,7 +1,6 @@ import logging import aiohttp import datetime -import json import re from lcwc import Client @@ -80,6 +79,12 @@ async def get_incidents( ], } + adapter = RestAdapter( + session, + "utility.arcgis.com", + "usrsvcs/servers/a1f6aa7faab44b1582029509c46dce86/rest/services/Maps/Public_LiveFeeds/MapServer/", + ) + incidents = [] for cat in IncidentCategory: @@ -92,81 +97,109 @@ async def get_incidents( layer_id = layer_mapping[cat] - """ Actual spatial extent of Lancaster County based LanCo GIS data - lanco_spatial = { - 'xmin': -8548898.732776089, - 'ymin': 4845979.963808246, - 'xmax': -8432714.449782776, - 'ymax': 4909881.3194545675, - 'spatialReference': { - 'wkid': 102100 - } - } - """ - - # seems we need to expand the spatial extent to get all incidents - lanco_spatial = { - "xmin": -8657540.868810708, - "ymin": 4794222.228992932, - "xmax": -8290643.133041878, - "ymax": 5048910.407239126, - "spatialReference": {"wkid": 102100}, + # The service routinely holds rows whose geometry is null, either + # because the address has not been geocoded yet or never will be. + # Any query that asks for geometry (and any spatial filter, which + # implies one) silently drops those rows, so the incidents are + # fetched geometry-free and the coordinates are looked up with a + # second query that only the geocoded rows answer. + attribute_params = { + "f": "json", + "where": "1=1", + "returnGeometry": "false", + "outFields": ",".join(fields[cat]), } - params = { + geometry_params = { "f": "json", "where": "1=1", "returnGeometry": "true", - "spatialRel": "esriSpatialRelIntersects", - "geometry": json.dumps(lanco_spatial), - "geometryType": "esriGeometryEnvelope", - "inSR": 102100, - "outFields": ",".join(fields[cat]), + "outFields": "IncidentNumber", "outSR": 4326, # return coordinates in WGS84 - "currentTimestamp": int( - datetime.datetime.now().timestamp() * 1000 - ), # add a timestamp to prevent caching } - adapter = RestAdapter( - session, - "utility.arcgis.com", - "usrsvcs/servers/a1f6aa7faab44b1582029509c46dce86/rest/services/Maps/Public_LiveFeeds/MapServer/", - ) - try: - resp = await adapter.get(endpoint=f"{layer_id}/query", ep_params=params) - - except RestException as e: + features = await self.__query_layer( + adapter, layer_id, cat, attribute_params + ) + except (RestException, ArcGISException) as e: self.logger.error(f"{cat} Error: {e}") if throw_on_error: raise e continue - self.logger.debug(f"{resp.url}") - - if resp.status_code != 200: - if throw_on_error: - raise ArcGISException(error) - self.logger.error(f"Error: {resp.status_code} for {cat}") + if not features: continue - error = resp.data.get("error", None) - if error: + # a failed lookup only costs the coordinates, so the incidents are + # still worth returning without them + try: + located = await self.__query_layer( + adapter, layer_id, cat, geometry_params + ) + except (RestException, ArcGISException) as e: + self.logger.error(f"{cat} Coordinates error: {e}") if throw_on_error: - raise ArcGISException(error) - self.logger.error(f"Response error: {error}") - continue - - if "features" not in resp.data: - continue - - for feature in resp.data["features"]: - incident = self.__parse_incident(cat, feature, self.agency_resolver) + raise e + located = [] + + coordinates = {} + for feature in located: + geometry = feature.get("geometry") + if geometry is None: + continue + number = feature["attributes"]["IncidentNumber"] + coordinates[number] = geometry + + # the same incident occasionally occupies more than one row, once + # geocoded and once not + seen = set() + + for feature in features: + number = feature["attributes"]["IncidentNumber"] + if number in seen: + self.logger.debug(f"Skipping duplicate row for incident {number}") + continue + seen.add(number) + + incident = self.__parse_incident( + cat, + { + "attributes": feature["attributes"], + "geometry": coordinates.get(number), + }, + self.agency_resolver, + ) incidents.append(incident) return incidents + async def __query_layer( + self, + adapter: RestAdapter, + layer_id: int, + category: IncidentCategory, + params: dict, + ) -> list[dict]: + """Queries a single layer and returns its raw features""" + + params = dict(params) + # add a timestamp to prevent caching + params["currentTimestamp"] = int(datetime.datetime.now().timestamp() * 1000) + + resp = await adapter.get(endpoint=f"{layer_id}/query", ep_params=params) + + self.logger.debug(f"{resp.url}") + + if resp.status_code != 200: + raise ArcGISException(f"Error: {resp.status_code} for {category}") + + error = resp.data.get("error", None) + if error: + raise ArcGISException(f"Response error: {error}") + + return resp.data.get("features", []) + def __parse_incident( self, category: IncidentCategory, @@ -174,7 +207,7 @@ def __parse_incident( agency_resolver: AgencyResolver = None, ) -> ArcGISIncident: attributes = incident["attributes"] - geometry = incident["geometry"] + geometry = incident.get("geometry") # IncidentOrigination is epoch milliseconds, which is already an absolute # instant, so it converts directly to UTC with no local timezone involved @@ -200,7 +233,7 @@ def __parse_incident( number = int(attributes["IncidentNumber"]) - if "Priority" in attributes: + if attributes.get("Priority") is not None: priority = int(attributes["Priority"]) else: priority = None @@ -208,7 +241,7 @@ def __parse_incident( public = bool(attributes["IsPublic"]) description = attributes["PublicType"] - coords = Coordinates(geometry["x"], geometry["y"]) + coords = Coordinates(geometry["x"], geometry["y"]) if geometry else None incident = ArcGISIncident( category, diff --git a/src/lcwc/arcgis/incident.py b/src/lcwc/arcgis/incident.py index b791560..0833158 100644 --- a/src/lcwc/arcgis/incident.py +++ b/src/lcwc/arcgis/incident.py @@ -1,4 +1,5 @@ from dataclasses import dataclass +from typing import Optional from lcwc.incident import Incident from collections import namedtuple @@ -30,5 +31,5 @@ class ArcGISIncident(Incident): """ Whether the incident is public """ public: bool - """ The coordinates of the incident """ - coordinates: Coordinates + """ The coordinates of the incident, or None if the service has not geocoded it """ + coordinates: Optional[Coordinates] diff --git a/src/lcwc/utils/encoding.py b/src/lcwc/utils/encoding.py index 0a97f77..428a228 100644 --- a/src/lcwc/utils/encoding.py +++ b/src/lcwc/utils/encoding.py @@ -49,7 +49,7 @@ def decode(self, s, incident_type): obj["date"] = datetime.datetime.fromisoformat(obj["date"]) if "units" in obj: obj["units"] = [Unit(**unit) for unit in obj["units"]] - if "coordinates" in obj: + if obj.get("coordinates") is not None: obj["coordinates"] = Coordinates(**obj["coordinates"]) return incident_type(**obj) diff --git a/tests/arcgis_client_test.py b/tests/arcgis_client_test.py index 950d5c8..749b95b 100644 --- a/tests/arcgis_client_test.py +++ b/tests/arcgis_client_test.py @@ -121,6 +121,22 @@ def test_date_is_independent_of_host_timezone(self): ) +class ArcGISGeometryTest(unittest.TestCase): + def test_incident_without_geometry_is_parsed(self): + """The service leaves geometry null on incidents it has not geocoded, + and those incidents still belong in the results""" + client = ArcGISClient() + feature = make_feature(DATE_CASES[0][0]) + del feature["geometry"] + + incident = client._ArcGISClient__parse_incident( + IncidentCategory.MEDICAL, feature, None + ) + + self.assertIsNone(incident.coordinates) + self.assertEqual(incident.number, 2026000123) + + class WebClientTest(IsolatedAsyncioTestCase): async def test_fetch(self): async with aiohttp.ClientSession() as session: @@ -130,6 +146,9 @@ async def test_fetch(self): self.assertIsNotNone(incidents) self.assertIsInstance(incidents, list, "") + if not incidents: + self.skipTest("no active incidents to check") + first_incident = incidents[0] self.assertIsNotNone(first_incident) self.assertIsInstance(first_incident, ArcGISIncident, "") @@ -142,7 +161,10 @@ async def test_fetch(self): self.assertIsNotNone(first_incident.number) # self.assertIsNotNone(first_incident.priority) self.assertIsNotNone(first_incident.agency) - self.assertIsNotNone(first_incident.coordinates) + # coordinates are absent for incidents the service has not geocoded + + numbers = [incident.number for incident in incidents] + self.assertCountEqual(numbers, set(numbers), "incidents must be unique") if __name__ == "__main__":