Skip to content

Upgrade to patchlevel/hydrator 2.0 - #900

Merged
DavidBadura merged 5 commits into
4.0.xfrom
hydrator-2
Sep 24, 2026
Merged

DavidBadura merged 5 commits into
4.0.xfrom
hydrator-2

Conversation

@DavidBadura

Copy link
Copy Markdown
Member

Updates the hydrator to 2.0 and drops our own upcaster in favor of the hydrator's UpcastExtension. Upcasters now work per event class, so renaming events is done with event aliases instead.

DefaultEventSerializer::createFromPaths() and DefaultSnapshotStore::createDefault() now take a configured hydrator instead of an upcaster and cryptographer, and the $upcaster constructor argument of the serializer is gone.

Hydrator 2.0 removed the legacy cryptography, so the old DoctrineCipherKeyStore (crypto_keys table) is removed and ExtensionDoctrineCipherKeyStore is renamed to DoctrineCipherKeyStore. Data encrypted with the legacy cryptographer can't be read anymore, the upgrade guide recommends migrating it while still on 3.x.

All breaking changes are documented in the 4.0 upgrade guide, and the upcasting and personal data docs are updated.

Replace our own upcaster with the UpcastExtension of the hydrator and switch
the serializer and snapshot store to take a configured hydrator instead of
an upcaster and cryptographer. The legacy cryptography is gone in hydrator
2.0, so the old DoctrineCipherKeyStore is removed and the extension based
store takes over its name.
@DavidBadura DavidBadura added this to the 4.0.0 milestone Sep 23, 2026
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Hello 👋

here is the most recent benchmark result:

SubscriptionEngineBatchBench
============================

+---------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
|                           | time (kde mode)                                     | memory                                     |
+---------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| subject                   | Tag: <current>     | Tag: base          | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+---------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| benchHandle10000Events () | 101.515ms (±0.00%) | 107.508ms (±0.00%) | -5.57%    | 36.239mb        | 36.244mb   | -0.02%      |
+---------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+

PersonalDataBench
=================

+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
|                                        | time (kde mode)                                     | memory                                     |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| subject                                | Tag: <current>     | Tag: base          | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| benchLoad1Event ()                     | 1.322ms (±0.00%)   | 1.024ms (±0.00%)   | +29.05%   | 36.875mb        | 35.809mb   | +2.98%      |
| benchLoad10000Events ()                | 151.798ms (±0.00%) | 89.608ms (±0.00%)  | +69.40%   | 36.875mb        | 35.810mb   | +2.98%      |
| benchSave1Event ()                     | 2.214ms (±0.00%)   | 1.848ms (±0.00%)   | +19.81%   | 36.875mb        | 35.809mb   | +2.98%      |
| benchSave10000Events ()                | 388.752ms (±0.00%) | 351.816ms (±0.00%) | +10.50%   | 36.930mb        | 35.811mb   | +3.12%      |
| benchSave10000Aggregates ()            | 12.434s (±0.00%)   | 13.946s (±0.00%)   | -10.84%   | 36.875mb        | 35.810mb   | +2.98%      |
| benchSave10000AggregatesTransaction () | 7.581s (±0.00%)    | 7.160s (±0.00%)    | +5.88%    | 36.875mb        | 36.050mb   | +2.29%      |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+

SubscriptionEngineBench
=======================

+---------------------------+-----------------+-----------------+-----------+-----------------+------------+-------------+
|                           | time (kde mode)                               | memory                                     |
+---------------------------+-----------------+-----------------+-----------+-----------------+------------+-------------+
| subject                   | Tag: <current>  | Tag: base       | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+---------------------------+-----------------+-----------------+-----------+-----------------+------------+-------------+
| benchHandle10000Events () | 2.456s (±0.00%) | 2.501s (±0.00%) | -1.82%    | 48.301mb        | 48.300mb   | +0.00%      |
+---------------------------+-----------------+-----------------+-----------+-----------------+------------+-------------+

NoopSubscriptionEngineBench
===========================

+---------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+
|                           | time (kde mode)                                   | memory                                     |
+---------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+
| subject                   | Tag: <current>    | Tag: base         | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+---------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+
| benchHandle10000Events () | 93.592ms (±0.00%) | 96.467ms (±0.00%) | -2.98%    | 48.301mb        | 48.300mb   | +0.00%      |
+---------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+

SplitStreamBench
================

+-------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
|                         | time (kde mode)                                     | memory                                     |
+-------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| subject                 | Tag: <current>     | Tag: base          | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+-------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| benchLoad10000Events () | 3.175ms (±0.00%)   | 3.835ms (±0.00%)   | -17.21%   | 35.774mb        | 35.773mb   | +0.00%      |
| benchSave10000Events () | 511.109ms (±0.00%) | 537.642ms (±0.00%) | -4.94%    | 35.909mb        | 35.908mb   | +0.00%      |
+-------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+

SimpleSetupTaggableStoreBench
=============================

+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
|                                        | time (kde mode)                                     | memory                                     |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| subject                                | Tag: <current>     | Tag: base          | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| benchLoad1Event ()                     | 1.365ms (±0.00%)   | 1.013ms (±0.00%)   | +34.74%   | 36.246mb        | 36.311mb   | -0.18%      |
| benchLoad10000Events ()                | 64.263ms (±0.00%)  | 64.020ms (±0.00%)  | +0.38%    | 36.246mb        | 36.311mb   | -0.18%      |
| benchSave1Event ()                     | 1.799ms (±0.00%)   | 1.333ms (±0.00%)   | +34.91%   | 36.246mb        | 36.311mb   | -0.18%      |
| benchSave10000Events ()                | 302.051ms (±0.00%) | 298.500ms (±0.00%) | +1.19%    | 36.358mb        | 36.311mb   | +0.13%      |
| benchSave10000Aggregates ()            | 8.392s (±0.00%)    | 9.324s (±0.00%)    | -10.00%   | 36.246mb        | 36.311mb   | -0.18%      |
| benchSave10000AggregatesTransaction () | 3.727s (±0.00%)    | 3.777s (±0.00%)    | -1.33%    | 36.247mb        | 36.311mb   | -0.18%      |
| benchAppend1Event ()                   | 1.618ms (±0.00%)   | 1.489ms (±0.00%)   | +8.63%    | 36.246mb        | 36.311mb   | -0.18%      |
| benchAppend100Events ()                | 8.226ms (±0.00%)   | 7.613ms (±0.00%)   | +8.05%    | 36.246mb        | 36.311mb   | -0.18%      |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+

SnapshotsBench
==============

+----------------------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+
|                                        | time (kde mode)                                   | memory                                     |
+----------------------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+
| subject                                | Tag: <current>    | Tag: base         | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+----------------------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+
| benchLoad10000EventsMissingSnapshot () | 62.148ms (±0.00%) | 62.441ms (±0.00%) | -0.47%    | 35.151mb        | 35.216mb   | -0.18%      |
| benchLoad10000Events ()                | 1.149ms (±0.00%)  | 1.196ms (±0.00%)  | -3.93%    | 35.151mb        | 35.216mb   | -0.18%      |
+----------------------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+

CommandToQueryBench
===================

+----------------+------------------+------------------+-----------+-----------------+------------+-------------+
|                | time (kde mode)                                 | memory                                     |
+----------------+------------------+------------------+-----------+-----------------+------------+-------------+
| subject        | Tag: <current>   | Tag: base        | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+----------------+------------------+------------------+-----------+-----------------+------------+-------------+
| benchCreate () | 3.748ms (±0.00%) | 4.373ms (±0.00%) | -14.28%   | 4.934mb         | 4.968mb    | -0.67%      |
| benchUpdate () | 6.984ms (±0.00%) | 5.439ms (±0.00%) | +28.42%   | 5.212mb         | 5.279mb    | -1.26%      |
| benchBoth ()   | 9.150ms (±0.00%) | 9.349ms (±0.00%) | -2.12%    | 5.072mb         | 5.416mb    | -6.35%      |
+----------------+------------------+------------------+-----------+-----------------+------------+-------------+

SimpleSetupStreamStoreBench
===========================

+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
|                                        | time (kde mode)                                     | memory                                     |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| subject                                | Tag: <current>     | Tag: base          | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| benchLoad1Event ()                     | 1.347ms (±0.00%)   | 983.200μs (±0.00%) | +37.00%   | 35.146mb        | 35.145mb   | +0.00%      |
| benchLoad10000Events ()                | 62.343ms (±0.00%)  | 63.159ms (±0.00%)  | -1.29%    | 35.146mb        | 35.145mb   | +0.00%      |
| benchSave1Event ()                     | 2.247ms (±0.00%)   | 1.181ms (±0.00%)   | +90.27%   | 35.146mb        | 35.145mb   | +0.00%      |
| benchSave10000Events ()                | 278.721ms (±0.00%) | 273.428ms (±0.00%) | +1.94%    | 35.146mb        | 35.145mb   | +0.00%      |
| benchSave10000Aggregates ()            | 8.113s (±0.00%)    | 8.137s (±0.00%)    | -0.30%    | 35.146mb        | 35.146mb   | +0.00%      |
| benchSave10000AggregatesTransaction () | 3.705s (±0.00%)    | 3.660s (±0.00%)    | +1.23%    | 35.147mb        | 35.146mb   | +0.00%      |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+

This comment gets update everytime a new commit comes in!

@DavidBadura

Copy link
Copy Markdown
Member Author

PersonalDataBench::benchLoad10000Events got a lot slower with this change (116ms before, 2.7s after). The old legacy DoctrineCipherKeyStore had an internal key cache, the new one doesn't, so loading 10k events now runs one query per event.

I don't want to put an unbounded array cache back into the store, since in long running workers it would never be cleared and removed keys would still be usable there. The plan is to use the Psr6/Psr16CacheStoreDecorator from the hydrator with a TTL and max items instead. That didn't work yet because the decorators used reserved characters in their cache keys and didn't evict everything on removal, which is fixed in patchlevel/hydrator#207.

Once that is merged up to 2.0.x and released, I'll bump the hydrator here, switch the benchmark to the decorator (symfony/cache as dev dependency) and add a section about caching to the personal data docs.

The event serializer now passes the stored event name and the event class
as context to the hydrator, so upcasters can tell under which name an event
was stored, which matters since renames are handled with aliases. The
headers serializer factories accept a custom hydrator as well.

The personal data page is renamed to sensitive data to match the new
SensitiveData attribute, and the upgrade guide covers the moved attributes.
@DavidBadura DavidBadura mentioned this pull request Sep 23, 2026
The old DoctrineCipherKeyStore cached keys internally, the new one does not,
so loading an aggregate ran one query per encrypted value. Instead of an
unbounded cache in the store, which never gets cleared in long running
workers, the benchmark and the docs now use the cache decorator of the
hydrator with a lifetime and a limit. This needs hydrator 2.0.2, which
fixes the cache keys and eviction of the decorators.
phpbench runs the benchmarks in a subprocess, which did not get the memory
limit from the Makefile. With xdebug enabled locally, benchSave10000Events
ran out of memory, and xdebug also skews the measured times.
The COLUMNS env in the phpunit config did not override an existing value,
so console tests failed in the infection subprocess, where the SymfonyStyle
error blocks got wrapped at 80 columns. The infection-diff target now uses
the same options as CI, since --only-covered no longer exists.
@DavidBadura
DavidBadura merged commit eceea45 into 4.0.x Sep 24, 2026
38 of 39 checks passed
@DavidBadura
DavidBadura deleted the hydrator-2 branch September 24, 2026 06:20
@DanielBadura DanielBadura linked an issue Oct 1, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support Upcasting of Headers (or Messages)

2 participants