Branching Loader - aggregating / sorting / grouping transformations bug - #2599
Merged
Conversation
- drive the wrapped loader from `->get()` instead of terminating the nested frame with `->load()->run()` - wrapped loader now receives the outer FlowContext - SpyLoader implements Loader\Closure so the close storm is observable - regression coverage for 9 stateful loaders across 7 packages - drop dead private helpers from ParquetTest
- RetryLoader, TransactionalDbalLoader and TransactionalPostgreSqlLoader now implement Closure and OverridingLoader - fixes silent data loss under overwrite(), where the destination file was never published and a temp file was orphaned - closing stays outside the retry loop, a failed commit is not resumable - documentation no longer recommends wrapping file loaders in write_with_retries()
…er Config - save mode and open write streams no longer leak between DataFrames sharing a Config - remove Config::filesystemStreams() and its constructor parameter - document the per-DataFrame scope in save-mode.md and upgrading.md
…a time - explain the Processor vs Transformer rule, list the affected and safe operations - call out offset() and cache() as data loss, not just wrong ordering - give the outer ->collect() / ->batchSize(-1) workaround - add characterization tests pinning today's batch-local behaviour
…whole stream - drive the nested pipeline once per run inside a Fiber, fed batch by batch - replace the public SwappableRowsExtractor with an internal FeedExtractor - route a failed drive through the ErrorHandler and refuse retrying a Transformation - exclude InvalidLogicException from the default retry strategy - drop the redundant collect() and sortBy() from withEntry()'s window branch
- walk the loader tree in BatchSizeOptimization::isFor() - add LoaderTree, first consumer of OverridingLoader::loaders() - benchmark a wrapped Doctrine write
…ltered stream - TransformationDrive extracted from TransformerLoader; BranchingLoader shares it instead of rebuilding a DataFrame per batch - FeedExtractor moved to the Pipeline namespace - new ReplayAware contract; RetryLoader refuses transformation drives at any nesting depth - ATTR_LOADING_ROWS on to_branch now counts input rows - docs: default retry strategy and the wrap refusal documented, upgrading entries extended - test hygiene: narrowed exception captures, optimizer test dropped its DbalLoader dependency, transformation benchmarks added
- to_dbal_transaction() / to_pgsql_transaction() open one final transaction around the forwarded closure() drain, so blocking transformations commit atomically and roll back on failure - a failing rollBack() never masks the original load/drain error - withIsolationLevel() applies to every transaction the wrapper opens - composition tests in both adapters + nested-transformation pins in core - docs: new doctrine Transactional Loading section, postgresql rewrite, retry/transaction cross-references, upgrading entry 10 - pgsql adapter DSL functions added to the dsl.json generator paths
…ument - BranchingLoader takes the optional Transformation in its constructor; withTransformation() replaces it - mark LoaderTree as @internal like its drive siblings - landing: new data_writing/branch example routing rows to different destinations by condition - regenerate DSL json dump and CodeMirror completions - document the new signature in transformations.md
…tream - move FeedExtractor from Flow\ETL\Pipeline to Flow\ETL\Extractor - LoaderTree: offsetExists()/offsetSet() instead of contains()/attach(), deprecated on PHP 8.5 - phpunit: fail on first-party deprecations (failOnDeprecation + ignoreIndirectDeprecations) - trim narration comments in wrapper loaders
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 1.x #2599 +/- ##
============================================
+ Coverage 86.45% 86.51% +0.06%
- Complexity 23309 23988 +679
============================================
Files 1834 1837 +3
Lines 72331 72461 +130
============================================
+ Hits 62531 62688 +157
+ Misses 9800 9773 -27 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Log
Added
flow-php/etl-to_branch()accepts aTransformationas an optional third argumentflow-php/etl-retry_any_throwable_except()DSL andAnyThrowableExceptretry strategyflow-php/etl-adapter-doctrine-TransactionalDbalLoader::loaders()exposes wrapped loadersflow-php/etl-adapter-postgresql-TransactionalPostgreSqlLoader::loaders()exposes wrapped loadersFixed
flow-php/etl- transformation-wrapped loader is closed once per pipeline, not once per batchflow-php/etl- loader wrappers forwardclosure()to the loaders they wrapflow-php/etl- operations inside aTransformationanswer for the whole stream, not per batchflow-php/etl-to_branch()->withTransformation()spans the whole filtered streamflow-php/etl- batch size optimization sees through loader wrappersflow-php/etl- save mode no longer leaks between frames sharing oneConfigflow-php/etl- noSplObjectStoragedeprecations on PHP 8.5flow-php/etl-adapter-doctrine-to_dbal_transaction()runsclosure()inside a transactionflow-php/etl-adapter-postgresql-to_pgsql_transaction()runsclosure()inside a transactionChanged
flow-php/etl-write_with_retries()no longer retriesInvalidLogicExceptionby defaultflow-php/etl-write_with_retries()refuses wrapping an armedTransformationflow-php/etl- aTransformationtriggering its own frame (fetch()/count()/schema()) throwsInvalidLogicExceptionflow-php/etl- unpartitioned window functions no longer injectcollect()andsortBy()stagesRemoved
flow-php/etl-Extractor\SwappableRowsExtractorflow-php/etl-Config::filesystemStreams()and theFilesystemStreamsconstructor parameterDeprecated
Security