feat: Add flag dependency test cases by khvn26 · Pull Request #59 · Flagsmith/engine-test-data

khvn26 · 2026-08-27T17:13:19Z

Closes #60.

Corpus coverage for dependent flags (Flagsmith/flagsmith#8394, schema in Flagsmith/flagsmith#8396).

Notes for some of the test cases:

cyclic__should_not_override covers input that should be unreachable, since cycles are meant to be rejected where dependencies are written; we need to make sure we won't get stack overflows in case of a bug, though!
cyclic_enabled_prerequisite__segment_should_not_match: both flags in cyclic__should_not_override are disabled, so cutting the cycle leaves nothing truthy for another condition to match on and any implementation passes it. Here one flag is enabled.
context_supplied_flags__should_be_ignored: $.flags is readOnly in the context schema because the engine populates it during evaluation. The engine should expect no pre-populated flags in the context.

How did you test this code?

Contexts and results validate against the schemas from feat(sdk): Add flag dependencies to the evaluation context flagsmith#8396.
All 18 pass against the implementation in feat: Evaluate flags that depend on other flags flagsmith-engine#343.
cyclic_enabled_prerequisite__segment_should_not_match was checked to fail against that implementation before it was fixed, so it is a regression test rather than a restatement of current behaviour.

Covers segments conditioned on another flag's result via a `$.flags.<feature name>` condition property, as used by dependent flags: - a dependency satisfied, and the same dependency unsatisfied - a dependency on a flag's value rather than on whether it is enabled - a transitive chain, declared in reverse dependency order so that resolving flags in context order is not enough to pass - a cycle, which must terminate rather than recurse. Both flags resolve to their defaults whichever is resolved first, so the expectation does not depend on iteration order - a context arriving with `$.flags` already populated, which must be discarded rather than allowed to satisfy its own dependency

REVERT BEFORE MERGE. `EvaluationContext.flags` only exists on the schema branch of Flagsmith/flagsmith#8396, so validating the new test cases against `refs/heads/main` silently proves nothing: the key is simply unknown to the schema, and `EvaluationContext` doesn't set `additionalProperties` to false, so anything at all passes. Points `schema.json` at the context schema on that branch, and the new test cases at `schema.json` on this one, so that validation is meaningful while both are in review. With this, `check-jsonschema` rejects e.g. a non-boolean `$.context.flags.<name>.enabled`, which it accepted before. Once #8396 is merged, both refs should go back to `main` (and the test cases to a tag, in line with the rest of the corpus).

Extends the dependent flags coverage with cases that were previously carried as unit tests in flagsmith-engine, and so proved nothing about any other implementation: - `$.flags.a['enabled']` and `$.flags['my feature'].enabled`, two spellings an implementation matching the property as text rather than parsing it is liable to miss - `$['flags']['a'].enabled`, which is deliberately *not* a dependency: only a `$.`-prefixed property is a JSONPath query, and a bracket-rooted one is a trait key - a dependency in a nested rule group rather than a top-level condition - a dependency on `variant` rather than `enabled` - a transitive chain whose root is disabled, the counterpart to the cascading case - a dependency on a feature absent from the context - competing overrides, to pin that resolving dependencies doesn't disturb override precedence - a segment conditioned on a flag but overriding nothing, which still has to be evaluated for its membership to be reported Expected results were generated by running the engine, as in #57.

Adds support for segments conditioned on another flag's result, via a `$.flags.<feature name>` condition property, as the evaluation half of dependent flags. Segment conditions are already JSONPath, so a dependency needs no new operator or condition type. What it does need is a change to the shape of evaluation: segments were evaluated in full before any flag, but a segment conditioned on `$.flags.checkout_v2.enabled` needs that flag resolved first. Flags and segment membership are therefore resolved lazily and memoised, with resolved flags accumulated into a single dict the context refers to, so no context copying is required. Which features a segment depends on is established by inspecting the compiled JSONPath query's segments, rather than by matching the property as text. Every spelling of a query normalises to the same selectors before we look at it, so there is no grammar to reimplement and no divergence to keep in sync across engines. Only a query naming a single flag is a dependency: conditions resolve to a scalar, so a wildcard, index, slice or descendant search is unsupported to begin with, and is left undetected rather than resolving every flag on the off-chance. Contexts whose segments have no flag conditions keep the existing single-pass evaluation, so results are unchanged unless dependencies are in use. They do, however, pay for the scan that establishes there are no dependencies: 10-19% of evaluation across the benchmark contexts, and 11-15% for segment-heavy ones. Precomputing this where segments are written, and carrying it on the context, would remove it; `get_segment_dependencies` is public so that a caller holding a long-lived environment document can at least scan once rather than per evaluation. A flag caught in a dependency cycle is left unresolved rather than raising, so a cycle reaching the engine degrades to a non-matching condition instead of breaking evaluation for the whole context. Cycles are expected to be rejected where dependencies are written. Behaviour is covered by test cases in Flagsmith/engine-test-data#59 rather than by unit tests here, so that every engine is held to it.

…wildcards Three further cases, each covering a path the existing ones leave untested in a reference implementation: - a dependency segment carrying metadata, which has to survive resolution and be reported as any segment's metadata is - one segment carrying overrides for two features, so that resolving one feature selects the override naming it rather than the segment's first - a wildcard query over the flags mapping, which is unsupported because conditions resolve to a scalar. Every flag in that case is disabled, so the condition fails whichever node the query happens to select, and the expectation holds whether or not an implementation treats such a query as a dependency. What it resolves to when flags differ is deliberately left unpinned.

REVERT BEFORE MERGE. Dependent flags behaviour is covered by test cases in Flagsmith/engine-test-data#59 rather than by unit tests here, so that every engine is held to it. Those cases aren't in a release yet, so without this the new code is exercised by nothing and CI's 100% coverage gate fails at 78%. Once #59 is merged and tagged, this goes back to a semver tag, which Renovate now tracks as of #337.

One segment gating two features on the same flag, with two further segments each depending on one of those features. A diamond rather than a chain, so the shared segment is reached twice while resolving, once per feature depending on it. Guards two things a chain doesn't reach: reusing a segment's verdict for the second feature, and selecting the override naming the feature being resolved rather than the segment's first override.

Adds support for segments conditioned on another flag's result, via a `$.flags.<feature name>` condition property, as the evaluation half of dependent flags. Segment conditions are already JSONPath, so a dependency needs no new operator or condition type. What it does need is for the flag to be resolved by the time the condition reads it, and evaluation resolved every segment before any flag. Rather than establish up front which segments depend on which flags, `$.flags` is a mapping that resolves a flag when a condition first reads it. Resolving a flag evaluates the segments overriding it, which in turn resolves whatever flags their conditions read, memoised throughout. The existing single pass is otherwise untouched: a context whose segments read no flag never enters the resolver, and pays only for one empty mapping. Resolving on read rather than in advance means there is no need to recognise a dependency in a condition property, and so no need to reimplement enough of the JSONPath grammar to tell that `$.flags.a['enabled']` and `$.flags.a.enabled` are the same query. Every spelling works because resolution is triggered by the read itself. It also costs only what is read: rule short-circuiting means a condition that is never evaluated resolves nothing. Measured against main across the benchmark contexts, interleaved to cancel drift: +3.4%, from the mapping and a shallow copy of the context to hold it. Establishing up front that an environment has no dependencies would remove that residual entirely, but needs a field on the context to carry it. A flag caught in a dependency cycle is left unresolved rather than raising, so a cycle reaching the engine degrades to a non-matching condition instead of breaking evaluation for the whole context. Cycles are expected to be rejected where dependencies are written. Behaviour is covered by test cases in Flagsmith/engine-test-data#59 rather than by unit tests here, so that every engine is held to it.

REVERT BEFORE MERGE. Dependent flags behaviour is covered by test cases in Flagsmith/engine-test-data#59 rather than by unit tests here, so that every engine is held to it. Those cases aren't in a release yet, so without this the new code is exercised by nothing and CI's 100% coverage gate fails. Once #59 is merged and tagged, this goes back to a semver tag, which Renovate now tracks as of #337.

Adds support for segments conditioned on another flag's result, via a `$.flags.<feature name>` condition property, as the evaluation half of dependent flags. Segment conditions are already JSONPath, so a dependency needs no new operator or condition type. What it does need is for the flag to be resolved by the time the condition reads it, and evaluation resolved every segment before any flag. Rather than establish up front which segments depend on which flags, `$.flags` is a mapping that resolves a flag when a condition first reads it. Resolving a flag evaluates the segments overriding it, which in turn resolves whatever flags their conditions read, memoised throughout. The existing single pass is otherwise untouched: a context whose segments read no flag never enters the resolver, and pays only for one empty mapping. Resolving on read rather than in advance means there is no need to recognise a dependency in a condition property, and so no need to reimplement enough of the JSONPath grammar to tell that `$.flags.a['enabled']` and `$.flags.a.enabled` are the same query. Every spelling works because resolution is triggered by the read itself. It also costs only what is read: rule short-circuiting means a condition that is never evaluated resolves nothing. Measured against main across the benchmark contexts, interleaved to cancel drift: +3.4%, from the mapping and a shallow copy of the context to hold it. Establishing up front that an environment has no dependencies would remove that residual entirely, but needs a field on the context to carry it. A flag caught in a dependency cycle is left unresolved rather than raising, so a cycle reaching the engine degrades to a non-matching condition instead of breaking evaluation for the whole context. Cycles are expected to be rejected where dependencies are written. Behaviour is covered by test cases in Flagsmith/engine-test-data#59 rather than by unit tests here, so that every engine is held to it.

REVERT BEFORE MERGE. Dependent flags behaviour is covered by test cases in Flagsmith/engine-test-data#59 rather than by unit tests here, so that every engine is held to it. Those cases aren't in a release yet, so without this the new code is exercised by nothing and CI's 100% coverage gate fails. Once #59 is merged and tagged, this goes back to a semver tag, which Renovate now tracks as of #337.

Adds support for segments conditioned on another flag's result, via a `$.flags.<feature name>` condition property, as the evaluation half of dependent flags. Segment conditions are already JSONPath, so a dependency needs no new operator or condition type. What it does need is for the flag to be resolved by the time the condition reads it, and evaluation resolved every segment before any flag. Rather than establish up front which segments depend on which flags, `$.flags` is a mapping that resolves a flag when a condition first reads it. Resolving a flag evaluates the segments overriding it, which in turn resolves whatever flags their conditions read, memoised throughout. The existing single pass is otherwise untouched: a context whose segments read no flag never enters the resolver, and pays only for one empty mapping. Resolving on read rather than in advance means there is no need to recognise a dependency in a condition property, and so no need to reimplement enough of the JSONPath grammar to tell that `$.flags.a['enabled']` and `$.flags.a.enabled` are the same query. Every spelling works because resolution is triggered by the read itself. It also costs only what is read: rule short-circuiting means a condition that is never evaluated resolves nothing. Measured against main across the benchmark contexts, interleaved to cancel drift: +3.4%, from the mapping and a shallow copy of the context to hold it. Establishing up front that an environment has no dependencies would remove that residual entirely, but needs a field on the context to carry it. A flag caught in a dependency cycle is left unresolved rather than raising, so a cycle reaching the engine degrades to a non-matching condition instead of breaking evaluation for the whole context. Cycles are expected to be rejected where dependencies are written. Behaviour is covered by test cases in Flagsmith/engine-test-data#59 rather than by unit tests here, so that every engine is held to it.

REVERT BEFORE MERGE. Dependent flags behaviour is covered by test cases in Flagsmith/engine-test-data#59 rather than by unit tests here, so that every engine is held to it. Those cases aren't in a release yet, so without this the new code is exercised by nothing and CI's 100% coverage gate fails. Once #59 is merged and tagged, this goes back to a semver tag, which Renovate now tracks as of #337.

A flag whose dependencies form a cycle serves its environment default, which was previously indistinguishable from a flag that was never gated at all. Report `ERROR; code=CIRCULAR_DEPENDENCY` instead, so the condition that could not be evaluated is visible to whoever is looking at the result. Only flags in the cycle are reported this way. A flag merely depending on one resolves normally against whatever the cycle settled on.

Adds support for segments conditioned on another flag's result, via a `$.flags.<feature name>` condition property, as the evaluation half of dependent flags. Segment conditions are already JSONPath, so a dependency needs no new operator or condition type. What it does need is for the flag to be resolved by the time the condition reads it, and evaluation resolved every segment before any flag. Rather than establish up front which segments depend on which flags, `$.flags` is a mapping that resolves a flag when a condition first reads it. Resolving a flag evaluates the segments overriding it, which in turn resolves whatever flags their conditions read, memoised throughout. The existing single pass is otherwise untouched: a context whose segments read no flag never enters the resolver, and pays only for one empty mapping. Resolving on read rather than in advance means there is no need to recognise a dependency in a condition property, and so no need to reimplement enough of the JSONPath grammar to tell that `$.flags.a['enabled']` and `$.flags.a.enabled` are the same query. Every spelling works because resolution is triggered by the read itself. It also costs only what is read: rule short-circuiting means a condition that is never evaluated resolves nothing. Measured against main across the benchmark contexts, interleaved to cancel drift: +3.4%, from the mapping and a shallow copy of the context to hold it. Establishing up front that an environment has no dependencies would remove that residual entirely, but needs a field on the context to carry it. A flag caught in a dependency cycle is left unresolved rather than raising, so a cycle reaching the engine degrades to a non-matching condition instead of breaking evaluation for the whole context. Such a flag serves its environment default and reports `ERROR; code=CIRCULAR_DEPENDENCY`, so that a flag which could not be resolved is distinguishable from one that was never gated. Only flags in the cycle are reported that way; a flag merely depending on one resolves normally. Cycles are still expected to be rejected where dependencies are written. Behaviour is covered by test cases in Flagsmith/engine-test-data#59 rather than by unit tests here, so that every engine is held to it.

REVERT BEFORE MERGE. Dependent flags behaviour is covered by test cases in Flagsmith/engine-test-data#59 rather than by unit tests here, so that every engine is held to it. Those cases aren't in a release yet, so without this the new code is exercised by nothing and CI's 100% coverage gate fails. Once #59 is merged and tagged, this goes back to a semver tag, which Renovate now tracks as of #337.

The existing cycle case has both flags disabled by default, so cutting the cycle leaves nothing for another condition to match on and any implementation passes it. Here one flag is enabled by its environment default. Cutting the cycle leaves that value in place, and an implementation that publishes it for other conditions to read will match the segment gated on it — reporting segment membership for a segment whose override was never applied. A flag resolved only by cutting a cycle has to be unreadable by another condition. Verified to fail against an implementation without that behaviour, and to be independent of the order features and segments are declared in.

Adds support for segments conditioned on another flag's result, via a `$.flags.<feature name>` condition property, as the evaluation half of dependent flags. Segment conditions are already JSONPath, so a dependency needs no new operator or condition type. What it does need is for the flag to be resolved by the time the condition reads it, and evaluation resolved every segment before any flag. Rather than establish up front which segments depend on which flags, `$.flags` is a mapping that resolves a flag when a condition first reads it. Resolving a flag evaluates the segments overriding it, which in turn resolves whatever flags their conditions read, memoised throughout. The existing single pass is otherwise untouched: a context whose segments read no flag never enters the resolver, and pays only for one empty mapping. Resolving on read rather than in advance means there is no need to recognise a dependency in a condition property, and so no need to reimplement enough of the JSONPath grammar to tell that `$.flags.a['enabled']` and `$.flags.a.enabled` are the same query. Every spelling works because resolution is triggered by the read itself. It also costs only what is read: rule short-circuiting means a condition that is never evaluated resolves nothing. Measured against main across the benchmark contexts, interleaved to cancel drift: +3.7%, from the mapping and a shallow copy of the context to hold it. Establishing up front that an environment has no dependencies would remove that residual entirely, but needs a field on the context to carry it. A flag whose dependencies form a cycle is not resolvable. It serves its environment default and reports `ERROR; code=CIRCULAR_DEPENDENCY`, so that a flag which could not be resolved is distinguishable from one that was never gated, and only flags in the cycle are reported that way. Crucially the value it falls back to is never published to `$.flags`, so no other condition can match on a value that exists only because the cycle was cut — otherwise a segment gated on a flag that defaults to enabled would be reported as matched while the flag it overrides stayed at its default. Cycles are still expected to be rejected where dependencies are written. Behaviour is covered by test cases in Flagsmith/engine-test-data#59 rather than by unit tests here, so that every engine is held to it.

REVERT BEFORE MERGE. Dependent flags behaviour is covered by test cases in Flagsmith/engine-test-data#59 rather than by unit tests here, so that every engine is held to it. Those cases aren't in a release yet, so without this the new code is exercised by nothing and CI's 100% coverage gate fails. Once #59 is merged and tagged, this goes back to a semver tag, which Renovate now tracks as of #337.

Reverts the temporary pointers added while the schema was in review. Flagsmith/flagsmith#8396 is merged, so `EvaluationContext.flags` is on `main` and the test cases no longer need to reference an in-flight branch to validate. `schema.json` goes back to the context schema on `refs/heads/main`, and the flag dependency cases to the `refs/tags/v2.0.0` schema every other case in the corpus uses.

emyller

Looks great! Just a few notes.

The case explained its own mechanism but never named the case it exists alongside, so it reads as a duplicate of `cyclic__should_not_override`. Both flags there are disabled by default, so cutting the cycle leaves nothing truthy to match on and any implementation passes it; that is the distinction worth stating.

`shared_segment__resolved_once_per_feature` also carries a segment with overrides for two features, so it already exercises selecting the override that names the feature being resolved, and additionally covers reusing that segment's verdict for the second feature.

Reporting a segment's metadata does not depend on how the segment was matched, and `segment__with_metadata__match__metadata_in_result` already covers it. The case was added when resolution reported matched segments through a separate pass, where metadata could be dropped on that path alone. That pass no longer exists — segments are reported by the same code whether or not a dependency was involved — so a dependency-specific metadata bug is no longer expressible.

Follows Flagsmith/engine-test-data#59, where two cases were dropped as redundant and one gained a clarifying note. Coverage is unaffected.

`cyclic_enabled_default` was ambiguous — it reads as though the cycle is enabled by default, and collides with the `DEFAULT` reason — and `should_not_match` did not say what must not match, which is the whole distinction from its sibling. Named for the two things that matter: the prerequisite in the cycle is enabled, and the segment gated on it must not match. Alongside the sibling the pair now reads as a contrast rather than a near-duplicate: cyclic__should_not_override the flag is not overridden cyclic_enabled_prerequisite__segment_should_not_match the segment is not matched

`IS_NOT_SET` is the one operator that matches on a flag which could not be resolved — every other operator fails against no value. So a segment overriding the very flag it tests for absence has its condition made true by the cycle, and the override applies because the cycle was cut. A segment whose evaluation cut a cycle read a flag that has no value, so neither its match nor its failure to match is sound, and it must not be applied either way.

`IS_NOT_SET` matches when a property has no value, and a flag left unresolved by a cycle has none. A segment overriding the very flag it tests for absence therefore had its condition made true by the cycle, and its override applied — reported with the circular dependency reason but carrying the override's value rather than the environment default the reason claims. A segment whose own evaluation cut a cycle read a flag that could not be resolved, so neither its match nor its failure to match rests on anything, and it must not be applied either way. The cycle counter moves onto the flags mapping so that both the resolver and the single pass over segments can see it, and the mapping is now passed to `evaluate_segments` explicitly rather than read back out of the context. Covered by `cyclic_is_not_set__segment_should_not_match` in Flagsmith/engine-test-data#59, which fails without this.

emyller

LGTM!

Discharges the REVERT BEFORE MERGE on 99dea3e. Flagsmith/engine-test-data#59 is merged and released, so the submodule goes back to a semver tag — the form Renovate tracks as of #337 — rather than a branch.

khvn26 added 3 commits August 27, 2026 17:13

khvn26 mentioned this pull request Aug 27, 2026

feat: Evaluate flags that depend on other flags Flagsmith/flagsmith-engine#343

Merged

This was referenced Aug 28, 2026

Support $.flags context values in segment conditions Flagsmith/flagsmith-rust-flag-engine#40

Open

Support $.flags context values in segment conditions Flagsmith/flagsmith-go-client#226

Open

khvn26 marked this pull request as ready for review August 31, 2026 12:05

khvn26 requested a review from a team as a code owner August 31, 2026 12:05

khvn26 requested review from emyller and removed request for a team August 31, 2026 12:05

flagsmith-engineering Bot assigned emyller Aug 31, 2026

emyller reviewed Aug 31, 2026

View reviewed changes

khvn26 added 3 commits September 1, 2026 12:09

khvn26 requested a review from emyller September 1, 2026 11:27

emyller approved these changes Sep 1, 2026

View reviewed changes

khvn26 merged commit 28363b3 into main Sep 1, 2026
1 check passed

emyller deleted the test/flag-dependencies branch September 1, 2026 17:05

Navigation Menu

Uh oh!

feat: Add flag dependency test cases - #59

feat: Add flag dependency test cases#59
khvn26 merged 13 commits into
mainfrom
test/flag-dependencies

khvn26 commented Aug 27, 2026 •

edited

Loading

Uh oh!

emyller left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emyller left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

khvn26 commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How did you test this code?

Uh oh!

emyller left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emyller left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

khvn26 commented Aug 27, 2026 •

edited

Loading