Identify lifted GroupBy groupings by a database-computed rank

[benedict-odonovan]

Fixes #39241

• I've read the guidelines for contributing and seen the walkthrough
• I've posted a comment on an issue with a detailed description of how I am planning to contribute and got approval from a member of the team
• The code builds and tests pass locally (also verified by our automated build checks)
• Commit messages follow this format
• Tests for the changes have been added (for bug fixes / features)
• Code follows the same patterns and style as existing code in this repo

Summary

Since #38963, a projection that only enumerates a grouping's elements drops the GROUP BY. The rows stream back ordered by the grouping key, and GroupBySingleQueryingEnumerable assembles the groupings on the client, starting a new one whenever the key changes. That decision compared each row's key with the previous row's using the key column's ValueComparer, which is ordinal for an ordinary string column.

The database orders by its collation, and the two can disagree. Under SQL Server's default case-insensitive collation, "Engineering" and "engineering" sort next to each other as one group, but the client compares them ordinally and splits them:

Query	Groupings
GroupBy(e => e.DepartmentName).Select(g => g.Count()) (server-side GROUP BY)	4, 1
GroupBy(e => e.DepartmentName).Select(g => g.Select(e => e.Name).ToList()), before	3, 1, 1
The same, after	4, 1

The fix lets the database decide where each group ends. Lifted projections now project DENSE_RANK() OVER (ORDER BY <grouping terms>), and the client starts a new grouping when that rank changes, compared as an integer, instead of comparing the key columns:

-- before
SELECT [e].[DepartmentName], [e].[Name]
FROM [Employees] AS [e]
ORDER BY [e].[DepartmentName]

-- after
SELECT [e].[DepartmentName], DENSE_RANK() OVER(ORDER BY [e].[DepartmentName]), [e].[Name]
FROM [Employees] AS [e]
ORDER BY [e].[DepartmentName]

Rows the database considers equal tie and share a rank, under the database's own comparison rules. That covers case-insensitive and accent-insensitive collations, SQL Server's trailing-space rule ('abc' = 'abc ') and value converters, for any key type, including composite keys. g.Key still comes from the first row of each grouping, the same arbitrary choice a GROUP BY makes between Engineering and engineering.

The query already sorts by the grouping terms, so the rank can be computed on that same sorted stream. The other cost is one extra column per row.

Why DENSE_RANK()

• The key's KeyComparer can't be made to match the database. SQL Server gives string key columns a CaseInsensitiveValueComparer (Use case-insensitive string comparisons by default on SQL Server #29955), but that assumes a case-insensitive collation and only covers case. Collations are never interpreted on the client anywhere in EF, and plain columns like the one above use ordinal comparison.
• A canonical key (MIN(key) OVER (PARTITION BY key)) stops the client splitting a database group, but not merging two: a case-insensitive comparer over a case-sensitive column collapses them. MIN also isn't available for every type, for example SQL Server bit.
• A "new group" flag from LAG(key) OVER (ORDER BY key) relies on the outer ORDER BY returning tied rows in the same order as the window's, which SQL doesn't guarantee.
• Declining the lift for string keys is correct, but it rules out most real queries.

Implementation

DenseRankExpression is a new SQL expression in Microsoft.EntityFrameworkCore.Query.Internal. It's deliberately internal, like the providers' own SqlServerAggregateFunctionExpression and GlobExpression, so this fix adds no public API. QuerySqlGenerator and SqlNullabilityProcessor handle it alongside RowNumberExpression; it's never nullable. It supports quoting for precompiled queries.

TryTranslateGroupingElementProjection passes the rank to the SelectExpression through an internal ApplyGroupIdentifier. ApplyProjection then uses it, instead of the grouping columns' comparers, to build the key identifier for the final GroupBy. The identifier has the same lifetime as _preGroupByIdentifier: it isn't cloned, and a pushdown drops it. Only lifted projections set it, so a plain final GroupBy(...).ToList() is unchanged.

Regenerated baselines

Every test that exercises the lift gains the rank column. Results are unchanged in every case; only the SQL changes:

Test	Change
NorthwindGroupByQuery: GroupBy_select_grouping_list/_array, GroupBy_selecting_grouping_key_list, GroupBy_selecting_grouping_element_list, GroupBy_selecting_grouping_element_array_with_key, GroupBy_with_element_selector_selecting_grouping_element_list, GroupBy_with_result_selector_selecting_grouping_element_list, and the three GroupBy_nullable_value_type_key_* tests	DENSE_RANK() OVER(ORDER BY <key>) added
Collection_projection_over_GroupBy_over_parameter (ComplexNavigationsCollections, …SharedType, …SplitQuery, and the two temporal variants)	the same
Final_GroupBy_projecting_grouping_elements (precompiled)	the same
GroupBy_nullable_foreign_key_in_anonymous_key (SQL Server and SQLite)	the same, ranking by every term of a composite key, e.g. DENSE_RANK() OVER(ORDER BY [c].[CountryId], [c].[Id])

Testing

GroupBy_elements_with_keys_differing_only_in_case in AdHocMiscellaneousQueryRelationalTestBase uses the data from the issue: three employees in "Engineering", one in "engineering", one in "Sales". It asserts that the lifted groupings match the groups of a server-side GROUP BY, which makes it independent of the provider's collation: SQL Server's default is case-insensitive and SQLite's is case-sensitive. On SQL Server it fails without the fix (Expected: [1, 4], Actual: [1, 1, 3]), in both its sync and async variants.

• Full SQL Server functional suite: 49,801 passed / 0 failed / 501 skipped
• Full SQLite functional suite: 38,424 passed / 0 failed / 286 skipped
• EFCore.ApiBaseline.Tests green, with no baseline change since DenseRankExpression is internal

[Copilot]

🔵 Needs a closer look

The provider-wide grouping and SQL-generation changes warrant final human validation despite the focused regression coverage.

0 open findings

What changed in this PR

Fixes lifted relational GroupBy projections by using a database-computed dense rank to preserve database grouping semantics.

Changes:

• Adds DenseRankExpression and SQL/nullability handling.
• Uses the rank as the client grouping identifier.
• Adds regression coverage and updates SQL baselines.

File	Description
src/​EFCore.Relational/​Query/​Internal/​DenseRankExpression.cs	Defines the internal rank expression.
src/​EFCore.Relational/​Query/​QuerySqlGenerator.cs	Generates DENSE_RANK SQL.
src/​EFCore.Relational/​Query/​SqlNullabilityProcessor.cs	Processes rank ordering expressions.
src/​EFCore.Relational/​Query/​SqlExpressions/​SelectExpression.cs	Uses the rank as the grouping identifier.
src/​EFCore.Relational/​Query/​RelationalQueryableMethodTranslatingExpressionVisitor.cs	Creates ranks for lifted groupings.
test/​EFCore.Relational.Specification.Tests/​Query/​AdHocMiscellaneousQueryRelationalTestBase.cs	Adds the collation regression test.
test/​EFCore.SqlServer.FunctionalTests/​Query/​AdHocMiscellaneousQuerySqlServerTest.cs	Verifies SQL Server SQL.
test/​EFCore.Sqlite.FunctionalTests/​Query/​AdHocMiscellaneousQuerySqliteTest.cs	Verifies SQLite SQL.
test/​EFCore.SqlServer.FunctionalTests/​Query/​NorthwindGroupByQuerySqlServerTest.cs	Updates GroupBy SQL baselines.
test/​EFCore.SqlServer.FunctionalTests/​Query/​PrecompiledQuerySqlServerTest.cs	Updates the precompiled-query baseline.
test/​EFCore.SqlServer.FunctionalTests/​Query/​ComplexNavigationsCollectionsQuerySqlServerTest.cs	Updates navigation-query SQL.
test/​EFCore.SqlServer.FunctionalTests/​Query/​ComplexNavigationsCollectionsSharedTypeQuerySqlServerTest.cs	Updates shared-type SQL.
test/​EFCore.SqlServer.FunctionalTests/​Query/​ComplexNavigationsCollectionsSplitQuerySqlServerTest.cs	Updates split-query SQL.
test/​EFCore.SqlServer.FunctionalTests/​Query/​TemporalComplexNavigationsCollectionsQuerySqlServerTest.cs	Updates temporal-query SQL.
test/​EFCore.SqlServer.FunctionalTests/​Query/​TemporalComplexNavigationsCollectionsSharedTypeQuerySqlServerTest.cs	Updates temporal shared-type SQL.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[benedict-odonovan]

As discussed in the description, there is always the alternative of trying to map a provider's collation and settings to a custom ValueComparer but it seems like a rabbit-hole.

Adding the dense_rank() to the generated SQL is ugly but performance-wise it's only a small hit, and its definitely better than reverting to the previous unnecessary GROUP BY.

[AndriySvyryd]

Please rebase on latest main

[Copilot]

🟡 Changes recommended

Rank ordering can conflict with existing group ordering, introducing an avoidable additional sort.

1 open finding

• Align rank ordering with effective grouping order · New

🧠 Review effort: Balanced

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[AndriySvyryd]

Thanks for your contribution!