BIDC-8: self-join — keep both aliases of one index (lossless From.aliasesToTable), reject one alias for two sources

[fupelaqu]

Story BIDC-8 — self-join: From.tableAliases keyed by table name collapses the legs (core half)

Refs SOFTNETWORK-APP/softclient4es-arrow#144 — closed by the arrow PR (softclient4es-arrow branch feature/BIDC-8, held until core's 0.23.0-SNAPSHOT is merged and re-published); this body deliberately carries no closing keyword, per the epic's issue-closure bookkeeping. Core half of the fix; the arrow half (planner leg attribution, the arrow#137 case-5 named rejection, the integration suites) rides train B after 0.23.0 publishes.

What was wrong (AC 1 — reproduced, not assumed)

From.tableAliases is ListMap[tableKey -> alias], so it can hold ONE alias per table by construction. For FROM idx a JOIN idx b both legs have the identical qualified reference, 21.2's aliasKey returns the bare idx for both, and the map keeps idx -> b. aliasesToTable was .swap of it, so alias a was gone from BOTH maps. Probe at the baseline b37ef940 (_bmad-output/implementation-artifacts/BIDC-8-reproduction-core.txt):

SELECT a.id, b.amount FROM idx a JOIN idx b ON a.id = b.id
  tableAliases   = ListMap(idx -> b)
  aliasesToTable = ListMap(b -> idx)
  a.id     -> name=a.id   table=None      tableAlias=None      <- never resolved
  b.amount -> name=amount table=Some(idx) tableAlias=Some(b)
  joinKeyMatches = Some(List())                                <- the ON clause lost its join key

JoinKey.apply needs Identifier.table, so the ON clause carried no key, the arrow JoinSpec.condition was empty and DuckDB received FROM "sq_a" INNER JOIN "sq_b" with no ON — the issue's "Parser Error: syntax error at end of input" (end-to-end capture on real ES 8.18 in BIDC-8-reproduction-arrow.txt). The issue's own hypothesis (registration keyed on the index) is refuted: sq_<alias> is alias-keyed.

The fix (AD-1, on top of 21.2's AD-6 — extends it, does not re-key it)

• From.aliasesToTable is built directly and LOSSLESS (alias -> table key, same aliasKey on the table side). It equals the old .swap wherever no two aliases share a key and keeps BOTH aliases of a self-join. tableAliases keeps its documented one-alias-per-table contract untouched — softclient4es-extensions reads it FORWARD (JoinDependencyGraph:520) and its FieldAnalyzer reverse-scans it; neither sees a byte of difference.
• The two in-core reverse-lookup consumers now read aliasesToTable: Identifier.update (sql/package.scala) and FieldSort.update (SQL parser: ORDER BY on a bare table alias produces an empty column name (dangling qualifier) #159's bareTableAlias). Identifier.table stays a tableAliases KEY, so schemas, joinSourceKeys and TemporalLiterals keep their key language.
• TemporalLiterals: the cross-index guard's set is joinSourceKeys - mainTableKey — on a self-join the join source IS the main index, whose mapping IS in hand (new From.mainTableKey).
• Tripwire 2 (lead ruling: keep or reject LOUDLY, never change silently): the comma-FROM duplicate FROM t a, t b is rejected in From.validate() naming the JOIN spelling. Baseline behaviour was a.x unresolved + b.y resolved (a silent wrong answer); after the re-key both would have resolved against one doubled index (a different one). FROM t, t (same alias) unchanged; 21.2's FROM "a".orders o, "b".orders p unchanged.

Found by the review, fixed here (AD-7)

CreateTable and CreateMaterializedView had no validate() override — both inherited the no-op default, so every SingleSearch.validate() rule was skipped inside CREATE TABLE … AS SELECT and CREATE MATERIALIZED VIEW … AS SELECT (measured: CREATE TABLE t AS SELECT a.x, b.y FROM t a, t b was accepted while the bare SELECT was rejected; INSERT … SELECT already delegated). Both now delegate to the embedded query's validate(); the column form of CREATE TABLE is unchanged. Blast radius: 0 over the sql corpus (1054/1054) and over softclient4es-extensions' unit suites (see evidence). Pinned by the "REJECTED inside every statement kind that embeds a query" case.

Round 2 (independent review) — also in this branch

• One EXPLICIT alias written for two sources is rejected in From.validate(), case-insensitively (FROM orders o JOIN customers o, FROM orders A JOIN orders a, FROM t a, t a): these parsed, every o.x resolved against the LAST source and the arrow planner registered two legs as one sq_o (DuckDB: "closed pending query result"). An alias-less table's bare name is NOT an alias (round 3, NEW-4): FROM t, t, FROM "prod_us".orders, "prod_eu".orders, FROM "a".orders, orders keep their 21.2 multi-index-search acceptance, and the alias-less FROM t JOIN t is left to the join planner's own named rejection. CLAUDE.md's "licensing infrastructure #73" for this is stale — gh issue view 73 is a merged PR — so nothing is closed.
• The lossless map also changes resolution for an UNNEST whose nested field is named like its table (FROM orders o JOIN UNNEST(o.orders) AS i): o.id used to stay a literal field name and now resolves — pinned, release-noted.
• Watcher-flip pin asserts CREATE WATCHER … FROM a JOIN b ON … parses the JOIN and then silently discards it #191's exact message; AD-6 scaladoc corrected; remedy text no longer invents an alias for an alias-less duplicate.

Deferred, pre-existing (recorded in the findings log, not changed)

• searchInput (watcher) never runs From.validate(): CREATE WATCHER … FROM orders o, orders p WHERE x = 1 (UNQUALIFIED) is still accepted and flattened to orders — behaviour-preserving; the QUALIFIED form is rejected by CREATE WATCHER … FROM a JOIN b ON … parses the JOIN and then silently discards it #191's guard.
• A second .update() on an already-updated statement collapses fieldAlias across same-named legs; FROM "a".orders o, orders p accepted while sources = [orders, orders] (21.2 preserve-don't-interpret residual); JoinKey.key collides on a self-join (no consumer).

AD-9 — the Painless FILTER emission (lead ruling, option (a)): a WHERE with a function now runs at all

Measured on live ES 8.18.3 before the change: WHERE UPPER(status) <> 'ZZZ' emitted
(param1 == null) ? null : param1.toUpperCase().compareTo("ZZZ") != 0, whose ternary branches are
null (Object) and a primitive boolean — Elasticsearch rejected the whole query at COMPILE time
(script_exception: compile error, caused by class_cast_exception: Cannot cast from [boolean] to [java.lang.Object]), data-independent: it failed against an EMPTY index. The numeric family
failed identically (Cannot cast from [java.lang.Double] to [int]); the date family escaped only
because YEAR(...) folds into the parameter assignment and took the already-correct guarded branch.
Two further defects sat on the same path: a composed predicate was not parenthesised, and ?:
is the loosest Painless operator, so a guard swallowed its sibling; and check chose its hard-coded
spelling from the raw operator, so a NOT was silently dropped.

The fix: in a Query context a left-hand side that reads a document field is bound to a
prologue-declared local and compared INSIDE its guard — (<v> == null ? false : (<check>)); every
Predicate operand is parenthesised; check dispatches on the effective (NOT-folded) operator.
script_fields, sorts, aggregations and ingest processors keep null (21.8 Part C) — the
script_fields expectations in SQLQuerySpec are byte-identical, which is the evidence the change
stayed inside the filter context. ANSI three-valued logic is preserved because NOT is folded into
the operator before the guard and NOT (<composite>) is not expressible in this dialect (measured);
the ruling's Boolean-boxing sketch was therefore not needed — recorded as the rejected alternative
in the story spec.

Row-set pins (new, GatewayApiIntegrationSpec, real ES, id1 status='A' / id2 'B' / id3 field
ABSENT, seeded via COPY INTO because an INSERT omitting a column writes an empty string):
UPPER(status) = 'A' → [1]; NOT UPPER(status) = 'A' → [2]; UPPER(status) <> 'A' → [2];
… OR id = 1 → [1]; … AND id = 1 → [1]; NOT … AND id = 1 → []; NOT … OR id = 1 →
[1,2]; ABS(amount) > 10 → [2,3]; LOWER(status) = 'a' AND ABS(amount) > 10 → []. Two
un-scripted shapes are pinned beside them to make the two routes visible: status = 'A' OR id = 1 →
[1] (term queries) and NOT status = 'A' → [2,3] — ES must_not INCLUDES a document lacking the
field, a pre-existing divergence from ANSI this story does not change.

Fixture pins changed — three, each justified:

fixture (SQLQuerySpec, both bridge copies)	old filter bytes	new filter bytes	why correct
arithmetic function as script field and condition	(param1 == null) ? null : (param1 * (param2 - 10)) > 10000	def left1 = (param1 == null) ? null : (param1 * (param2 - 10)); (left1 == null ? false : ((left1 > 10000)))	old text does not compile on any ES (measured); new binds the value and compares inside the guard
mathematic function as script field and condition	(param1 == null) ? null : Double.valueOf(Math.sqrt(param1)) > 100.0	def left1 = …Math.sqrt(param1); (left1 == null ? false : ((left1 > 100.0)))	same defect, numeric family
string function as script field and condition	(param1 == null) ? null : param1.trim().length() > 10	def left1 = …param1.trim().length(); (left1 == null ? false : ((left1 > 10)))	same defect, string family

Two re-spacing rules (defleft → def left, false: → false : ) were added to the
whitespace-stripped expectation chains; they normalise the EXPECTED text only and weaken no
assertion. No script_fields expectation moved.

Gate integrity: the arrow branch's P-1 integration case, which round 6 had narrowed to avoid this
surface, is restored to the leg-local pushdown shape and passes on real ES.

Pins deliberately changed (AC 7 — 21.2 coordination; every one pre-authorised by LR-4)

pin	21.2 said	now
QuotedTableNameSpec "leave two IDENTICALLY qualified same-name tables collapsing, as before (AD-6 scope)"	tableAliases == ListMap(orders -> p) for the comma shape	tableAliases still orders -> p, aliasesToTable keeps o AND p (JOIN spelling); the comma spelling is now REJECTED
QuotedTableNameSpec watcher FROM orders o, orders p WHERE o.x = 1 "still defeats it, exactly as before" — accepted	accepted	REJECTED by #191's guard (a qualifier over a doubled single index is still a qualifier a multi-index search cannot scope)
ParserSpec "reject a self-correlation through duplicate table names"	rejected because only p.parent_id resolved	still rejected, now with both sides resolved — comment updated
Parser.scala qualifiedOverManyIndices scaladoc	"a WHOLLY unqualified self-join still defeats it"	updated

#159's two canary tests stay green untouched; the canary now also covers ORDER BY a / ORDER BY b on a self-join (SelfJoinAliasSpec).

Evidence

• sql/test 1058/1058 (baseline 1044 + 12 new: SelfJoinAliasSpec ×11, TemporalLiteralsSpec +1) · core/test 979/979 · + sql/compile · + core/compile · ++ 2.12.20 sql/Test/compile.
• BEFORE-form pins were run green at the baseline before the re-key (T4), then flipped with the before-state recorded in their comments.
• Downstream (arrow, held branch feature/BIDC-8): planner/executor unit suites and the JOIN integration suites on real ES — see the arrow PR body.

Release note (0.23.0)

• A self-join (FROM idx a JOIN idx b ON …) now resolves both legs; every column must be qualified (a bare name is ambiguous, as for any JOIN).
• Newly rejected: a table listed twice in a comma-separated FROM under different aliases (FROM t a, t b); one explicit alias for two sources, case-insensitively (FROM orders o JOIN customers o, FROM orders A JOIN orders a, FROM t a, t a); a watcher input FROM orders o, orders p WHERE o.x = 1 — all used to be accepted with a silently wrong reading.
• Resolution change: an UNNEST whose nested field is named like its table now resolves the table's own qualifier (o.id in FROM orders o JOIN UNNEST(o.orders) AS i was a literal field name before).
• Newly rejected: a CREATE TABLE … AS SELECT / CREATE MATERIALIZED VIEW … AS SELECT whose SELECT violates a SELECT-level rule (e.g. OFFSET is silently dropped under GROUP BY #295 OFFSET under GROUP BY, an inline HAVING aggregate, ORDER BY <bare table alias>) — the embedded query was never validated before (AD-7).
• No binary-incompatible change: From/SingleSearch gain lazy vals only; CreateTable/CreateMaterializedView gain an override.

---

Round 9 — the independent AD-9 audit, fixed forward

The audit executed the round-8 emission instead of reading it, and found one blocking regression
plus four shapes that had never worked. All are fixed here; the safety claim that covered them
(round 8's "C1") is recorded as FALSIFIED, with what replaces it.

What the audit falsified. C1 said the false collapse could not reach a projection because a
PainlessContext is only supplied on the filter path. A CASE over a function takes a context too,
and the condition's local was bound on a throwaway context while the prologue came from another, so
Elasticsearch answered HTTP 400 … cannot resolve symbol [left1] — a script referencing a name it
never declared. The containment half of the claim rested on "no expectation moved", which is absence
of evidence: four more shapes were broken and no unit test could see any of them.

What replaces it is a different KIND of claim — enumeration. PainlessNullSurvivalSpec names every
consumer of a criteria rendering in the repository (script query, predicate composition, CASE
condition, HAVING bucket_selector) and every one is a BOOLEAN position, which is why the collapse
is right in all of them. ⚠️ That is a deliberate deviation from the audit's prescription of a
context discriminator: measurement says such a flag would have exactly one legal value, because an
Expression IS a boolean and a projected VALUE goes through a different renderer that still emits
? null :. Pinned three ways (script_fields, script sort, terms script).

Red then green, measured on live Elasticsearch 8.18.3 over five documents (A, B, field
ABSENT, A with a second field absent, C):

item	before	after
CASE over a function	HTTP400 … cannot resolve symbol [left1]	1/0/0/1/0, and WHERE (CASE …) = 1 → [1, 4]
NOT after AND/OR	NOT UPPER(status)='A' AND amount>10 → [2] but the mirror order → [2,3]	both [2, 5]
the same NOT inside a CASE	the field-absent row took THEN	0/0/0/0/1 — ANSI
both operands guarded	null_pointer_exception on the shard	[1, 2, 5]
non-atomic operand	Cannot cast from [int] to [java.lang.Object]	[1, 4]
NOT LIKE over a function	scala.MatchError: LIKE escaping the query builder	[2, 5]
IN / BETWEEN over a function	{"terms":{"":…}} rejected as [bool] failed to parse field [filter]	[1, 2, 4] / all five
LIKE over a function	invalid sequence of tokens near ['"A%"']	[1, 4]

Two Painless claims measured and refuted before the shipped LIKE spelling: neither
String.matches(String) nor Pattern.compile is whitelisted (dynamic method … matches/1 not found, static method … compile/1 not found) — a regex may only enter through a /…/ literal. So
a %-only pattern decomposes into whitelisted String methods and only a pattern that truly needs
a regex uses the literal. That is also the only form that runs on ES 6.8, where script regexes
are disabled by default — verified green there.

Reported, not fixed (pre-existing, outside this story's surface): REGEXP_LIKE emits
Pattern.compile(...), which by the refutation above cannot run in any script context; and
ORDER BY UPPER(status) over a column some documents do not carry LOSES ROWS SILENTLY — a SORT
script, whose emission this story leaves correct (? null :, pinned). On a single-shard index the
search is rejected; on a multi-shard index it returns HTTP 200 with the failing shard's rows simply
absent (measured: 3 shards, 7 docs, _shards.failed: 1, hits.total: 5), because core has no
_shards.failures consumer.

New gates, each shown failing. PainlessOperandFormSpec fails on 14 shapes when
PainlessContext.bindLocal stops hoisting into the prologue (a statement sequence spliced into a
predicate or a CASE is a compile-time 400). PainlessNullSurvivalSpec's declared-vs-referenced
check fails when a local is bound without being declared — the exact B-1 root cause.

Regression. sql 1061, core 979, bridge 202 × es6/es7/es8/es9, + sql/compile, ++ 2.12.20 for
sql and core test sources, scalafmt, headerCheck. On real Elasticsearch the extended
GatewayApiIntegrationSpec ANSI test is green on 6.8.23, 7.17.29, 8.18.3 and 9.0.3, and the
full es8 JavaClientGatewayApiSpec is 75/75.

---

Round 10 — the final audit, fixed forward

One blocking regression this PR introduced, two measurably false documentation claims, and four
smaller defects. Every one measured on live Elasticsearch 8.18.3 before and after, over documents
A / B / field-absent / A.B / AXB1 / A-with-no-amount.

🔴 BLOCKING — NOT over a function-wrapped BETWEEN turned a loud 400 into a silent wrong
answer. BetweenExpr had no negated override, so the predicate's NOT fell back to wrapping
the un-negated criterion in a must_not, whose semantics INCLUDE a document lacking the field —
inverting the guard's false into a match. WHERE status = 'A' AND NOT ABS(amount) BETWEEN 1 AND 10
returned the row with no amount; the mirror order returned nothing. At the previous head that
statement was a loud HTTP 400, so this PR had made it worse. Fixed, and the new pins cover
BETWEEN, IN and IS NULL mirror-orders — the round-9 pair used only UPPER(status) = 'A', the
one criteria class that already had the override, which is exactly why the hole survived.

🔴 LIKE meant three different things at once. toRegex translated % and _ and nothing
else, so every other regex metacharacter kept its regex meaning on the paths that end in a regex.
Measured: status LIKE 'A.B%' (native regexp) matched A.B AND AXB1; UPPER(status) LIKE 'A.B%' (string methods) matched only A.B; UPPER(status) LIKE 'A.B%1' (Painless regex) matched
only AXB1. Fixed in the SHARED translation — fixing it only in the Painless emitter would have
produced a fourth reading.

Two documentation claims were false and are corrected in both twins. "The same predicate returns
the same rows whichever way round you write it" was falsified by the BLOCKING item (true again now).
"Patterns using only % work on every supported version" was simply wrong: 'A%B' is pure % and
still compiles to a regex, which stock ES 6.8 refuses. The docs now state the real rule — the
string-method fast path is taken when the pattern has no _ and % only at the ends.

Smaller: LIKE '' over a function emitted left1 ==~ //, and // opens a Painless comment
(unexpected character [//))))]) — the empty core now takes the equality path. CASE … THEN x END
with no ELSE emitted a truncated ternary (param2 ? 1), then, once completed, a
Cannot cast from [int] to [java.lang.Object] — SQL's implicit NULL is now emitted with the result
boxed, which is the spelling that compiles without a return (measured both ways).

🔴 One audit prescription was REFUTED and NOT applied, with the evidence. The audit asked for a
fourth consumer of Predicate.not — ElasticBridge's nested arm — to be aligned with the other
three by folding. It was tried: SQLQuerySpec's "predicate with distinct nested" fixture turned
must_not[nested] + filter into must[nested, nested(… == false)]. Under a nested relation the
criterion runs over a CHILD document inside an existential, so NOT outside and NOT inside are
different questions — "no reply is before that date" (a blog with no replies qualifies) versus "some
reply is not before it" (it does not). The fold applies where the negated criterion is evaluated
over the SAME document, and that site is now marked NOT-FOLD: with the reason. What the audit was
right about is that nothing ENUMERATED these consumers: all four are now listed on
Predicate.emittedRight, and a source scan fails if a new one states neither side.

Release notes for this version

• LIKE reads only % and _ as wildcards. Every other character in a pattern is now matched
  literally, on the scripted and the native path. WHERE status LIKE 'A.B%' previously matched
  AXB1; it now matches only values that really begin with A.B. Patterns relying on the old
  reading must be rewritten with _ or %. RLIKE is unaffected — its operand is a regular
  expression by definition.
• A CASE over a function inside SCRIPT AS emits a different script. The condition now
  collapses an absent field to false where it previously rendered null, so an ingest pipeline
  stored before this version differs from what the engine generates now. Re-run
  CREATE TABLE / ALTER … SET SCRIPT AS for tables built earlier if the pipeline must match.
  (Same class as the 21.8 Part C note.)
• ORDER BY over a function of a sparsely populated column LOSES ROWS SILENTLY on a multi-shard
  index — disclosed, not introduced here. The emitted sort script preserves null correctly;
  Elasticsearch fails the shard while building the comparator. Single-shard: the search is
  rejected. Multi-shard: HTTP 200 and the failing shard's documents are absent from the result,
  with no error reaching the caller (core does not inspect _shards.failures). Documented in both
  doc twins; surfacing shard failures is recorded as a separate issue.

Recorded, not fixed

• REGEXP_LIKE emits java.util.regex.Pattern.compile(...), which is not whitelisted in any script
  context — measured in round 9, untouched here, no test exercises it.
• A future improvement for scripted LIKE: UPPER(x) / LOWER(x) over a plain column is really a
  case-insensitive match, and ES ≥ 7.10 supports case_insensitive: true on regexp / wildcard
  natively. It would not replace the decomposition — a genuinely computed operand
  (SUBSTRING(x,1,3) LIKE 'AB%') cannot be a native query — and it is unavailable on 6.8. Recorded
  at the decomposition site; no issue filed.

🤖 Generated with Claude Code