diff --git a/README.md b/README.md index a2a8b09..1dc3334 100644 --- a/README.md +++ b/README.md @@ -268,10 +268,10 @@ machine that compiled it. `evaluate`, `execute` and `executeValue` each take that source text directly as well, in place of the instruction list, and compile it before running it. The string is compiled as an EXPRESSION at all three: a source that needs the -statement grammar is refused identically at every one of them, and compiling a -statement program from source text is not yet implemented anywhere here. A -caller with a statement program to run still compiles it elsewhere and passes -the instruction list. +statement grammar is refused identically at every one of them. A caller with +a statement program to run compiles it with `compileProgram`, under +[Statement programs](#statement-programs) below, and passes the instruction +list. ```ts import { evaluate, execute } from "@riddler/predicator"; @@ -569,6 +569,51 @@ if (!run.ok || run.value !== "treatment" || run.context.variant !== "treatment") } ``` +`compileProgram` compiles that list from source text. A program is one or +more statements separated by `;`: an assignment to a name, a property or an +index, an `if` with an optional `else` or `else if`, a `while`, or a bare +expression, whose value is what `executeValue` answers when it is the last +one to run. A statement that ends in `}` needs no `;` after it, and a block +opens no scope of its own. + +```ts +import { compileProgram, compileProgramWithSpans, executeValue } from "@riddler/predicator"; + +// A signup wizard's step script, as an author writes it in the editor. +const script = "if visitor.bucket < 50 { variant = 'treatment' } else { variant = 'control' }; variant"; + +const compiled = compileProgram(script); + +if (!compiled.ok) { + throw new Error("a well-formed script compiles"); +} + +const run = executeValue(compiled.instructions, { visitor: { bucket: 12 } }); + +if (!run.ok || run.value !== "treatment" || run.context.variant !== "treatment") { + throw new Error("the compiled script runs as the hand-written list above does"); +} + +// A refusal is a value here as it is at `compile`, and the statement grammar +// brings reasons of its own: a left side that is not a location, a block +// with no opening brace, and an `else` with no `if` before it. +const misplaced = compileProgram("plan.trial + 1 = 14"); + +if (misplaced.ok || misplaced.error.reason !== "unassignable_location") { + throw new Error("only a name, a property or an index can be assigned"); +} + +// `compileProgramWithSpans` carries the spans table `compileWithSpans` does, +// where the instruction ending each statement spans that whole statement, +// and a second table keyed by each `store`, with one span per segment of the +// location it writes. `compileProgramWithPositions` carries points instead. +const located = compileProgramWithSpans("plan.trial = 14"); + +if (!located.ok || located.segmentSpans.get(3)?.length !== 2) { + throw new Error("a store into plan.trial writes a location of two segments"); +} +``` + Three properties of a statement run are worth knowing before a host relies on one. diff --git a/changelog.d/pts-xmz6.md b/changelog.d/pts-xmz6.md new file mode 100644 index 0000000..a7201e9 --- /dev/null +++ b/changelog.d/pts-xmz6.md @@ -0,0 +1,4 @@ +### Added + +- `compileProgram`, `compileProgramWithPositions` and `compileProgramWithSpans` compile a statement program from source text - assignments, `;`-separated statements, `if`/`else`, `else if` and `while` - to the instruction list `execute` and `executeValue` run, answering the results `compile`, `compileWithPositions` and `compileWithSpans` answer, with one more table on the two located variants: `segmentPositions` or `segmentSpans`, one entry per segment of the location each `store` writes. +- `ParseReason` gains `unexpected_else`, `unassignable_location` and `expected_open_brace`, which only the three program entry points answer; a caller that switches exhaustively on the union adds the three cases. diff --git a/docs/adr/0004-the-compiler-surface.md b/docs/adr/0004-the-compiler-surface.md index 8d51a40..8767e6e 100644 --- a/docs/adr/0004-the-compiler-surface.md +++ b/docs/adr/0004-the-compiler-surface.md @@ -1341,3 +1341,276 @@ falsified are where it says, and `decompile` answered a bare string at `64a6e9d`. Run in a detached export of predicator-ex `v9.4.2` under Elixir 1.18.3 and OTP 27, the reference compiled a source of three hundred nested parentheses and rendered its tree, as the amendment says. + +## Amendment: the statement grammar compiles, through three program entry points, with the transcript's program rows as its evidence (2026-10-01) + +Status: proposed (2026-10-01) + +Recorded for `pts-xmz6`, under two rulings: that the grammar is the +reference's full statement grammar - assignment to every location shape, a +bare expression, the separator, `if`/`else` and `else if`, and `while` - and +that its evidence is a set of program rows in the compile transcript, taken +from the reference at the tag (both ruled by the operator, 2026-10-01). This +entry is appended and removes no line above. `src/` is cited as the change +carrying this entry leaves it; the transcript is cited as it stands on main at +`6149371`, unchanged by this entry. + +### What this amends + +The record's section headed "The expression grammar only" put the statement +grammar out of scope until it had evidence, and said why: "A statement +compiler written now would be written against no conformance evidence". The +evidence now exists. `conformance/transcript/compile.json` carries a +`program` row for each statement program the authored list in +`scripts/lib/program-sources.mjs` expects the reference to compile, with the +instruction list and both of the reference's side tables, and a +`program_refusal` row for each one it expects refused, with the message, +position and span; the rows were written by running +`Predicator.compile_program_with_spans/1` in a detached export of +predicator-ex at `v9.4.2`. So the sentence "The statement grammar - +assignment, the statement separator, `if`/`else` and the loop keyword - is out +of scope for this record and arrives in a later release with its own +evidence." is superseded by this entry, which is that release's record. What +the same section says of `compile` itself still holds: `compile` compiles the +expression grammar and nothing else, and refuses statement syntax with +`statement_keyword` and `assignment_in_expression` exactly as before. + +Four more statements above are superseded, each only as far as it says. + +- In "The closed reason union", the paragraph headed "There is no + unassignable-location reason." Through `compile` its run still holds - + `user.age = 30` answers `assignment_in_expression` at the `=` - but the + union now carries `unassignable_location`, answered by the program entry + points. +- In the same section, the sentence closing the paragraph on unreachable + families, "Its statement and block messages belong to the statement + grammar." The statement grammar is now in the surface, and its messages are + mapped to members below. +- In "Typespecs", the `ParseReason` union, which gains three members below. +- In "Consequences", the paragraph opening "Leaving the statement grammar out + means the package compiles a strict subset of what the reference compiles". + The program entry points now compile what the reference's + `Predicator.compile_program/1` compiles; the paragraph's account of why the + two keyword refusals are in the union still holds of `compile`. + +One sentence nearby is NOT superseded and is named so a reader does not go +looking. "The three entry points take a source string" says that "A +program-mode `execute(source)` that compiles the statement grammar arrives +with that grammar, not here." This entry brings the grammar and leaves +`evaluate`, `execute` and `executeValue` exactly as they were: each still +compiles a source string as an expression. Switching the two run entry points +to program mode is a later change with its own record. + +### The decision + +**Three entry points compile a statement program from source text: +`compileProgram`, `compileProgramWithPositions` and +`compileProgramWithSpans`.** Each is the program-shaped counterpart of an +expression entry point, as the reference's `compile_program/1`, +`compile_program_with_positions/1` and `compile_program_with_spans/1` are of +its own, and each answers the result union its counterpart answers, a refusal +being a `ParseError` value on the failing arm and never a throw. + +The grammar is the reference's `parse_program` at the tag, read in +`parseProgram` in `src/parser.ts`: + +- a program is one or more statements separated by `;`, with one trailing + `;` allowed, so an empty source, a lone `;` and a doubled `;` anywhere are + refusals; +- a statement is an `if`, a `while`, an assignment, or a bare expression; +- an assignment is a location, `=`, and an expression, where a location is an + identifier followed by any run of property and bracket accesses, a + parenthesis around any part of it changing nothing; the left side is + probed as an additive expression and refused at the `=` when it is not a + location; +- `if` takes an expression and a block, then optionally `else` and either a + block or another `if`, an `else if` being an else block that holds one + `if`; `while` takes an expression and a block; +- a block is `{`, an optional statement sequence, `}`, and opens no scope; +- a statement ending in `}` needs no `;` before the next statement. + +The emission is the reference's instructions visitor at the tag, read in +`visitStatement` and `visitAssignment` in `src/emitter.ts`. An assignment +pushes its location's segments root first - a name as a string literal, a +bracket's key as whatever its expression compiles to - then its value, then +`["store", n]` with `n` the number of segments. A bare expression ends in +`["pop"]`. `if c { A }` is `c`, `["pop_jump_if_falsy", len(A) + 1]`, `A`; +with an else block `B` it is `c`, `["pop_jump_if_falsy", len(A) + 2]`, `A`, +`["jump", len(B) + 1]`, `B`; `while c { A }` is `c`, +`["pop_jump_if_falsy", len(A) + 2]`, `A`, +`["jump_backward", len(c) + len(A) + 1]`. Every statement leaves the stack as +it found it, so statements are emitted one after another. + +The side tables follow the reference's annotations, both built in one walk as +the expression tables are: + +| instruction | position | span | +|---|---|---| +| a segment's string literal | the identifier, or the property name | the identifier, or the access from the location's root through the property | +| a bracket key's instructions | the key expression's own, as in any expression | the key expression's own | +| `store` | the location's root | the whole assignment | +| `pop` | the expression's own | the expression's own | +| the jumps of an `if` or a `while` | the keyword | the keyword through the last closing brace | + +Each `store` also carries one annotation per segment, root first: the root +identifier's, each property access node's, and each bracket key's. + +**Evidence.** Every `program` row is reproduced exactly by +`compileProgramWithSpans` - the instruction list compared as values, the +`positions` table and the `segment_positions` table entry by entry - and +every `program_refusal` row by `compileProgram`, the message verbatim with its +position and span; `test/reference-compile.test.ts` reads the rows through +`compileTranscriptLines` and holds each to that. The rows carry span tables +only, because the reference was run in its span mode to write them, so the +point tables were checked by running `Predicator.compile_program_with_positions/1` +in the same export on 2026-10-01 over every program the rows hold; every +table agreed, and `test/compile-program.test.ts` pins a selection of them, each +quoted as the run printed it. + +### The closed reason union + +The program grammar meets eight message families. Five are families the +union already names, and three are new. + +Of the five, three are reached by the expression production inside a +statement, with the expression grammar's own message: `expected_primary` (an +empty statement, a missing condition, a block that ends with the source), +`statement_keyword` (`status = if`) and `assignment_in_expression` (a second +`=` in `renewals = fines = 0`, or one in a condition). The other two are the +statement grammar's own wording of a family the union already names, and are +mapped to it by the rule the union already uses for `unterminated_string`, +whose single-quoted wording names the quote and is the same family: a message +that is one template up to the token and the construct it names is one +family. + +| `reason` | message, as run at the tag | +|---|---| +| `trailing_token` | `Unexpected token identifier 'fines' after statement` | +| `expected_close_brace` | `Expected '}' to close the block but found end of input` | + +The three members this entry adds name families the union lacked. Each is the +reference's message family, and each message is the reference's verbatim: + +| `reason` | message, as run at the tag | +|---|---| +| `unexpected_else` | `Unexpected 'else' - an 'else' block must follow an 'if' block.` | +| `unassignable_location` | `Left side of '=' must be an assignable location - an identifier, a property access, or a bracket access.` | +| `expected_open_brace` | `Expected '{' to open a block but found identifier 'status'` | + +Each message above is quoted from a `program_refusal` row: +`after-statement/missing-separator`, `unterminated-block/end-of-input`, +`stray-else/leading`, `not-a-location/literal` and `missing-block/if-token`, +under the prefix `program-refusal/`. A row carries no reason, because the +reason is this package's token and the reference has none; the mapping is +this entry's, and `test/compile-program.test.ts` pins one source per family to +its member. Only the three program entry points answer the three new members. + +### The depth bound reaches the statement grammar + +The source depth bound the amendment on nesting depth declares applies to a +program as it does to an expression, and with the same reason and message. A +block's statements are one level deeper than the block in the grammar +(`descend` in `src/parser.ts`), and an `if` or a `while` is one level deeper +than the sequence that holds it in the emission (`deeper` in +`src/emitter.ts`), so blocks nested inside one another are bounded as any +other nesting is. A statement sequence is read and emitted in a loop, so its +length costs no depth, as a chain's does not. An `else if` chain is written +flat and is nesting in the tree, an else block holding an `if`, and it counts +a level per link in both walks; that is the one flat construct of the program +grammar whose length is bounded: a chain is refused a few links short of two +hundred and fifty-six, since each link's own block and condition sit a level +below it. The message still says "Expression nests past the depth +limit"; it is this package's own message, and a program is held to it +unchanged rather than given a second message for the same family. + +### Typespecs + +The union in the Typespecs section above gains three members, appended: + +```typescript +export type ParseReason = + // the members above and those the amendments above add, unchanged, and: + | "unexpected_else" + | "unassignable_location" + | "expected_open_brace"; +``` + +Three functions join the main entry point: + +```typescript +export declare function compileProgram(source: string): CompileResult; + +export declare function compileProgramWithPositions(source: string): + | { + readonly ok: true; + readonly instructions: Program; + readonly positions: ReadonlyMap; + readonly segmentPositions: ReadonlyMap; + } + | { readonly ok: false; readonly error: ParseError }; + +export declare function compileProgramWithSpans(source: string): + | { + readonly ok: true; + readonly instructions: Program; + readonly spans: ReadonlyMap; + readonly segmentSpans: ReadonlyMap; + } + | { readonly ok: false; readonly error: ParseError }; +``` + +The two located results are the expression results with one more member on +the succeeding arm, so a value of each is a value of +`CompileWithPositionsResult` or `CompileWithSpansResult`, and a caller that +handles the expression entry point's answer handles this one unchanged. The +member is the reference's `segment_positions` table, keyed by the index of +each `store` and holding one annotation per segment of the location it +writes. It is a separate member rather than folded into `positions` for the +reason the reference gives for its own field: the positions table's meaning +does not change. It is named for the kind of annotation it holds, as the two +expression tables are, rather than carrying one name whose type depends on +the function that produced it - the choice "Two envelopes, neither +serialized" made above. A program that assigns nothing answers an empty one. +No name is exported for either result type, and `ParseError`, `Program`, +`Position` and `Span` are the types already exported. + +### What this does not decide + +It does not change what any existing function answers. `compile`, +`compileWithPositions`, `compileWithSpans`, `parse`, `evaluate`, `execute` and +`executeValue` answer exactly what they answered before, and every expression +row of the transcript is diffed as it was. + +It does not render a program back. `decompile` takes the expression tree +`parse` answers, and no entry point answers a program's tree. + +It does not move the conformance registry's compiler claim. The corpus still +carries no statement source - every case in `tier-6.json`, `tier-8.json` +and `tier-9.json` has a null `source` - so under ADR-0003's case-set rule the compiler surface's case +set is unchanged, and the transcript, not the corpus, is this grammar's +evidence. `conformance/registry.json` is unchanged. + +It does not decide anything about a location API, a way for a host to name a +location in a context outside a program. + +### Consequences + +A host can now compile the statement programs the evaluator already runs, +from the same source text the reference compiles, and a statement program it +compiled elsewhere and stored is no longer the only way to run one. + +The union is a compatibility surface and it gained three members, exactly as +the Consequences of the amendments above say of the members they added. A +caller that switches exhaustively on `ParseReason` is told by the typechecker, +and a caller that never calls a program entry point never receives one of +the three. + +The verbatim-message rule now ties three more messages, and the two +statement-grammar wordings of existing families, to the reference's text; a +message reworded in predicator-ex is a diff here at the next refresh of the +transcript, as it is for the expression grammar's. + +An `else if` chain is bounded where the reference's is not. That is the +divergence the amendment on nesting depth declares for nesting in general, +reached through the one flat construct of the program grammar the tree +nests; no program in the transcript comes near it. diff --git a/src/ast.ts b/src/ast.ts index 4132e33..417eba0 100644 --- a/src/ast.ts +++ b/src/ast.ts @@ -1,5 +1,6 @@ /** - * The syntax tree the expression grammar produces. + * The syntax tree the expression grammar produces, and the statement nodes the + * program grammar builds over it, at the end of this module. * * Every node carries two pieces of source metadata rather than one. The * reference implementation carries a single slot holding either a position or @@ -338,3 +339,74 @@ export type Node = | CastNode | DurationNode | RelativeDateNode; + +/** + * An assignment: a location, an `=`, and the expression whose value is + * written there. + * + * The target is an ordinary node of the expression grammar, restricted to the + * three location shapes - an identifier, optionally followed by any run of + * property and bracket accesses - and the grammar refuses any other left side + * before it builds this node. Its `position` is the `=` and its `span` runs + * from the location's start to the value's end, which is what the reference + * gives the same node. + */ +export interface AssignmentStatement extends Located { + readonly kind: "assignment"; + readonly target: Node; + readonly value: Node; +} + +/** + * A statement sequence inside braces. It opens no scope: a block is the + * program production terminated by a closing brace rather than by the end of + * the source. + * + * Its `position` is the opening brace and its `span` runs through the closing + * one. The block an `else if` stands for has no braces of its own, and borrows + * both from the `if` it holds. + */ +export interface Block extends Located { + readonly kind: "block"; + readonly statements: readonly Statement[]; +} + +/** + * `if` with an optional `else`. An `else if` is an `else` whose block holds + * one nested `if`, so a chain is a nesting rather than a list. + * + * Its `position` is the `if` keyword and its `span` runs through the closing + * brace of the last block present. + */ +export interface IfStatement extends Located { + readonly kind: "if"; + readonly condition: Node; + readonly consequent: Block; + readonly alternative: Block | null; +} + +/** `while` and its body, positioned and spanned the way an `if` is. */ +export interface WhileStatement extends Located { + readonly kind: "while"; + readonly condition: Node; + readonly body: Block; +} + +/** + * One statement of a program: an assignment, one of the two control-flow + * statements, or a bare expression, which is any node of the expression + * grammar standing alone. + */ +export type Statement = AssignmentStatement | IfStatement | WhileStatement | Node; + +/** + * A statement program: one or more statements, never none. + * + * Its `position` is its first token and its `span` runs from its first + * statement's start to its last statement's end, so a trailing separator is + * outside it. + */ +export interface ProgramNode extends Located { + readonly kind: "program"; + readonly statements: readonly Statement[]; +} diff --git a/src/compile.ts b/src/compile.ts index b1bf6b3..de970d1 100644 --- a/src/compile.ts +++ b/src/compile.ts @@ -2,14 +2,18 @@ * Source text becomes a program, through the scanner, the grammar and the * emitter in that order. * - * The three entry points here differ only in what they hand back beside the - * instruction list. There is one compilation underneath them: the emitter - * builds the positions table and the spans table in the same walk that builds - * the instructions, so `compileWithPositions` and `compileWithSpans` are two - * readings of one result rather than two compilations. That is why a caller - * wanting both tables pays for one walk here where the reference parses twice. + * There are two families of entry point, three in each. `compile` and its two + * siblings read an expression; `compileProgram` and its two siblings read a + * statement program, through the program grammar and the program walk of the + * same emitter. Within a family the three differ only in what they hand back + * beside the instruction list. There is one compilation underneath them: the + * emitter builds the positions table and the spans table in the same walk + * that builds the instructions, so the positions and the spans entry points + * are two readings of one result rather than two compilations. That is why a + * caller wanting both tables pays for one walk here where the reference + * parses twice. * - * The failing arm is the same at all three. Each stage answers the first + * The failing arm is the same at all six. Each stage answers the first * refusal it meets as a value, and this module passes that value straight out * without rewrapping it, so the `reason`, the `message`, the `position` and the * span a caller reads are the refusing stage's own. A source that fails to @@ -28,11 +32,11 @@ * it. */ -import { emit } from "./emitter.js"; +import { emit, emitProgram, type ProgramEmitResult } from "./emitter.js"; import type { ParseError, Position, Span } from "./errors.js"; import type { Program } from "./instructions.js"; import { tokenize } from "./lexer.js"; -import { parse } from "./parser.js"; +import { parse, parseProgram } from "./parser.js"; /** A compiled program, or the first refusal that stopped it. */ export type CompileResult = @@ -57,6 +61,39 @@ export type CompileWithSpansResult = } | { readonly ok: false; readonly error: ParseError }; +/** + * A compiled statement program with the position of each instruction's node + * beside it, and the position of each segment of the location every `store` + * writes. + * + * It is `CompileWithPositionsResult` with one more table on the succeeding + * arm, so a value of it is also a value of that type, and a caller that + * handles the expression entry point's answer handles this one unchanged. + */ +type CompileProgramWithPositionsResult = + | { + readonly ok: true; + readonly instructions: Program; + readonly positions: ReadonlyMap; + readonly segmentPositions: ReadonlyMap; + } + | { readonly ok: false; readonly error: ParseError }; + +/** + * A compiled statement program with the span of each instruction's node + * beside it, and the span of each segment of the location every `store` + * writes. It is `CompileWithSpansResult` with one more table, as the type + * above is `CompileWithPositionsResult` with one more. + */ +type CompileProgramWithSpansResult = + | { + readonly ok: true; + readonly instructions: Program; + readonly spans: ReadonlyMap; + readonly segmentSpans: ReadonlyMap; + } + | { readonly ok: false; readonly error: ParseError }; + /** * Everything one compilation produces, before an entry point picks from it. * @@ -135,3 +172,86 @@ export function compileWithSpans(source: string): CompileWithSpansResult { if (!compiled.ok) return { ok: false, error: compiled.error }; return { ok: true, instructions: compiled.instructions, spans: compiled.spans }; } + +/** Scans, parses a statement program and emits it, answering the first refusal. */ +function compileProgramAll(source: string): ProgramEmitResult { + const scanned = tokenize(source); + if (!scanned.ok) return { ok: false, error: scanned.error }; + + const parsed = parseProgram(scanned.tokens); + if (!parsed.ok) return { ok: false, error: parsed.error }; + + return emitProgram(parsed.program); +} + +/** + * Compiles a statement program into the instruction list `evaluate` + * consumes. + * + * A program is one or more statements separated by `;`, with one trailing + * `;` allowed. A statement is an assignment to a location - an identifier, + * a property access or a bracket access - an `if` with an optional `else` or + * `else if`, a `while`, or a bare expression, whose value is discarded. A + * statement that ends in `}` needs no separator before the next one, and a + * block opens no scope of its own. + * + * It answers the result `compile` answers. A refusal is a `ParseError` value + * on the failing arm, never a throw, under the same depth bound the + * expression compiler declares; the statement grammar adds its own reasons + * to the closed union for the refusals only it can meet. + */ +export function compileProgram(source: string): CompileResult { + const compiled = compileProgramAll(source); + if (!compiled.ok) return { ok: false, error: compiled.error }; + return { ok: true, instructions: compiled.instructions }; +} + +/** + * Compiles a statement program, with the position of each instruction beside + * it and the position of each location segment beside every `store`. + * + * The positions table is keyed and valued as `compileWithPositions`'s is. A + * `store` carries the position of the location's root, a `pop` the position + * of the expression whose value it discards, and the jumps of an `if` or a + * `while` the position of its keyword. `segmentPositions` is keyed by the + * index of each `store` and holds one position per segment of the location + * it writes, root first: the root identifier's, each property name's, and + * each bracket key's. A program that assigns nothing has an empty one. + * + * The instruction list is identical to `compileProgram`'s for the same + * source. + */ +export function compileProgramWithPositions(source: string): CompileProgramWithPositionsResult { + const compiled = compileProgramAll(source); + if (!compiled.ok) return { ok: false, error: compiled.error }; + return { + ok: true, + instructions: compiled.instructions, + positions: compiled.positions, + segmentPositions: compiled.segmentPositions, + }; +} + +/** + * Compiles a statement program, with the span of each instruction beside it + * and the span of each location segment beside every `store`. + * + * The tables are keyed the way `compileProgramWithPositions`'s are. The + * instruction that ends a statement - a `store` or a `pop` - carries that + * statement's own extent, so a failure inside a long program underlines the + * one statement rather than the whole program, and the jumps of an `if` or a + * `while` carry the whole statement through its last closing brace. The + * instruction list is identical to `compileProgram`'s, and the tables cannot + * disagree with `compileProgramWithPositions`'s about an index, because one + * walk builds all four. + */ +export function compileProgramWithSpans(source: string): CompileProgramWithSpansResult { + const compiled = compileProgramAll(source); + if (!compiled.ok) return { ok: false, error: compiled.error }; + return { + ok: true, + instructions: compiled.instructions, + spans: compiled.spans, + segmentSpans: compiled.segmentSpans, + }; +} diff --git a/src/emitter.ts b/src/emitter.ts index 90aecff..1ab54db 100644 --- a/src/emitter.ts +++ b/src/emitter.ts @@ -61,11 +61,16 @@ import type { ArithmeticOperator, + AssignmentStatement, + Block, ComparisonOperator, DurationNode, + Located, MembershipOperator, Node, + ProgramNode, RelativeDirection, + Statement, UnaryOperator, } from "./ast.js"; import { ParseError, type Position, type Span } from "./errors.js"; @@ -86,11 +91,34 @@ export type EmitResult = } | { readonly ok: false; readonly error: ParseError }; -/** One instruction with the position and the span of the node that emitted it. */ +/** + * A statement program with both side tables over it and the two segment + * tables beside them, or the one refusal that stopped it. + * + * A segment table is keyed by the index of a `store` and holds one entry per + * segment of the location that store writes, root first: the position, or + * the span, of the node that produced the segment's value. + */ +export type ProgramEmitResult = + | { + readonly ok: true; + readonly instructions: Program; + readonly positions: ReadonlyMap; + readonly spans: ReadonlyMap; + readonly segmentPositions: ReadonlyMap; + readonly segmentSpans: ReadonlyMap; + } + | { readonly ok: false; readonly error: ParseError }; + +/** + * One instruction with the position and the span of the node that emitted it, + * and, on a `store`, the annotation of each segment of the location it writes. + */ interface Annotated { readonly instruction: Instruction; readonly position: Position; readonly span: Span; + readonly segments?: readonly Located[]; } /** @@ -118,28 +146,79 @@ class EmitSignal extends Error { * there is one list and the tables are built beside it. */ export function emit(ast: Node): EmitResult { - let annotated: readonly Annotated[]; + const walked = walk(() => visit(ast)); + if (!walked.ok) return walked; + const { instructions, positions, spans } = tables(walked.annotated); + return { ok: true, instructions, positions, spans }; +} + +/** + * Compiles a statement program into a program, its two side tables and its + * two segment tables. + * + * Every statement leaves the stack as it found it: an assignment ends in the + * `store` that consumes its value, a bare expression in a `pop`, and the two + * control-flow statements consume their condition in the jump that tests it. + * So the statements are emitted one after another, and the instruction list + * has no statement boundary of its own. + */ +export function emitProgram(program: ProgramNode): ProgramEmitResult { + const walked = walk(() => visitStatements(program.statements)); + if (!walked.ok) return walked; + return { ok: true, ...tables(walked.annotated) }; +} + +/** Runs one walk from a depth of zero, answering its refusal as a value. */ +function walk( + run: () => readonly Annotated[], +): + | { readonly ok: true; readonly annotated: readonly Annotated[] } + | { readonly ok: false; readonly error: ParseError } { depth = 0; try { - annotated = visit(ast); + return { ok: true, annotated: run() }; } catch (signal) { if (signal instanceof EmitSignal) return { ok: false, error: signal.error }; throw signal; } +} +/** + * The instruction list and the four tables, read off one annotated list in + * one pass, so no two of them can disagree about an index. + */ +function tables(annotated: readonly Annotated[]): { + readonly instructions: Program; + readonly positions: ReadonlyMap; + readonly spans: ReadonlyMap; + readonly segmentPositions: ReadonlyMap; + readonly segmentSpans: ReadonlyMap; +} { const positions = new Map(); const spans = new Map(); + const segmentPositions = new Map(); + const segmentSpans = new Map(); const instructions: Instruction[] = []; for (const [index, entry] of annotated.entries()) { instructions.push(entry.instruction); positions.set(index, entry.position); spans.set(index, entry.span); + if (entry.segments !== undefined) { + segmentPositions.set( + index, + entry.segments.map((segment) => segment.position), + ); + segmentSpans.set( + index, + entry.segments.map((segment) => segment.span), + ); + } } - return { ok: true, instructions, positions, spans }; + return { instructions, positions, spans, segmentPositions, segmentSpans }; } /** One instruction, annotated with the node that emitted it. */ -function own(node: Node, instruction: Instruction): Annotated { +function own(node: Located, instruction: Instruction): Annotated { return { instruction, position: node.position, span: node.span }; } @@ -168,6 +247,14 @@ let depth = 0; * reaches this test. */ function visit(node: Node): readonly Annotated[] { + return deeper(node, () => visitNode(node)); +} + +/** + * Runs one step of the walk a level deeper than its caller, or refuses at the + * node it was about to enter because the tree is already at the limit. + */ +function deeper(node: Located, step: () => readonly Annotated[]): readonly Annotated[] { if (depth >= SOURCE_DEPTH_LIMIT) { throw new EmitSignal( new ParseError( @@ -180,12 +267,138 @@ function visit(node: Node): readonly Annotated[] { } depth += 1; try { - return visitNode(node); + return step(); } finally { depth -= 1; } } +/** + * A statement sequence, one statement after another, at the level of the + * program or the block that holds it. A sequence is written flat, and is + * walked in a loop. + */ +function visitStatements(statements: readonly Statement[]): readonly Annotated[] { + const emitted: Annotated[] = []; + for (const statement of statements) append(emitted, visitStatement(statement)); + return emitted; +} + +/** + * One statement. + * + * The two control-flow statements are each a level deeper than the sequence + * that holds them, so blocks nested inside one another are bounded as any + * other nesting is; an `else if` is a block holding an `if`, and counts the + * same way. Their jumps are relative, counted from the jump itself, and each + * carries the statement's own position and span: + * + * - `if c { A }` is `c`, a falsy jump past `A`, then `A`; + * - `if c { A } else { B }` is `c`, a falsy jump past `A` and the jump that + * ends it, `A`, a jump past `B`, then `B`; + * - `while c { A }` is `c`, a falsy jump past `A` and the back edge, `A`, then + * a backward jump to the first instruction of `c`. + * + * A bare expression is the expression and a `pop` carrying its node's own + * position and span. + */ +function visitStatement(statement: Statement): readonly Annotated[] { + switch (statement.kind) { + case "assignment": + return visitAssignment(statement); + + case "if": + return deeper(statement, () => { + const emitted: Annotated[] = []; + append(emitted, visit(statement.condition)); + const then = visitBlock(statement.consequent); + if (statement.alternative === null) { + emitted.push(own(statement, ["pop_jump_if_falsy", then.length + 1])); + append(emitted, then); + return emitted; + } + const otherwise = visitBlock(statement.alternative); + emitted.push(own(statement, ["pop_jump_if_falsy", then.length + 2])); + append(emitted, then); + emitted.push(own(statement, ["jump", otherwise.length + 1])); + append(emitted, otherwise); + return emitted; + }); + + case "while": + return deeper(statement, () => { + const condition = visit(statement.condition); + const body = visitBlock(statement.body); + const emitted: Annotated[] = []; + append(emitted, condition); + emitted.push(own(statement, ["pop_jump_if_falsy", body.length + 2])); + append(emitted, body); + emitted.push(own(statement, ["jump_backward", condition.length + body.length + 1])); + return emitted; + }); + + default: { + const emitted: Annotated[] = []; + append(emitted, visit(statement)); + emitted.push(own(statement, ["pop"])); + return emitted; + } + } +} + +function visitBlock(block: Block): readonly Annotated[] { + return visitStatements(block.statements); +} + +/** + * An assignment: the location's segments root first, the value, then one + * `store` whose operand is the number of segments. + * + * A segment is a name pushed as a string - the root identifier's, or a + * property's - or a bracket's key expression, which may be several + * instructions and is still one segment. The `store` carries the root's + * POSITION and the whole statement's SPAN: a failed write is blamed on the + * location rather than on the `=` that is the statement's own position, and + * a span already starts at the root and underlines the statement. Its + * segment annotations are the root identifier's, each property access + * node's, and each bracket key's. + * + * The location is a chain written flat, and is walked in a loop. + */ +function visitAssignment(statement: AssignmentStatement): readonly Annotated[] { + const links: Node[] = []; + let root: Node = statement.target; + while (root.kind === "property_access" || root.kind === "bracket_access") { + links.push(root); + root = root.object; + } + if (root.kind !== "identifier") { + // The grammar builds an assignment over a location only. + throw new Error("an assignment's target is not a location"); + } + + const emitted: Annotated[] = [own(root, ["lit", root.name])]; + const segments: Located[] = [root]; + for (let at = links.length - 1; at >= 0; at -= 1) { + const link = links[at] as Node; + if (link.kind === "property_access") { + emitted.push(own(link, ["lit", link.property])); + segments.push(link); + } else if (link.kind === "bracket_access") { + append(emitted, visit(link.key)); + segments.push(link.key); + } + } + append(emitted, visit(statement.value)); + emitted.push({ + instruction: ["store", segments.length], + position: root.position, + span: statement.span, + segments, + }); + return emitted; +} + function visitNode(node: Node): readonly Annotated[] { switch (node.kind) { case "integer": diff --git a/src/errors.ts b/src/errors.ts index d20b816..f43e175 100644 --- a/src/errors.ts +++ b/src/errors.ts @@ -192,6 +192,11 @@ export type ParseReason = | "expected_duration" | "duration_fraction" | "duration_unit_twice" + // Statements: what the program grammar refuses that the expression grammar + // has no site for. Each names one message family of the reference. + | "unexpected_else" + | "unassignable_location" + | "expected_open_brace" // Emission: what building the domain value for a literal refuses. The // reference raises here rather than answering, so this member's message is // this package's own and not one quoted from it. diff --git a/src/index.ts b/src/index.ts index 97a4457..1537f7c 100644 --- a/src/index.ts +++ b/src/index.ts @@ -35,7 +35,14 @@ export type { CompileWithPositionsResult, CompileWithSpansResult, } from "./compile.js"; -export { compile, compileWithPositions, compileWithSpans } from "./compile.js"; +export { + compile, + compileProgram, + compileProgramWithPositions, + compileProgramWithSpans, + compileWithPositions, + compileWithSpans, +} from "./compile.js"; export type { UnboundPolicy } from "./context.js"; export type { Ast, DecompileOptions } from "./decompile.js"; export { decompile } from "./decompile.js"; diff --git a/src/parser.ts b/src/parser.ts index 2e86a02..89a2ee2 100644 --- a/src/parser.ts +++ b/src/parser.ts @@ -11,12 +11,14 @@ * contract. One refusal is not the reference's: the depth bound below, which * the reference has no counterpart for, so its message is authored here. * - * The grammar is the expression production alone. The statement production - - * assignment, the separator, the two control-flow keywords - is not written - * here, and the two ways an expression can run into it are refusals rather - * than omissions: a bare `=` says that `==` is the equality operator, and a - * statement keyword says where control flow is valid. A grammar that could not - * say why it refused would be worse than one that can. + * There are two entry points, as there are in the reference. `parse` reads the + * expression production alone, and the two ways an expression can run into + * statement syntax are refusals rather than omissions there: a bare `=` says + * that `==` is the equality operator, and a statement keyword says where + * control flow is valid. `parseProgram` reads the statement production - + * assignment, the separator, `if`/`else` and `while` - over the same + * expression production, and never answers a bare expression tree: a program + * of one statement is still a program. * * Three things about the shape are worth naming. * @@ -48,15 +50,20 @@ import type { ArithmeticOperator, + Block, ComparisonOperator, DurationNode, DurationUnit, + IfStatement, MembershipOperator, Node, ObjectEntry, ObjectKey, ObjectKeyStyle, + ProgramNode, RelativeDirection, + Statement, + WhileStatement, } from "./ast.js"; import { DURATION_UNIT_TABLE } from "./duration-units.js"; import { ParseError, type ParseReason, type Position, type Span } from "./errors.js"; @@ -113,6 +120,86 @@ export function parse(tokens: readonly Token[]): ParseResult { }; } +/** A statement program, or the one refusal that stopped it. */ +export type ProgramParseResult = + | { readonly ok: true; readonly program: ProgramNode } + | { readonly ok: false; readonly error: ParseError }; + +/** + * Parses a token stream into a statement program. + * + * The production is `program := statement (";" statement)* [";"]`, where a + * statement is an assignment, an `if`, a `while` or a bare expression, and a + * statement ending in a closing brace needs no separator before the next one. + * A program holds at least one statement, so an empty source and a lone + * separator are refusals, and so is a separator doubled anywhere. Every token + * has to be consumed: what is left over past the last statement is refused at + * the first leftover token. + */ +export function parseProgram(tokens: readonly Token[]): ProgramParseResult { + const parser = new Parser(tokens); + const start = tokenStart(parser.peek()); + const statements = parser.statementSequence("eof"); + if (!statements.ok) return { ok: false, error: statements.error }; + + const trailing = parser.peek(); + if (trailing.type === "eof") { + const first = statements.value[0] as Statement; + const last = statements.value[statements.value.length - 1] as Statement; + return { + ok: true, + program: { + kind: "program", + statements: statements.value, + position: start, + span: { start: first.span.start, end: last.span.end }, + }, + }; + } + return { ok: false, error: afterStatement(trailing) }; +} + +/** + * The refusal for a token left over after a finished statement: the stray + * `else` has a message of its own, and every other token is named in the + * general one. + */ +function afterStatement(token: Token): ParseError { + if (token.type === "else_kw") return strayElse(token); + return refuse("trailing_token", `Unexpected token ${formatToken(token)} after statement`, token); +} + +/** An `else` where no `if` block has just closed. */ +function strayElse(token: Token): ParseError { + return refuse( + "unexpected_else", + "Unexpected 'else' - an 'else' block must follow an 'if' block.", + token, + ); +} + +/** A statement that ends in a closing brace, which the next may follow directly. */ +function braceTerminated(statement: Statement): boolean { + return statement.kind === "if" || statement.kind === "while"; +} + +/** + * Whether a node is a location an assignment can write: an identifier, + * followed by any run of property and bracket accesses. A parenthesis around + * any part of it changes nothing, because the grammar keeps no node for one. + * + * The chain is walked in a loop rather than by recursion, for the reason the + * emitter gives for its chains: its length is written flat. + */ +function isLocation(node: Node): boolean { + let link = node; + for (;;) { + if (link.kind === "identifier") return true; + if (link.kind !== "property_access" && link.kind !== "bracket_access") return false; + link = link.object; + } +} + class Parser { private readonly tokens: readonly Token[]; private index = 0; @@ -182,6 +269,215 @@ class Parser { return this.descend(() => this.logicalOr()); } + /** + * One or more statements joined by separators, up to the token that ends + * them, which is left unconsumed for the caller to read. + * + * A separator is consumed and, unless it is the one trailing separator the + * grammar allows before the terminator, another statement is read after it. + * Without a separator, another statement follows only a statement that ended + * in a closing brace. The sequence is read in a loop, so its length costs + * no depth. + */ + statementSequence(terminator: "eof" | "rbrace"): Parsed { + const first = this.statement(); + if (!first.ok) return first; + const statements: Statement[] = [first.value]; + + for (;;) { + if (this.peek().type === "semicolon") { + this.advance(); + if (this.atTerminator(terminator)) break; + } else if ( + !braceTerminated(statements[statements.length - 1] as Statement) || + this.atTerminator(terminator) + ) { + break; + } + const next = this.statement(); + if (!next.ok) return next; + statements.push(next.value); + } + return { ok: true, value: statements }; + } + + /** + * Whether the cursor is at the token that ends a sequence. A stream that has + * run out counts, as the reference counts it; only a hand-built list can. + */ + private atTerminator(terminator: "eof" | "rbrace"): boolean { + return this.peek().type === terminator || this.index >= this.tokens.length; + } + + // statement -> if_statement | while_statement | assignment | expression + private statement(): Parsed { + const token = this.peek(); + if (token.type === "if_kw") return this.ifStatement(token); + if (token.type === "while_kw") return this.whileStatement(token); + if (token.type === "else_kw") return { ok: false, error: strayElse(token) }; + return this.assignmentOrExpression(); + } + + /** + * An assignment when the statement opens with something location-shaped + * followed by `=`, and a bare expression otherwise. + * + * The probe reads an additive expression, one level below the comparison + * production that refuses a bare `=`, and looks at the token after it. A + * probe that refuses, or that is not followed by `=`, is discarded and the + * statement is read again from its start as an expression - which refuses + * the same way wherever the probe did, since the expression grammar meets + * the same leading tokens first. Once an `=` is seen the statement is an + * assignment, and a left side that is not a location is refused at the `=`. + */ + private assignmentOrExpression(): Parsed { + const start = this.index; + const candidate = this.descend(() => this.addition()); + const equals = this.peek(); + if (!candidate.ok || equals.type !== "eq") { + this.index = start; + return this.expression(); + } + + if (!isLocation(candidate.value)) { + return { + ok: false, + error: refuse( + "unassignable_location", + "Left side of '=' must be an assignable location - an identifier, a property " + + "access, or a bracket access.", + equals, + ), + }; + } + this.advance(); + const value = this.expression(); + if (!value.ok) return value; + return { + ok: true, + value: { + kind: "assignment", + target: candidate.value, + value: value.value, + position: tokenStart(equals), + span: infixSpan(candidate.value, value.value), + }, + }; + } + + // if_statement -> "if" expression block ( "else" ( block | if_statement ) )? + private ifStatement(keyword: Token): Parsed { + this.advance(); + const condition = this.expression(); + if (!condition.ok) return condition; + const then = this.block(); + if (!then.ok) return then; + + let otherwise: Block | null = null; + if (this.peek().type === "else_kw") { + this.advance(); + const next = this.peek(); + if (next.type === "if_kw") { + // An `else if` is an else block holding one `if`, which is a level of + // nesting like any block. + const nested = this.descend(() => this.ifStatement(next)); + if (!nested.ok) return nested; + otherwise = { + kind: "block", + statements: [nested.value], + position: nested.value.position, + span: nested.value.span, + }; + } else { + const block = this.block(); + if (!block.ok) return block; + otherwise = block.value; + } + } + + const last = otherwise ?? then.value; + return { + ok: true, + value: { + kind: "if", + condition: condition.value, + consequent: then.value, + alternative: otherwise, + position: tokenStart(keyword), + span: { start: tokenStart(keyword), end: last.span.end }, + }, + }; + } + + // while_statement -> "while" expression block + private whileStatement(keyword: Token): Parsed { + this.advance(); + const condition = this.expression(); + if (!condition.ok) return condition; + const body = this.block(); + if (!body.ok) return body; + return { + ok: true, + value: { + kind: "while", + condition: condition.value, + body: body.value, + position: tokenStart(keyword), + span: { start: tokenStart(keyword), end: body.value.span.end }, + }, + }; + } + + /** + * block -> "{" ( statement (";" statement)* ";"? )? "}" + * + * The statements inside are one level deeper than the block, which is what + * bounds how far blocks may nest inside one another. + */ + private block(): Parsed { + const open = this.peek(); + if (open.type !== "lbrace") { + return { + ok: false, + error: refuse( + "expected_open_brace", + `Expected '{' to open a block but found ${formatToken(open)}`, + open, + ), + }; + } + this.advance(); + + let statements: readonly Statement[] = []; + if (this.peek().type !== "rbrace") { + const body = this.descend(() => this.statementSequence("rbrace")); + if (!body.ok) return body; + statements = body.value; + } + + const close = this.peek(); + if (close.type !== "rbrace") { + return { + ok: false, + error: refuse( + "expected_close_brace", + `Expected '}' to close the block but found ${formatToken(close)}`, + close, + ), + }; + } + this.advance(); + return { + ok: true, + value: { + kind: "block", + statements, + position: tokenStart(open), + span: { start: tokenStart(open), end: tokenEnd(close) }, + }, + }; + } + // logical_or -> logical_and ( ("OR" | "||") logical_and )* private logicalOr(): Parsed { let left = this.logicalAnd(); diff --git a/test/compile-program.test.ts b/test/compile-program.test.ts new file mode 100644 index 0000000..a6138d9 --- /dev/null +++ b/test/compile-program.test.ts @@ -0,0 +1,327 @@ +// The statement compiler's own properties, beside the transcript. +// +// `test/reference-compile.test.ts` diffs every program row of the compile +// transcript against `compileProgram` and `compileProgramWithSpans`. Those +// rows carry the reference's SPAN tables, because the reference was run in +// its span mode to write them, so three things are pinned here instead. +// +// The POINT tables. The expected tables below were produced by running +// `Predicator.compile_program_with_positions/1` against a detached export of +// predicator-ex at `v9.4.2` (`mix.exs` `@version` reads `9.4.2` in that +// export) on 2026-10-01, over the same sources the transcript's rows hold; +// each is quoted as the run printed it, a point per instruction index and a +// point per segment of each store. Every program row was compared the same +// way when this file was written and all of them agreed. +// +// The REASONS. A program refusal row carries the reference's message, +// position and span and no reason, because the reason is this package's token +// for a message family. Which member each family maps to is decided by the +// compiler-surface record's amendment on the statement grammar, and pinned +// here family by family. +// +// The BOUNDS, and what did not move. The program grammar and its emission +// count their nesting against the one declared source limit, a statement +// sequence costs no depth however long it is, and the expression entry points +// answer exactly what they answered before the statement grammar existed. + +import { describe, expect, it } from "vitest"; +import { + type CompileResult, + type CompileWithPositionsResult, + type CompileWithSpansResult, + compile, + compileProgram, + compileProgramWithPositions, + compileProgramWithSpans, + type ParseReason, + type Position, +} from "../src/index.js"; +import { SOURCE_DEPTH_LIMIT } from "../src/nesting.js"; + +/** A point table as the reference run printed it: index, line, column. */ +type PointRow = readonly [number, number, number]; + +/** A segment table as the reference run printed it. */ +type SegmentRow = readonly [number, readonly (readonly [number, number])[]]; + +function pointRows(table: ReadonlyMap): readonly PointRow[] { + return [...table.entries()] + .sort(([left], [right]) => left - right) + .map(([index, point]) => [index, point.line, point.column] as const); +} + +function segmentRows(table: ReadonlyMap): readonly SegmentRow[] { + return [...table.entries()] + .sort(([left], [right]) => left - right) + .map( + ([index, points]) => + [index, points.map((point) => [point.line, point.column] as const)] as const, + ); +} + +/** One source and the point tables the reference answered for it. */ +interface PointCase { + readonly source: string; + readonly positions: readonly PointRow[]; + readonly segments: readonly SegmentRow[]; +} + +const POINT_CASES: readonly PointCase[] = [ + { + // A store carries the location's root, not the `=`; a dotted segment + // carries its property name. + source: "loan.renewals = loan.renewals + 1", + positions: [ + [0, 1, 1], + [1, 1, 6], + [2, 1, 17], + [3, 1, 22], + [4, 1, 33], + [5, 1, 31], + [6, 1, 1], + ], + segments: [ + [ + 6, + [ + [1, 1], + [1, 6], + ], + ], + ], + }, + { + // A bracket segment carries its key. + source: "holds[0] = patron.id", + positions: [ + [0, 1, 1], + [1, 1, 7], + [2, 1, 12], + [3, 1, 19], + [4, 1, 1], + ], + segments: [ + [ + 4, + [ + [1, 1], + [1, 7], + ], + ], + ], + }, + { + // A parenthesis widens a span and leaves a point alone. + source: "(loan).renewals = 0", + positions: [ + [0, 1, 2], + [1, 1, 8], + [2, 1, 19], + [3, 1, 2], + ], + segments: [ + [ + 3, + [ + [1, 2], + [1, 8], + ], + ], + ], + }, + { + // A `pop` carries the expression it discards; a program that assigns + // nothing has an empty segment table. + source: "len(patron.holds)", + positions: [ + [0, 1, 5], + [1, 1, 12], + [2, 1, 1], + [3, 1, 1], + ], + segments: [], + }, + { + // Both jumps of an if/else carry the `if` keyword. + source: "if loan.overdue { loan.status = 'late' } else { loan.status = 'on-time' }", + positions: [ + [0, 1, 4], + [1, 1, 9], + [2, 1, 1], + [3, 1, 19], + [4, 1, 24], + [5, 1, 33], + [6, 1, 19], + [7, 1, 1], + [8, 1, 49], + [9, 1, 54], + [10, 1, 63], + [11, 1, 49], + ], + segments: [ + [ + 6, + [ + [1, 19], + [1, 24], + ], + ], + [ + 11, + [ + [1, 49], + [1, 54], + ], + ], + ], + }, + { + // Both jumps of a while carry the `while` keyword. + source: "while renewals < 3 { renewals = renewals + 1 }", + positions: [ + [0, 1, 7], + [1, 1, 18], + [2, 1, 16], + [3, 1, 1], + [4, 1, 22], + [5, 1, 33], + [6, 1, 44], + [7, 1, 42], + [8, 1, 22], + [9, 1, 1], + ], + segments: [[8, [[1, 22]]]], + }, +]; + +describe("compileProgramWithPositions answers the reference's point tables", () => { + // Sabotage, each run and reverted: a `store` given the assignment's own + // position (the `=`) rather than the root's turns every case that assigns + // red; a property segment annotated with the root identifier rather than the + // access node turns red the three cases that write a property. + it.each(POINT_CASES.map((entry) => [entry.source, entry] as const))("%s", (_source, entry) => { + const compiled = compileProgramWithPositions(entry.source); + expect(compiled.ok).toBe(true); + if (!compiled.ok) return; + expect(pointRows(compiled.positions)).toEqual(entry.positions); + expect(segmentRows(compiled.segmentPositions)).toEqual(entry.segments); + }); +}); + +describe("each statement refusal family answers its reason", () => { + // One source per family the program grammar adds or reaches, the reason + // each maps to, and the message family it stands for. The messages + // themselves are the transcript's to pin. + const FAMILIES: readonly (readonly [string, ParseReason])[] = [ + ["else { status = 'late' }", "unexpected_else"], + ["status = 'late' else { status = 'on-time' }", "unexpected_else"], + ["renewals = 1 fines = 0", "trailing_token"], + ["3 = renewals", "unassignable_location"], + ["len(holds) = 0", "unassignable_location"], + ["if loan.overdue status = 'late'", "expected_open_brace"], + ["if loan.overdue { fines = 1", "expected_close_brace"], + ["status = while", "statement_keyword"], + ["renewals = fines = 0", "assignment_in_expression"], + ["renewals = 1;;", "expected_primary"], + ["", "expected_primary"], + ]; + + // Sabotage, each run and reverted: the stray-else refusal given + // `statement_keyword` turns the first two red, and the unterminated block + // given a member of its own turns its row red. + it.each(FAMILIES)("%j refuses with %s", (source, reason) => { + const compiled = compileProgram(source); + expect(compiled.ok ? "compiled" : compiled.error.reason).toBe(reason); + }); + + it("names an end-of-input token rather than running out", () => { + const compiled = compileProgram("if loan.overdue"); + expect(compiled.ok ? null : [compiled.error.reason, compiled.error.message]).toEqual([ + "expected_open_brace", + "Expected '{' to open a block but found end of input", + ]); + }); +}); + +describe("the three entry points answer their expression counterparts' unions", () => { + // Assignability is the claim: a caller that handles the expression entry + // point's answer handles this one unchanged. The extra segment table is a + // member the expression result does not name, not a different shape. + it("are assignable to them, on both arms", () => { + const plain: CompileResult = compileProgram("renewals = 0"); + const pointed: CompileWithPositionsResult = compileProgramWithPositions("renewals = 0"); + const spanned: CompileWithSpansResult = compileProgramWithSpans("renewals = 0"); + const refused: CompileResult = compileProgram("3 = renewals"); + expect([plain.ok, pointed.ok, spanned.ok, refused.ok]).toEqual([true, true, true, false]); + }); + + it("refuse as values, never by throwing", () => { + const sources = ["", ";", "}", "else", "{", "if if", "while { }", "a = ", "a.b[", "x = 1 ="]; + for (const source of sources) { + for (const entry of [compileProgram, compileProgramWithPositions, compileProgramWithSpans]) { + const answer = entry(source); + expect(answer.ok ? "compiled" : answer.error.type, source).toBe("ParseError"); + } + } + }); +}); + +describe("the statement grammar is bounded the way the expression grammar is", () => { + /** `if c { ... }` nested `depth` times around one assignment. */ + function nestedIfs(depth: number): string { + return `${"if ready { ".repeat(depth)}renewals = 0${" }".repeat(depth)}`; + } + + // Sabotage, each run and reverted: the `descend` around a block's statements + // removed in src/parser.ts leaves this green on its own, because the emitter + // still counts the level each `if` takes; removed together with that level, + // in `visitStatement` in src/emitter.ts, it turns this red. + it("refuses blocks nested past the declared source limit, as a value", () => { + const compiled = compileProgram(nestedIfs(SOURCE_DEPTH_LIMIT + 1)); + expect(compiled.ok ? "compiled" : compiled.error.reason).toBe("nesting_depth_exceeded"); + }); + + it("compiles blocks nested well inside the limit", () => { + expect(compileProgram(nestedIfs(100)).ok).toBe(true); + }); + + it("costs no depth for the length of a statement sequence", () => { + // Each statement is four instructions: the root, the index, the value, the store. + const sequence = Array.from({ length: 5000 }, (_, index) => `holds[${index}] = 0`).join("; "); + const compiled = compileProgram(sequence); + expect(compiled.ok ? compiled.instructions.length : compiled.error.reason).toBe(5000 * 4); + }); + + // Sabotage, run and reverted: the `descend` around an `else if` removed in + // src/parser.ts, together with the level an `if` takes in src/emitter.ts, + // turns this red. + it("counts an else-if chain as the nesting it is", () => { + const chain = `if fines > 0 { status = 'a' }${" else if fines > 0 { status = 'b' }".repeat( + SOURCE_DEPTH_LIMIT, + )}`; + const compiled = compileProgram(chain); + expect(compiled.ok ? "compiled" : compiled.error.reason).toBe("nesting_depth_exceeded"); + }); +}); + +describe("the expression entry points answer what they answered before", () => { + // The statement grammar is reached only through the three program entry + // points; `compile` still refuses statement syntax with the expression + // grammar's own reasons. + it.each([ + ["renewals = 0", "assignment_in_expression"], + ["if loan.overdue { fines = 1 }", "statement_keyword"], + ["renewals = 0; fines = 0", "assignment_in_expression"], + ] as const)("compile(%j) still refuses with %s", (source, reason) => { + const compiled = compile(source); + expect(compiled.ok ? "compiled" : compiled.error.reason).toBe(reason); + }); + + it("compiles an expression to the list compileProgram ends in a pop", () => { + const expression = compile("loan.renewals < 3"); + const program = compileProgram("loan.renewals < 3"); + expect(expression.ok && program.ok).toBe(true); + if (!expression.ok || !program.ok) return; + expect(program.instructions).toEqual([...expression.instructions, ["pop"]]); + }); +}); diff --git a/test/export-surface.json b/test/export-surface.json index f54eb4c..5148c4a 100644 --- a/test/export-surface.json +++ b/test/export-surface.json @@ -37,6 +37,9 @@ "UndefinedVariableError", "Value", "compile", + "compileProgram", + "compileProgramWithPositions", + "compileProgramWithSpans", "compileWithPositions", "compileWithSpans", "decompile", diff --git a/test/reference-compile.test.ts b/test/reference-compile.test.ts index fb9d87b..fce55b4 100644 --- a/test/reference-compile.test.ts +++ b/test/reference-compile.test.ts @@ -35,13 +35,18 @@ // A PROGRAM ROW and a PROGRAM REFUSAL ROW hold what the reference's statement // compiler answered for one of the programs `scripts/lib/program-sources.mjs` // lists: an instruction list with its `positions` and `segment_positions` -// tables, or a refusal with its message, position and span. This package does -// not compile statements yet, so nothing here diffs them against an answer of -// its own; what is asserted is that every row is well formed and that the rows -// are exactly the list, in its order, each under the kind it was authored to -// draw. They are the oracle the statement compiler is built against, and an -// oracle that has quietly lost a row, gained one, or carries a table that does -// not fit its own program is worse than none. +// tables, or a refusal with its message, position and span. Each is diffed +// against `compileProgram` and `compileProgramWithSpans` in full: the +// instruction list compared as values, both tables entry by entry and index +// by index, and a refusal's message verbatim with its position and span. The +// rows are also held well formed, and exactly the list, in its order, each +// under the kind it was authored to draw: they are the oracle the statement +// compiler is built against, and an oracle that has quietly lost a row, +// gained one, or carries a table that does not fit its own program is worse +// than none. A program refusal row carries no reason label, so a refusal's +// reason is asserted to be a member of the closed union, as a refusal row's +// is, and which member each message family maps to is pinned where the +// grammar is tested. // // DIVERGENCES ARE DECLARED, NEVER NARROWED. `DECLARED`, in // `test/conformance/compile-divergences.ts`, holds both answers for any row @@ -78,7 +83,14 @@ import { fileURLToPath } from "node:url"; import { describe, expect, it } from "vitest"; import { PROGRAM_SOURCES } from "../scripts/lib/program-sources.mjs"; import type { DecompileOptions, ParseReason, Position, Span } from "../src/index.js"; -import { compile, decompile, parse } from "../src/index.js"; +import { + compile, + compileProgram, + compileProgramWithPositions, + compileProgramWithSpans, + decompile, + parse, +} from "../src/index.js"; import { SOURCE_DEPTH_LIMIT } from "../src/nesting.js"; import { decodeTagged } from "../src/tagged.js"; import type { Value } from "../src/values.js"; @@ -122,6 +134,9 @@ const PARSE_REASONS = [ "expected_duration", "duration_fraction", "duration_unit_twice", + "unexpected_else", + "unassignable_location", + "expected_open_brace", "number_out_of_range", "nesting_depth_exceeded", ] as const satisfies readonly ParseReason[]; @@ -670,3 +685,112 @@ describe("what the reference renders, this package renders", () => { expect(ours === row.rendered, `${id}: the two now agree`).toBe(false); }); }); + +/** A table entry as the transcript writes it, from a map this package answers. */ +function spanEntries(table: ReadonlyMap): readonly PositionEntry[] { + return [...table.entries()] + .sort(([left], [right]) => left - right) + .map(([instruction, span]) => ({ instruction, span: plainSpan(span) })); +} + +function segmentEntries(table: ReadonlyMap): readonly SegmentEntry[] { + return [...table.entries()] + .sort(([left], [right]) => left - right) + .map(([instruction, spans]) => ({ instruction, spans: spans.map(plainSpan) })); +} + +/** A span copied field by field, so a key order or a frozen object cannot matter. */ +function plainSpan(span: Span): Span { + return { + start: { line: span.start.line, column: span.start.column }, + end: { line: span.end.line, column: span.end.column }, + }; +} + +const COMPILED_PROGRAM_ROWS = PROGRAM_ROWS.filter( + (row): row is ProgramRow => row.kind === "program", +); +const REFUSED_PROGRAM_ROWS = PROGRAM_ROWS.filter( + (row): row is ProgramRefusalRow => row.kind === "program_refusal", +); + +describe("what the reference compiles as a program, this package compiles", () => { + // Sabotage, each run and reverted: the `jump` that ends a then block dropped + // in `visitStatement` in src/emitter.ts turns red every row whose `if` has an + // `else`; the `pop` after a bare expression statement dropped turns red every + // row holding one; the `while` back edge dropped turns red every row holding + // a loop; a `store` spanned from the `=` rather than from the location turns + // red every row that assigns; a bracket segment annotated with the access + // node rather than its key turns the bracket rows red; and the separator + // before a closing brace no longer accepted as a trailing one, in + // `statementSequence` in src/parser.ts, turns the block-with-separators row + // red. Every one failed on an assertion. + it.each(COMPILED_PROGRAM_ROWS.map((row) => [row.id, row] as const))("%s", (id, row) => { + const plain = compileProgram(row.source); + const spanned = compileProgramWithSpans(row.source); + const pointed = compileProgramWithPositions(row.source); + expect( + [plain.ok, spanned.ok, pointed.ok], + `${id}: the reference compiled this program and this package refused it`, + ).toEqual([true, true, true]); + if (!plain.ok || !spanned.ok || !pointed.ok) return; + + const ours: Value = plain.instructions.map((instruction) => [...instruction]); + expect( + sameValue(ours, row.instructions), + `${id}: this package and the reference emit different programs - ${JSON.stringify(ours)}`, + ).toBe(true); + // The three entry points run one compilation, so their lists are one list. + expect(spanned.instructions).toEqual(plain.instructions); + expect(pointed.instructions).toEqual(plain.instructions); + + expect(spanEntries(spanned.spans), `${id}: the spans table is not the reference's`).toEqual( + row.positions, + ); + expect( + segmentEntries(spanned.segmentSpans), + `${id}: the segment table is not the reference's`, + ).toEqual(row.segment_positions); + // The point tables have no row to diff against; they are keyed exactly as + // the span tables are, which is what one walk building all four promises. + expect([...pointed.positions.keys()]).toEqual([...spanned.spans.keys()]); + expect( + [...pointed.segmentPositions.entries()].map(([index, points]) => [index, points.length]), + ).toEqual([...spanned.segmentSpans.entries()].map(([index, spans]) => [index, spans.length])); + }); +}); + +describe("what the reference refuses as a program, this package refuses the same way", () => { + // Sabotage, each run and reverted: the stray-else message's wording changed + // in src/parser.ts turns the message assertion red on the three stray-else + // rows, and the unassignable-location message's wording changed there turns + // it red on the three not-a-location rows. + it.each(REFUSED_PROGRAM_ROWS.map((row) => [row.id, row] as const))("%s", (id, row) => { + const compiled = compileProgram(row.source); + expect( + compiled.ok, + `${id}: the reference refused this program and this package compiled it`, + ).toBe(false); + if (compiled.ok) return; + const error = compiled.error; + expect(PARSE_REASONS, `${id}: the reason is outside the closed union`).toContain(error.reason); + expect(error.message, `${id}: the message is not the reference's`).toBe(row.message); + expect( + samePosition(error.position, row.position), + `${id}: the position is not the reference's - ${JSON.stringify(error.position)} against ${JSON.stringify(row.position)}`, + ).toBe(true); + expect( + sameSpan(error.span, row.span), + `${id}: the span is not the reference's - ${JSON.stringify(error.span)} against ${JSON.stringify(row.span)}`, + ).toBe(true); + // The other two entry points refuse with the same value. + for (const other of [ + compileProgramWithPositions(row.source), + compileProgramWithSpans(row.source), + ]) { + expect(other.ok ? null : other.error, `${id}: the entry points refuse differently`).toEqual( + error, + ); + } + }); +});