[{"data":1,"prerenderedAt":4},["ShallowReactive",2],{"b8ojfNfLgg":3},"# Machine-Checked Grammar-Constrained Decoding\n\nA Lean 4 formalization of grammar-constrained decoding (GCD), following the\nalgorithm of [*Flexible and Efficient Grammar-Constrained Decoding*](https://arxiv.org/pdf/2502.05111)\n(Park et al.). It builds an executable next-token checker by composing a\ndetokenizing lexer transducer with a pushdown parser, and proves that the\nchecker is **sound** and **complete**: after any token prefix it allows exactly\nthe next tokens that can still be extended to a grammatical output.\n\n## Highlights\n\n- **An end-to-end machine-checked correctness theorem** (`GCDChecker_correct`)\n  for an executable GCD checker, under a small, explicit bundle of assumptions.\n- **A bug found in the published lexing construction.** The original lexer has no\n  distinguished start state, so it accepts end-of-stream whenever the lexing\n  automaton happens to return to its start state, i.e. in the middle of an\n  unfinished terminal. The regex `(ab)*a` triggers it. We give a corrected\n  construction with a separate start state and verify it.\n- **The assumptions made precise.** The paper leaves several conditions implicit.\n  The one that does real work is the *universal separator*: a distinguished\n  whitespace terminal that ends any lexeme and is a no-op for the parser. It is\n  what collapses the otherwise unbounded set of lexer continuations to a finite\n  check.\n- **Reusable infrastructure** with no GCD-specific dependencies: partial finite\n  automata and transducers with executable composition (`Automata.lean`),\n  nondeterministic pushdown automata with stack lemmas and a stack-forgetting NFA\n  over-approximation (`PDA.lean`), and a verified finite graph search\n  (`Producible.lean`).\n\n## Building\n\nRequires [Lean 4](https://lean-lang.org/), toolchain `leanprover/lean4:v4.29.0-rc6`\n(see `lean-toolchain`); mathlib is pinned in `lake-manifest.json`.\n\n```bash\nlake exe cache get   # prebuilt mathlib oleans, recommended before the first build\nlake build ConstrainedDecodingFormalization ConstrainedDecodingFormalization.GCDTest\n```\n\nThe development contains no `sorry` and declares no axioms.\n\n## The pipeline\n\nAn LLM emits **tokens**; a grammar is defined over **terminals** that a lexer\ngroups from **characters**. The target semantics is the composition\n\n```\ntokens --detokenize--> characters --lex--> terminals --parse--> accept\n```\n\nA complete token sequence is valid exactly when this pipeline succeeds. The\nchecker decides, incrementally, whether a candidate token keeps that pipeline\ncompletable.\n\n1. **Lexing.** `BuildLexingFST` compiles a lexer specification (a character\n   automaton labeled with terminals) into a one-lookahead maximal-munch\n   transducer. `Lexing/Correctness.lean` proves it equivalent to the relational\n   specification `PartialLexRel`.\n2. **Detokenization.** `BuildDetokenizingFST` flattens tokens to characters;\n   composing it with the lexer gives `BuildDetokLexer`, driven directly by tokens.\n3. **Realizable tails.** Modulo whitespace, the terminal sequences the lexer can\n   still produce from a state are exactly those whose first terminal is\n   *single-producible* there. This is the finiteness result the checker rests on.\n4. **Tables.** `BuildInverseTokenSpannerTable` records, per lexer state, the\n   *realizable sequence heads* each token exposes and inverts that map back to\n   tokens. `PreprocessParser` sorts heads into always-allowed, always-rejected,\n   and stack-dependent using stack invariance and NFA over-approximation.\n5. **Online mask.** `ComputeValidTokenMask` seeds the mask with the\n   always-allowed tokens and tests each stack-dependent head against the live\n   parser configuration. `GCDChecker` wires this to a prefix evaluation.\n\n## Module map\n\n| File | Role |\n|------|------|\n| `Char.lean` | EOS-extended alphabet `ExtChar α` (abbrev `Ch α`) |\n| `Language.lean` | Prefix closure `Language.prefixes`, bridging to `Mathlib.Computability.Language` |\n| `Automata.lean` | Partial deterministic FSA and FST, executable composition, mathlib DFA/NFA conversions |\n| `PDA.lean` | Pushdown automaton, stack semantics, `evalFrom`, `toNFA` over-approximation |\n| `Producible.lean` | Depth-first search for single-producible terminals, with its correctness proof |\n| `Vocabulary.lean` | `Vocabulary α β` typeclass: tokens to character strings, with the singleton-token law |\n| `Lexing/Base.lean` | Lexer specs, the partial lexer `PartialLex`/`PartialLexRel`, and `BuildLexingFST` |\n| `Lexing/Correctness.lean` | Equivalence of partial lexing, the relational lexer, and the lexing FST |\n| `Lexing/Detokenizing.lean` | Detokenizing FST and its composition `BuildDetokLexer` |\n| `Lexing/Whitespace.lean` | Whitespace-exchange lemmas and the realizable-tail characterization |\n| `Lexing.lean` | Compatibility import for the four `Lexing/` modules |\n| `RealizableSequence.lean` | Realizable sequence heads and the inverse token-spanner table |\n| `Checker.lean` | The executable `Checker β` interface and its language-level semantics |\n| `ParserWithEOS.lean` | EOS-augmented parser used when lexer output carries an end marker |\n| `GCDAssumptions.lean` | The `GCDAssumptions` bundle, including the universal-separator condition |\n| `GCDAlgorithm.lean` | `PreprocessParser`, `ComputeValidTokenMask`, and the executable `GCDChecker` |\n| `GCDStepProofs.lean` | Step-level mask correctness: `Soundness`, `Completeness`, `EOSCompleteness` |\n| `GCDCheckerLanguage.lean` | Bridge to `checkerLanguage = TargetLanguage` |\n| `GCDProductivity.lean` | Productivity, path independence, and the final `GCDChecker_correct` |\n| `GrammarConstrainedDecoding.lean` | Compatibility import for the GCD proof stack |\n| `GCDTest.lean` | A finite JSON-like grammar with all assumptions discharged (`jsonChecker_correct`) |\n\n## Main theorems\n\n| Theorem | Statement |\n|---------|-----------|\n| `GCDChecker_correct` | The checker satisfies the full `checkerCorrect` interface: EOS is allowed iff the prefix is in the target language, and every allowed token sequence is a prefix of some target-language word. |\n| `GCDChecker_checkerLanguage_eq_TargetLanguage` | The accepted token language of `GCDChecker spec P` equals the lexer/parser target language. |\n| `GCDChecker_intermediateLanguage_eq_TargetLanguage_prefixes` | The prefixes the checker allows equal the prefix closure of the target language. |\n| `GCDChecker_productive` | Every incrementally allowed prefix extends to an accepted word. |\n| `GCDChecker_pathIndependent` | The checker depends only on the flattened character content of the prefix. |\n| `Soundness` / `Completeness` / `EOSCompleteness` | Step level: a token's mask bit is true iff a viable continuation exists through the composed FST and parser. |\n| `computeSingleProducible_correct` | The executable DFS enumerates exactly the single-producible terminals. |\n| `mem_ComputeValidTokenMask_preprocess_iff` | Semantic membership characterization of the online mask. |\n\n## Assumptions\n\nThe final theorems are parameterized by one package,\n`GCDAssumptions spec P tnonwhite twhite qnonwhite qwhite`:\n\n```lean\nstructure GCDAssumptions\n    (spec : LexerSpec α Γ σa) (P : PDA Γ π σp)\n    (tnonwhite twhite : α) (qnonwhite qwhite : σa) : Prop where\n  hempty        : [] ∉ spec.automaton.accepts\n  lexer_pruned  : spec.automaton.pruned\n  parser_pruned : P.pruned\n  whitespace    : GCDWhitespaceAssumption spec P tnonwhite twhite qnonwhite qwhite\n```\n\n- **No empty lexeme (`hempty`).** The lexer automaton's start state is not\n  accepting, so no lexeme is empty.\n- **Lexer prunedness.** Every reachable lexer state can still reach an accepting\n  state. Used by the realizable-tail argument.\n- **Parser prunedness.** Every reachable parser configuration has an accepted\n  continuation.\n- **Universal separator (`GCDWhitespaceAssumption`).** The condition is organized\n  around a distinguished whitespace character `twhite` (with a witnessing\n  non-whitespace character `tnonwhite`, and the lexer states `qwhite`/`qnonwhite`\n  they lead to). On the lexer side, `twhite` belongs to no lexeme other than the\n  whitespace terminal, so it always ends the preceding lexeme and returns the\n  lexer to a clean post-separator state. On the parser side,\n  `ParserIgnoresTerminal` holds: every state reads the whitespace terminal with\n  the identity transition.\n- **Singleton tokens.** Carried by the `Vocabulary α β` instance: every single\n  character is itself a token (`flatten (embed a) = [a]`) and no token flattens\n  to nothing (`flatten b ≠ []`).\n\nTo instantiate the end-to-end theorem for a grammar, supply finite/enumerable\nalphabets and states, a `Vocabulary` instance, and a proof of `GCDAssumptions`.\nThe generic theorems are not reproved. `GCDTest.lean` does this for a shallow\nJSON grammar, using newline as the separator; `native_decide` discharges the\nfinite side conditions there.\n\n## Dependency visualizer\n\nAn interactive declaration dependency graph is at\n**[ucsd-formal.github.io/constrained-decoding-formalization](https://ucsd-formal.github.io/constrained-decoding-formalization/)**.\n\n```bash\n./lean-dep-viz serve                     # serve at localhost:3000\n./lean-dep-viz build --output-dir site   # generate a static site\n```\n\n## Paper-to-formalization reference\n\nMaps definitions, algorithms, and results from Park et al. to their Lean\ncounterparts.\n\n### Structures and definitions\n\n| Paper | Notation | Lean | File |\n|-------|----------|------|------|\n| EOS-extended alphabet | Σ ∪ {EOS} | `ExtChar α` (abbrev `Ch α`) | `Char.lean` |\n| Finite-state automaton | (Σ, Q, q₀, δ, F) | `FSA α σ` | `Automata.lean` |\n| Finite-state transducer | (Σ, Γ, Q, q₀, δ, F) | `FST α Γ σ` | `Automata.lean` |\n| Pushdown automaton | (Σ, Π, Q, q₀, Z₀, δ, F) | `PDA Γ π σ` | `PDA.lean` |\n| Lexer specification | automaton + terminal label per class | `LexerSpec α Γ σ` | `Lexing/Base.lean` |\n| Token vocabulary | V ⊆ Σ⁺ | `Vocabulary α β` | `Vocabulary.lean` |\n| Grammar language | L(G) | `PDA.accepts` | `PDA.lean` |\n| Prefix language | prefixes of L(G) | `Language.prefixes` | `Language.lean` |\n| Single-producible terminals (Def. C.1) | Prod(q) | `FST.singleProducible q` | `Producible.lean` |\n| Realizable sequence heads (Def. 3.2) | Re | `RealizableSequenceHeads fst_comp` | `RealizableSequence.lean` |\n| Realizable terminal sequences | — | `FST.realizableSequences q` | `Automata.lean` |\n| Inverse token-spanner table (Def. 3.3) | T_inv(q, a) | `InverseTokenSpannerTable fst_comp` | `RealizableSequence.lean` |\n| Always-allowed tokens | A(q_lex, q_parse) | `PPTable` first component | `GCDAlgorithm.lean` |\n| Stack-dependent heads | D(q_lex, q_parse) | `PPTable` second component | `GCDAlgorithm.lean` |\n| Checker | C | `Checker β` | `Checker.lean` |\n| GCD target language | Lex-language of G | `TargetLanguage spec P` | `GCDCheckerLanguage.lean` |\n\n### Algorithms\n\n| Paper | Lean | File |\n|-------|------|------|\n| Alg. 1: ConstrainedDecoding | `GCDChecker spec P` | `GCDAlgorithm.lean` |\n| Alg. 2: BuildLexingFST | `BuildLexingFST spec` | `Lexing/Base.lean` |\n| Alg. 3: BuildDetokenizingFST | `BuildDetokenizingFST` | `Lexing/Detokenizing.lean` |\n| FST composition (detok ∘ lex) | `Detokenizing.BuildDetokLexer spec` | `Lexing/Detokenizing.lean` |\n| Alg. 4: BuildInverseTokenSpannerTable | `BuildInverseTokenSpannerTable fst_comp` | `RealizableSequence.lean` |\n| Alg. 5: PreprocessParser | `PreprocessParser fst_comp P` | `GCDAlgorithm.lean` |\n| Alg. 6: ComputeValidTokenMask | `ComputeValidTokenMask P itst table qa qp st` | `GCDAlgorithm.lean` |\n| Partial lexer (Lex) | `PartialLex spec` | `Lexing/Base.lean` |\n| PDA → NFA over-approximation | `PDA.toNFA` | `PDA.lean` |\n| DFS for single-producible terminals | `FST.computeSingleProducible q` | `Producible.lean` |\n\n### Propositions and theorems\n\n| Paper result | Lean | File |\n|--------------|------|------|\n| Stack invariance (Prop. 3.1) | `PDA.stackInvariance` | `PDA.lean` |\n| Over-approximation via FSA (Prop. 3.2) | `PDA.overApproximation` | `PDA.lean` |\n| Lexer-FST equivalence (Thm. C.1) | `PartialLex_to_LexingFST`, `LexingFST_to_PartialLexRel` | `Lexing/Correctness.lean` |\n| Single-producibility (Lemma C.3) | `computeSingleProducible_correct` | `Producible.lean` |\n| Valid-mask characterization | `mem_ComputeValidTokenMask_preprocess_iff` | `GCDStepProofs.lean` |\n| Soundness (Thm. C.4) | `Soundness` | `GCDStepProofs.lean` |\n| Completeness (Thm. C.5) | `Completeness`, `EOSCompleteness` | `GCDStepProofs.lean` |\n| Mask ⇒ viable continuation | `accept_if_ComputedValidTokenMask` | `GCDStepProofs.lean` |\n| checkerLanguage = target language | `GCDChecker_checkerLanguage_eq_TargetLanguage` | `GCDCheckerLanguage.lean` |\n| Checker productivity | `GCDChecker_productive` | `GCDProductivity.lean` |\n| Checker path independence | `GCDChecker_pathIndependent` | `GCDProductivity.lean` |\n| Full checker interface | `GCDChecker_correct` | `GCDProductivity.lean` |\n\n### Type parameters\n\n| Variable | Role | Paper |\n|----------|------|-------|\n| `α` | Character / input alphabet | Σ |\n| `β` | Token alphabet | V |\n| `Γ` | Terminal / output alphabet | Γ |\n| `π` | Stack alphabet | Π |\n| `σ`, `σa`, `σp` | Automaton / parser state types | Q |\n\nMost carry `FinEnum`, `DecidableEq`, or `BEq`/`LawfulBEq` instances.\n\n## License\n\nApache License 2.0. See [LICENSE](LICENSE).\n",1789154840943]