[{"data":1,"prerenderedAt":4},["ShallowReactive",2],{"ojEJSQNfGG":3},"# Specimen \nSpecimen complements the Plausible property-based testing library by automatically deriving generators, enumerators, and checkers for inductive relations.\n\nSpecimen's design is heavily inspired by [Coq/Rocq's QuickChick](https://github.com/QuickChick/QuickChick) library and the following papers:\n- *Testing Theorems, Fully Automatically* (OOPSLA 2026)\n- [*Computing Correctly with Inductive Relations* (PLDI 2022)](https://dl.acm.org/doi/10.1145/3519939.3523707)\n- [*Generating Good Generators for Inductive Relations* (POPL 2018)](https://dl.acm.org/doi/10.1145/3158133)\n\n*Specimen is a testing and verification tool* - it is designed to help find bugs during development, not to serve as a security guarantee or correctness proof for production or enterprise workloads. Intended use is development-time property-based testing, rapid prototyping of invariants, and pre-proof exploration of conjectures.\n\n## Overview\nLike QuickChick, Specimen uses the following typeclasses:\n- `Arbitrary`: unconstrained random generators for inhabitants of algebraic data types. This is imported from Plausible\n- `ArbitrarySuchThat`: constrained generators which only produce random values that satisfy a user-supplied inductive relation\n- `ArbitraryFueled`, `ArbitrarySizedSuchThat`: versions of the two typeclasses above where the generator's size parameter is made explicit (the former is imported from Plausible)\n- `Enum, EnumSuchThat, EnumSized, EnumSizedSuchThat`: Like their `Arbitrary` counterparts but for deterministic enumerators instead\n- `DecOpt`: Checkers (partial decision procedures that return `Except GenError Bool`) for inductive propositions\n\nSpecimen provides various top-level commands which automatically derive generators for Lean `inductive`s (the file [Specimen/README.md](Specimen/README.md) has more details):\n\n**1. Deriving unconstrained generators/enumerators**              \nAn *unconstrained* generator produces random inhabitants of an algebraic data type, while an unconstrained enumerator *enumerates* (deterministically) these inhabitants. \n          \nUsers can write `deriving Arbitrary` and/or `deriving Enum` after an inductive type definition, e.g.\n```lean \ninductive Foo where\n  ...\n  deriving Arbitrary, Enum\n```\nAlternatively, users can also write `deriving instance Arbitrary for T1, ..., Tn` (or `deriving instance Enum ...`) as a top-level command to derive `Arbitrary` / `Enum` instances for types `T1, ..., Tn` simultaneously. This also works for mutually recursive types:\n```lean\nmutual\n  inductive MutEven where\n    | zero : MutEven\n    | succOdd : MutOdd → MutEven\n  inductive MutOdd where\n    | succEven : MutEven → MutOdd\nend\n\nderiving instance Enum for MutEven, MutOdd\n```\n\nTo sample from a derived unconstrained generator, users can simply call `runArbitrary`, specify the type \nfor the desired generated values and provide some `Nat` to act as the generator's size parameter (`10` in the example below):\n\n```lean\n#eval runArbitrary (α := Tree) 10\n```\n\nSimilarly, to return the elements produced from a derived enumerator, users can call `runEnum` like so:\n```lean\n#eval runEnum (α := Tree) 10\n```\n\n**2. `derive_mutual` — the recommended command for constrained derivation**\n\n`derive_mutual` is the primary command for deriving constrained generators, enumerators, and checkers. It supersedes the older `derive_generator`/`derive_enumerator`/`derive_checker` commands by providing:\n- Automatic dependency discovery (derives instances for sub-relations)\n- Multi-output generation (a single hypothesis step can produce multiple existential variables)\n- True mutual recursion (multiple specs compiled into a shared `mutual` block)\n- Quality scoring and schedule search with branch-and-bound optimization\n\n**Syntax**:\n```lean\nset_option specimen.autoDeriveDeps true\nset_option specimen.multiOutput true\n\n-- Derive a constrained generator (default sort is `generator`)\nderive_mutual\n  (fun n => ∃ (t : BinaryTree), balancedTree n t)\n\n-- Derive multiple specs at once (they can call each other)\nderive_mutual\n  (fun G t => ∃ (e : term), typing G e t)\n\n-- Derive with explicit sort keywords\nderive_mutual\n  generator (fun lo hi => ∃ (t : BinaryTree), BST lo hi t),\n  checker (fun lo hi t => BST lo hi t)\n\n-- Derive an enumerator\nderive_mutual enumerator\n  (fun n => ∃ (t : BinaryTree), balancedTree n t)\n\n-- Multi-output: generate all existentials at once\nderive_mutual\n  (∃ (Γ : List type) (e : term) (τ : type), typing Γ e τ)\n```\n\nEach entry can be prefixed with `generator` (default), `enumerator`, or `checker`. When `specimen.autoDeriveDeps` is `true`, Specimen automatically discovers and derives instances for sub-relations referenced in the constructors. When `specimen.multiOutput` is `true`, the scheduler can produce multiple existential outputs in a single hypothesis step.\n\nTo sample from a generator derived via `derive_mutual`:\n```lean\n#eval runSizedGen (ArbitrarySizedSuchThat.arbitrarySizedST (fun t => balanced 5 t)) 10\n```\n\n**3. `derive_generator` / `derive_enumerator` — single-spec constrained derivation**\n\nThese commands derive a constrained generator or enumerator for a single specification. They are still supported and useful for quick one-off derivations:\n\n```lean\nderive_generator (fun n => ∃ t, balanced n t)\nderive_enumerator (fun n => ∃ t, balanced n t)\n```\n\nIn the command `derive_generator (fun x1 ... xn => ∃ x, P x1 ... x ... xn)`:\n- `P` must be an inductively defined relation\n- `x` is the value to be generated (bound by `∃`)\n- `x1 ... xn` are input parameters (bound by `fun`)\n- Multiple existential outputs are supported: `derive_generator (fun n => ∃ a b, Split n a b)`\n\nTo sample from the derived producer:\n```lean\n#eval runSizedGen (ArbitrarySizedSuchThat.arbitrarySizedST (fun t => balanced 5 t)) 10\n#eval runSizedEnum (EnumSizedSuchThat.enumSizedST (fun t => balanced 3 t)) 3\n```\n\n**4. `derive_checker` — partial decision procedures**\n\nA checker for an inductively-defined `Prop` is a `Nat -> Except GenError Bool` function, which \ntakes a `Nat` argument as fuel and returns an error if it can't decide whether the `Prop` holds (e.g. it runs out of fuel),\nand otherwise returns `ok true` / `ok false` depending on whether the `Prop` holds.\n\n```lean\nderive_checker (fun n t => balanced n t)\n```\n\n**5. Options**\n\n| Option | Default | Description |\n|--------|---------|-------------|\n| `specimen.autoDeriveDeps` | `false` | Automatically derive dependency instances for sub-relations in `derive_mutual` |\n| `specimen.multiOutput` | `false` | Allow multi-output production steps (multiple `∃` vars generated per hypothesis) |\n| `specimen.scoreType` | `\"Scoring.DefaultScore\"` | Scoring strategy for schedule quality evaluation (see below) |\n| `specimen.fuel` | `10000` | Fuel (termination budget) for derived generators/enumerators/checkers |\n| `specimen.richOutput` | `true` | Emit rich HTML widget output in the Lean infoview |\n| `specimen.textOutput` | `0` | Plain-text output verbosity (0=off, 1=summary, 2=problems, 3=full) |\n| `specimen.searchLimit` | `200000` | Max hypothesis orderings to evaluate per constructor during schedule search |\n\n**Scoring strategies** control how Specimen evaluates and selects among candidate schedules during derivation. The `specimen.scoreType` option selects the active strategy:\n\n| Strategy | Option value | Description |\n|----------|-------------|-------------|\n| Default | `\"Scoring.DefaultScore\"` | Sum of (checks, length, unconstrained) — the original heuristic. Minimizes total checking work. |\n| Worst-leaf | `\"Scoring.WorstLeafScore\"` | Takes the max (not sum) across coverage-trie leaves — penalizes worst-case input paths. |\n| Density | `\"Scoring.DensityScore\"` | Categorical density classification (Total/Partial/Backtracking/Checking) from Section 4 of *Testing Theorems, Fully Automatically*. Prefers schedules that avoid backtracking. |\n\nFor example, to use the density scoring strategy from the *Testing Theorems* paper:\n```lean\nset_option specimen.scoreType \"Scoring.DensityScore\"\nderive_mutual\n  (fun lo hi => ∃ (t : BinaryTree), BST lo hi t)\n```\n\nSee [`ScheduleQualityRegressionTest.lean`](./SpecimenTest/ScheduleQualityRegressionTest.lean) for a comparison of all three strategies on the same relation.\n\n## Repo overview\n\n**Building & compiling**:\n- To compile, run `lake build` from the top-level repository.\n- To run snapshot tests, run `lake test`.\n- To run linter checks, run `lake lint`. \n  + This invokes the linter provided via the [Batteries](https://github.com/leanprover-community/batteries/tree/main) library.\n\n**Typeclass definitions**:\n- [`ArbitrarySizedSuchThat.lean`](./Specimen/ArbitrarySizedSuchThat.lean): The `ArbitrarySuchThat` & `ArbitrarySizedSuchThat` typeclasses for constrained generators, adapted from QuickChick\n- [`DecOpt.lean`](./Specimen/DecOpt.lean): The `DecOpt` typeclass for partially decidable propositions, adapted from QuickChick\n- [`Enumerators.lean`](./Specimen/Enumerators.lean): The `Enum, EnumSized, EnumSuchThat, EnumSizedSuchThat` typeclasses for constrained & unconstrained enumeration\n\n**Combinators for generators & enumerators**:\n- [`GeneratorCombinators.lean`](./Specimen/GeneratorCombinators.lean): Extra combinators for Plausible generators (e.g. analogs of the `sized` and `frequency` combinators from Haskell QuickCheck)\n- [`EnumeratorCombinators.lean`](./Specimen/EnumeratorCombinators.lean): Combinators over enumerators \n\n**Algorithm for deriving constrained producers & checkers** (adapted from the QuickChick papers):\n- [`UnificationMonad.lean`](./Specimen/UnificationMonad.lean): The unification monad described in [*Generating Good Generators*](https://dl.acm.org/doi/10.1145/3158133)\n- [`DeriveConstrainedProducer.lean`](./Specimen/DeriveConstrainedProducer.lean): Algorithm for deriving constrained generators, including the `derive_mutual` command for multi-spec mutual derivation\n- [`MExp.lean`](./Specimen/MExp.lean): An intermediate representation for monadic expressions (`MExp`), used when compiling schedules to Lean code\n- [`MakeConstrainedProducerInstance.lean`](./Specimen/MakeConstrainedProducerInstance.lean): Auxiliary functions for creating instances of typeclasses for constrained producers (`ArbitrarySuchThat`, `EnumSuchThat`)\n- [`DeriveChecker.lean`](./Specimen/DeriveChecker.lean): Deriver for automatically deriving checkers (instances of the `DecOpt` typeclass)\n- [`Schedules.lean`](./Specimen/Schedules.lean): Type definitions for generator schedules\n- [`DeriveSchedules.lean`](./Specimen/DeriveSchedules.lean): Algorithm for deriving generator schedules\n- [`SearchTree.lean`](./Specimen/SearchTree.lean): Dependency-aware hypothesis ordering via lazy search tree with branch-and-bound pruning\n\n**Schedule scoring & quality analysis**:\n- [`Score.lean`](./Specimen/Score.lean): Type-erased score values used by the modular scoring framework\n- [`Scoring.lean`](./Specimen/Scoring.lean): Modular scoring framework with pluggable strategies (DefaultScore, WorstLeafScore, DensityScore) for evaluating schedule quality\n- [`PatternCoverage.lean`](./Specimen/PatternCoverage.lean): Pattern coverage trie that partitions the input space of an inductive relation, identifies weak spots, and annotates leaves with constructor coverage\n\n**Derivers for unconstrained producers**:\n- [`DeriveArbitrary.lean`](./Specimen/DeriveArbitrary.lean): Deriver for unconstrained generators (instances of the `Arbitrary` / `ArbitrarySized` typeclasses), including support for mutually recursive and parameterized types\n- [`DeriveEnum.lean`](./Specimen/DeriveEnum.lean): Deriver for unconstrained enumerators \n(instances of the `Enum` / `EnumSized` typeclasses), including nested and mutually recursive types\n\n**Miscellany**:\n- [`TSyntaxCombinators.lean`](./Specimen/TSyntaxCombinators.lean): Combinators over `TSyntax` for creating monadic `do`-blocks & other Lean expressions via metaprogramming\n- [`LazyList.lean`](./Specimen/LazyList.lean): Implementation of lazy lists (used for enumerators)\n- [`LazyRoseTree.lean`](./Specimen/LazyRoseTree.lean): Lazy rose tree data structure\n- [`Idents.lean`](./Specimen/Idents.lean): Utilities for dealing with identifiers / producing fresh names \n- [`Utils.lean`](./Specimen/Utils.lean): Other miscellaneous utils\n- [`Debug.lean`](./Specimen/Debug.lean): Debug tracing and option flags for Specimen\n\n### Tests\n**Overview of test corpus**:\n- The [`SpecimenTest`](./SpecimenTest/) subdirectory contains [snapshot tests](https://www.cs.cornell.edu/~asampson/blog/turnt.html) (aka [expect tests](https://blog.janestreet.com/the-joy-of-expect-tests/)) for the derivation commands. \n- Run `lake test` to check that the derived generators in [`SpecimenTest`](./SpecimenTest/) typecheck, and that the code for the derived generators match the expected output.\n- Key test directories:\n  + [`DeriveArbitrarySuchThat/`](./SpecimenTest/DeriveArbitrarySuchThat/) — constrained generators (BST, balanced tree, STLC, regex, permutations, multi-output, mutual recursion)\n  + [`DeriveEnumSuchThat/`](./SpecimenTest/DeriveEnumSuchThat/) — constrained enumerators\n  + [`DeriveDecOpt/`](./SpecimenTest/DeriveDecOpt/) — checkers\n  + [`DeriveArbitrary/`](./SpecimenTest/DeriveArbitrary/) — unconstrained generators (parameterized types, mutually recursive types, structures)\n  + [`DeriveEnum/`](./SpecimenTest/DeriveEnum/) — unconstrained enumerators (nested recursion, mutual recursion)\n  + [`CedarExample/`](./SpecimenTest/CedarExample/) — real-world application: well-typed Cedar policy expression generators\n  + [`ArithCompiler/`](./SpecimenTest/ArithCompiler/) — end-to-end example: compiler correctness testing\n\n## Security\n\nSee [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.\n\n## License\n\nThis project is licensed under the Apache-2.0 License.\n",1782661971077]