[{"data":1,"prerenderedAt":4},["ShallowReactive",2],{"3ya987t0ok":3},"# Lean Eval\n\n**[View the leaderboard →](https://lean-lang.org/eval/)**\n\nThis repository is a comparator-based Lean benchmark for formal mathematics.\nBenchmark authors write trusted problem statements once in shared Lean modules, and the\ntooling generates one comparator workspace per problem under `generated/`.\n\nA submission is scored entirely by comparator results: a problem counts as solved iff\ncomparator accepts the submitted solution.\n\nThe main user-facing entrypoint is:\n\n```bash\nlake exe lean-eval --help\n```\n\n## Quick Start For Benchmark Problem Contributors\n\nUse this path if you are adding or editing benchmark problems.\n\n### 1. Install and fetch dependencies\n\n```bash\nlake exe cache get\nlake build\n```\n\n### 2. Add or edit a trusted theorem\n\nPut the statement in one of the shared modules under `LeanEval/` and mark it with\n`@[eval_problem]`.\n\n```lean\n@[eval_problem]\ntheorem my_new_problem : ... := by\n  sorry\n```\n\nCurrent source modules live in topic folders such as:\n\n- `LeanEval/NumberTheory/`\n- `LeanEval/Topology/`\n- `LeanEval/ComplexAnalysis/`\n- `LeanEval/EasyProblems.lean`\n\n### 3. Add the manifest entry\n\nEach tagged declaration must be listed by exactly one file under\n[`manifests/problems/`](manifests/problems/). One file per problem,\nnamed `\u003Cid>.toml`, with top-level keys (no `[[problem]]` wrapper). The\nfilename stem must match the `id` field. The `holes` array names every\n`@[eval_problem]`-tagged declaration in the module that the problem\nowns; for the common single-theorem case it has one element.\n\n```toml\n# manifests/problems/my_new_problem.toml\nid = \"my_new_problem\"\ntitle = \"My new problem\"\ntest = false\nmodule = \"LeanEval.SomeModule\"\nholes = [\"my_new_problem\"]\nsubmitter = \"Your Name\"\nnotes = \"Optional notes.\"\nsource = \"Optional citation or URL.\"\ninformal_solution = \"Optional proof sketch or reference.\"\n```\n\nThe required fields are:\n\n- `id` (must equal the filename stem)\n- `title`\n- `test`\n- `module`\n- `holes`\n- `submitter`\n\nThe one-file-per-problem layout means two PRs adding distinct problems\nnever conflict on the manifest.\n\n#### Multi-hole problems\n\nA problem may bundle several `def`s, `instance`s and `theorem`s — list\nthem all in `holes`. Comparator then asks the participant to fill every\nlisted declaration in their `Submission.lean`. Two conventions:\n\n- **Name every instance.** The generator addresses holes by their\n  declaration name, and Lean's auto-generated names for anonymous\n  `instance`s (e.g. `instTopologicalSpaceJacobian`) are not stable. Use\n  `instance instAddCommGroup : ... := sorry` rather than `instance : ...\n  := sorry`.\n- **Use `holes` even for one declaration.** There is no `theorem = \"...\"`\n  shorthand; a singleton hole is just `holes = [\"my_thm\"]`.\n\nSee [`LeanEval/Sandbox/DefHoleExample.lean`](LeanEval/Sandbox/DefHoleExample.lean)\nand [`LeanEval/Sandbox/InstanceHoleExample.lean`](LeanEval/Sandbox/InstanceHoleExample.lean)\nfor the smallest working examples, or\n[`LeanEval/Geometry/JacobianChallenge.lean`](LeanEval/Geometry/JacobianChallenge.lean)\nfor a realistic multi-hole problem.\n\n### 4. Validate the authored source of truth (optional)\n\n```bash\nlake exe lean-eval validate-manifest\nlake exe lean-eval check-problem-build\n```\n\n`validate-manifest` checks that `@[eval_problem]` declarations and manifest entries\nmatch. `check-problem-build` builds the problem modules so warning-producing Lean changes\ndo not slip through. Both are cheap and catch the most common mistakes before a CI\nroundtrip.\n\n### 5. Open a PR\n\nThat's it — push your branch and open a PR. CI regenerates the comparator workspaces\nunder `generated/` and verifies they build, so you do not need to commit anything\nunder `generated/` yourself. Once the PR merges, a separate workflow regenerates\n`generated/` on `main` and pushes the result.\n\nIf CI fails with a generation or build error, you'll need to fix the source. The\nfastest local equivalent is:\n\n```bash\nlake exe lean-eval generate --problem my_new_problem\nlake exe lean-eval check-generated-builds --problem my_new_problem\n```\n\n## Quick Start For Solvers\n\nUse this path if you want to prove benchmark problems without touching the trusted\nbenchmark files.\n\n### 1. Pick a problem\n\nGenerated workspaces live under `generated/`, one directory per problem. The current\ncatalog is summarized in [`generated/index.json`](/home/kim/lean-evals/generated/index.json).\n\nExamples:\n\n- `generated/two_plus_two/`\n- `generated/list_append_singleton_length/`\n- `generated/cyclotomic_integer_house_le_two/`\n\n### 2. Create your local workspace\n\nCopy a clean starter workspace:\n\n```bash\nlake exe lean-eval start-problem two_plus_two\n```\n\nThat creates `workspaces/two_plus_two/` by default. You can also pass a destination:\n\n```bash\nlake exe lean-eval start-problem two_plus_two /tmp/two_plus_two\n```\n\n### 3. Install workspace dependencies\n\n```bash\ncd workspaces/two_plus_two\nlake update\n```\n\n### 4. Write your proof\n\nSolver-owned files are:\n\n- `Submission.lean`\n- `Submission/Helpers.lean`\n- any additional Lean files you add under `Submission/`\n\nTrusted files you should not edit in the normal solver workflow are:\n\n- `Challenge.lean`\n- `Solution.lean`\n- `config.json`\n- `lakefile.toml`\n\n`Challenge.lean` contains the benchmark statement. `Solution.lean` is the fixed bridge\nthat tells comparator to check your theorem from the `Submission` namespace.\n\n### 5. Run comparator locally\n\n```bash\nlake test\n```\n\n`lake test` shells out to three external tools that you must install yourself:\n`landrun` (the sandbox), `lean4export` (exports oleans to text), and `comparator`\n(the verifier). All three are pinned to immutable commits; the authoritative pin\ntable lives in [`SECURITY.md`](SECURITY.md) (\"Trusted dependencies and pin\npolicy\"), and CI installs exactly these in\n[`.github/workflows/ci.yml`](.github/workflows/ci.yml). Reproduce that setup\nlocally:\n\n```bash\n# landrun — install at the pinned commit (NOT @main; the version string is\n# unreliable, and tagged releases through v0.1.15 lack fixes comparator needs).\ngo install github.com/zouuup/landrun/cmd/landrun@5ed4a3db3a4ad930d577215c6b9abaa19df7f99f\nexport PATH=\"$(go env GOPATH)/bin:$PATH\"\n\n# lean4export — clone, check out the pin, and build.\ngit clone https://github.com/leanprover/lean4export.git\n( cd lean4export\n  git checkout 3de59f10bc4b4a0f2de698597aeb1246caa0df0a\n  lake build lean4export )\nexport PATH=\"$PWD/lean4export/.lake/build/bin:$PATH\"\n\n# comparator — clone, check out the pin (adds `def`-hole support), and build.\ngit clone https://github.com/leanprover/comparator.git\n( cd comparator\n  git checkout 71b52ec29e06d4b7d882726553b1ceb99a2499e0\n  lake build comparator )\nexport PATH=\"$PWD/comparator/.lake/build/bin:$PATH\"\n```\n\n`lean4export` and `comparator` are Lean programs: `lake build` compiles each with\nthe Lean toolchain pinned in *its own* `lean-toolchain` at the commit above (the\nv4.32.0-rc1 toolchain). This must match the toolchain that builds the workspace,\nbecause comparator builds `Challenge.olean` with the workspace toolchain and then\nreads it back with `lean4export`. If the two differ you get\n`failed to read file '.../Challenge.olean', incompatible header` — that error\nmeans a Lean/olean version mismatch (or a stale `.lake` artifact left over from\nan earlier toolchain), never a problem with your proof. If you hit it, rebuild\n`lean4export` (and `comparator`) at the pinned commits with the v4.32.0-rc1 toolchain\nrather than your `elan` default, and clear the affected workspace's `.lake/build`\nbefore retrying.\n\nOnce the tools are on your `PATH`, verify the whole pipeline against the starter\nproblem before attempting a real one — this builds and scores `two_plus_two` end\nto end, so it isolates an install problem from a proof problem:\n\n```bash\nlake exe lean-eval check-comparator-installation\n```\n\nIf you keep the comparator binary somewhere off your `PATH`, point `lake test` at\nit explicitly (`landrun` and `lean4export` must still be on `PATH`, since\ncomparator invokes them):\n\n```bash\nCOMPARATOR_BIN=/path/to/comparator lake test\n```\n\n### 6. Check your local score\n\nFrom the repo root:\n\n```bash\nlake exe lean-eval run-eval\nlake exe lean-eval run-eval --json\n```\n\nThe scorer prefers `workspaces/\u003Cproblem-id>/` when present and falls back to\n`generated/\u003Cproblem-id>/` otherwise.\n\n## Submission Rules\n\nTo **submit a solution** to the public leaderboard, open a submission issue on\nthe submissions repository:\n\n> **[github.com/leanprover/lean-eval-submissions](https://github.com/leanprover/lean-eval-submissions)**\n\nThat repository owns the hosted submission pipeline and the stored results.\nThis repository (`leanprover/lean-eval`) holds only the problem set and the\ncomparator/sandbox integration.\n\nParticipants may use Mathlib freely.\n\nIf a proof needs helper code that is not already in Mathlib, that code must be included\ninside the submission workspace itself. Multi-file submissions are allowed through\n`Submission.lean` and extra local modules under `Submission/`.\n\nFor benchmark-repo submissions (a PR that edits a `generated/` workspace in place),\nvalidate changed paths with:\n\n```bash\nlake exe lean-eval validate-submission --file generated/two_plus_two/Solution.lean\n```\n\nThe current validator accepts:\n\n- modifications to `generated/\u003Cproblem-id>/Solution.lean` and\n  `generated/\u003Cproblem-id>/Submission.lean`\n- additions, modifications, deletions, renames, or copies of `.lean` files under\n  `generated/\u003Cproblem-id>/Submission/`\n- additions (only) of markdown files anywhere inside\n  `generated/\u003Cproblem-id>/`, other than the generated `README.md`\n- additions (only) of a top-level `generated/\u003Cproblem-id>/LICENCE` or\n  `generated/\u003Cproblem-id>/LICENSE` file\n\nIn practice, solvers should normally work in `Submission.lean` and `Submission/`.\n\n## Repository Layout\n\n- [`LeanEval/`](/home/kim/lean-evals/LeanEval): trusted authored problem statements\n- [`manifests/problems/`](manifests/problems/): one TOML file per problem, named `\u003Cid>.toml`\n- [`generated/`](/home/kim/lean-evals/generated): generated comparator workspaces\n- [`scripts/`](/home/kim/lean-evals/scripts): generation, validation, and scoring helpers\n- [`PLAN.md`](/home/kim/lean-evals/PLAN.md): deferred design and roadmap notes\n\n## End-To-End Repo Checks\n\nFor a local health pass over the repository:\n\n```bash\nlake exe lean-eval validate-manifest\nlake exe lean-eval check-problem-build\nlake exe lean-eval generate --check\nlake exe lean-eval check-generated-builds\nlake exe lean-eval run-eval\n```\n\nThere is also an end-to-end workflow self-check:\n\n```bash\nlake exe lean-eval check-eval-workflow\n```\n",1782661956728]