[{"data":1,"prerenderedAt":4},["ShallowReactive",2],{"qQ5NK3XWcy":3},"# SafeVerify\n\nThe purpose of this script is to check whether a file of submitted Lean code and/or proof matches the specifications.\nThis is safer than direct checking with the Lean compiler or REPL, because it guards against potential exploits, including manipulation of environment via metaprogramming,\nusing additional axioms, and exploitation of buggy tactics.\nCurrently it serves as the proof-checking backend of\n- [Provably-Correct Vibe Coding](http://ProvablyCorrectVibeCoding.com), a web app for vibe coding in Lean,\n- [Code with Proofs: the Arena](https://github.com/GasStationManager/CodeProofTheArena), a website for coding problems with proofs of correctness, and\n- [TheoremMarketplace](https://github.com/wadimiusz/lean-contract-interact), smart contracts for theorem bounties.\n\nThe branch `minif2f-deepseek-check` contains a version backported to Lean 4.9.0. This can be used to check [DeepSeek Prover V2's solutions to MiniF2F](https://github.com/deepseek-ai/DeepSeek-Prover-V2/tree/main). Similarly, the branch `minif2f-kimina-check` contains a version with Lean 4.15.0 that can be used to check [Kimina-Prover-Preview's and Kimina-Prover's solutions to MiniF2F](https://github.com/MoonshotAI/Kimina-Prover-Preview). The branch `abc-trinity-check` contains a version with Lean 4.20.0 that can be used to check [Trinity's autoformalization of the de Bruijin bound on the ABC conjecture](https://github.com/morph-labs/lean-abc-true-almost-always/). The branch `seed-prover-check` contains a version with Lean 4.14.0, that can be used to check [Seed Prover's published solutions, including IMO 2025](https://github.com/ByteDance-Seed/Seed-Prover/tree/main/SeedProver). SafeVerify has been used by [PutnamBench's official leaderboard](https://trishullab.github.io/PutnamBench/leaderboard.html) to verify some of the submitted solutions. \n\n\nThis is part of a broader effort to create [safe and hallucination-free coding AIs](https://gasstationmanager.github.io/ai/2024/11/04/a-proposal.html).\n\nIn more detail: the script takes two olean files, and checks whether the second file\nimplements the theorems and definitions specified in the first file.\nThe first file (the target) may contain theorem / function signatures with `sorry` in their bodies;\nthe second file is expected to fill them.\nUses `Environment.replay` to defend against manipulation of environment.\nChecks the second file's theorems to make sure they only use the three standard axioms.\n\nMost of the code is adapted from [lean4checker](https://github.com/leanprover/lean4checker/). With suggestions taken from users on [Lean Zulip](https://leanprover.zulipchat.com/).\n\n# List of checks performed by the script\n\n- For both input files, run the content of the files through `Environment.replay`.\n  - This is the same check as what `lean4checker` performs, re-checking each declaration with the kernel. Emits an exception if a declaration is not accepted by the kernel (possibly due to environment manipulation).\n  - This only replays the content of the file, not the imports. To also replay the imports, you'll need to modify the script to match what `lean4checker --fresh` does.\n- The remaining checks are done on the replayed environments of both files. This ensures that the checks are not affected by any environment manipulations\n- For each declaration from the target file, make sure a declaration of the same name, kind (def / theorem) and type is in the submission file.\n- For definitions, also check that their bodies are the same. Execpt for cases where the target file's definition depends on `sorry`, in which case the submission file's definition body is allowed to be different.\n  - This tries to capture both the case of complete definitions that are not meant to be changed, and definition stubs with sorrys that are meant to be filled.\n  - What if there is a function `g` that is complete, but in its body calls a function `f` that contains a sorry? Then function `g` also depends on `sorry` and therefore its body (not type) can be modified. If you don't want `g` to be modified, one approach is to make `g` take a function (with f's type) as input. Or use a different mechansim to denote which defs / theorems are allowed to be modified. \n- Check the submission file's definitions and theorems to make sure they only depends on the three standard axioms: `propext`, `Quot.sound`, `Classical.choice`.\n  - uses `CollectAxioms.collect`\n  - You may modify the `AllowedAxioms` list in the script to tighten or widen the set of allowed axioms.\n- For each definition in the target or submission file, if it is marked `partial` or `unsafe`, throw an exception.\n  - This requirement is perhaps more specific to the use case of verifying [solutions to coding tasks with proofs of correctness](https://github.com/GasStationManager/CodeProofTheArena). There the use of partial/unsafe functions could allow infinite loops that satisfy the type requirement.\n\nThings that SafeVerify does not check, that you may want to check via other means:\n\n- Use of keywords like `implemented_by`, `extern`, `noncomputable`: these are difficult to catch at the level of olean files which SafeVerify works in, but depending on use case you may choose to scan for and ban them at the source level. see e.g. [judge.py in CodeProofTheArena](https://github.com/GasStationManager/CodeProofTheArena/blob/main/app/services/judge.py).\n\n# Usage\n\nFirst step is to compile lean files into `.olean` files. E.g.\n```\nlake env lean -o submission.olean submission.lean\n```\nThen pass the olean files to the tool:\n```\nlake env lean --run Main.lean target.olean submission.olean\n```\n\n# Building an executable\n\n```\nlake build\n```\nwill build the script as an executable at `.lake/build/bin/safe_verify`. You can then run the executable by\n```\nlake exe safe_verify target.olean submission.olean\n```\n\n",1784558031122]