[{"data":1,"prerenderedAt":4},["ShallowReactive",2],{"EGZaIBBvKu":3},"# Comparator\nComparator is a trustworthy judge for Lean proofs. It relies on having an existing Lean installation as\nwell as:\n1. [`landrun`](https://github.com/Zouuup/landrun), compiled from the `main` branch's source, present in `PATH`\n2. [`lean4export`](https://github.com/leanprover/lean4export/), at a version that is compatible with whatever Lean version your project is targeting, present in `PATH`\n3. (optional) [nanoda](https://github.com/ammkrn/nanoda_lib/), compiled with a recent version of Rust.\n   This is only necessary if you want to check with the nanoda kernel in addition to the builtin one.\n   `cargo build --release` will place `nanoda_bin` in the `target/release` directory of the checked-out directory,\n   this directory must be present in `PATH`\n\n> [!NOTE]\n> Alternatively full paths to these binaries can be specified using the environment variables\n> `COMPARATOR_LANDRUN`, `COMPARATOR_LEAN4EXPORT`, and `COMPARATOR_NANODA` when invoking Comparator.\n\nComparator is configured through a JSON file:\n```\n{\n    \"challenge_module\": \"Challenge\",\n    \"solution_module\": \"Solution\",\n    \"theorem_names\": [\"todo1\"],\n    \"permitted_axioms\": [\"propext\", \"Quot.sound\", \"Classical.choice\"],\n    \"enable_nanoda\": false\n}\n```\nWhere `Challenge.lean` contains at least a theorem named `todo1` that has a `sorry` (or any other proof)\nand `Solution.lean` is provided by a party trying to convince you that they have proven `todo1` by\nwriting out the same theorem but with a proper proof attached.\n\nGiven the following assumptions:\n1. The transitive closure of imports of `Challenge.lean` as well as `lakefile.toml`/`lakefile.lean`\n   are controlled by you or trustworthy.\n2. You have not previously tried to compile the `Solution` file or any other potentially adversarial\n   files (as that might compromise your `Challenge` file to make it seem like you are looking for a\n   different proof than you actually are)\n3. You have the `landrun` and `lean4export` binary in `PATH`\n4. `landrun` works correctly on your system and `Solution.lean` does not\n   exploit any bugs in `landrun` that allow a process to escape its sandbox\n5. The Lean kernel is correct (with `enable_nanoda` this can be reduced to \"The lean or the nanoda kernel\n   are correct\")\n6. You are not running this under a privileged user\n\nIf the following command succeeds:\n```\nsystemd-run --property=RestrictAddressFamilies=~AF_UNIX --user --pty -E PATH=\"$PATH\" --working-directory $(pwd) -- bash -c 'lake env path/to/comparator/binary path/to/config.json'\n```\n\nAll theorems in `Solution` that are listed in `theorem_names` are guaranteed to:\n1. Prove the same statement as provided in `Challenge`\n2. Use no more axioms than listed in `permitted_axioms`\n3. Be accepted by the Lean kernel\n\n> [!NOTE]\n> The Trusted Code Base of Landrun naturally includes the operating system and hardware it is running on, plus its sandboxing mechanism.\n> The systemd-run part explicitly guard against a vulnerability in landrun, Comparator's current sandboxing solution, that will be fixed in Linux 7.1\n\nNote that running `lake exe cache get` to download a Mathlib cache is acceptable before running the\ncomparator if you trust the cache to not be modified as to, e.g. contain different definitions from\nthe one you would expect.\n\nFurthermore, it is possible to avoid trusting `landrun`'s ability to sandbox the `Solution.lean` file:\nif you have obtained a fully pre-built `.lake` directory through other means and without compromising your\nchecking environment, `Solution.lean` will not be rebuilt.\n\n## Checking with Additional Kernels\nComparator currently supports checking with the [nanoda](https://github.com/ammkrn/nanoda_lib)\nkernel in addition to the builtin Lean one. To do this you need to set the `enable_nanoda` flag in\nthe JSON configuration to `true`, and the `nanoda_bin` binary must be available to comparator through\n`PATH` or the `COMPARATOR_NANODA` environment variable.\n\n## Definition Holes\nSometimes challenges want to leave open definitions for solutions to fill in. This can range from\nsimple things like filling in a `Prop` valued definition to resolve whether a conjecture is true or\nfalse, all the way to constructing complex mathematical objects. For these types of solutions,\ncomparator can guarantee that:\n1. They use no more axioms than listed in `permitted_axioms`\n2. They are accepted by the Lean kernel\n3. The name, type, universe levels and safety levels of all definition holes match\n\nCrucially, many definition hole challenges can be gamed without additional oversight.\nFor example, given a conjecture-style challenge:\n```lean\ndef ChallengeSolution : Prop := sorry\ntheorem challenge : RiemannHypothesis ↔ ChallengeSolution := sorry\n```\na solution could define `ChallengeSolution` as:\n```lean\ndef ChallengeSolution : Prop := RiemannHypothesis\n```\nand conduct a simple proof of `challenge` by reflexivity. The intention of the challenge though was\nof course to ask for a `True` or `False` value for `ChallengeSolution`. For this reason, all\ndefinition hole solutions **must** always be checked with an additional (potentially human)\nverifier.\n\nTo establish a definition hole, the challenge must provide it as a sorried definition:\n```lean\ndef large : Nat := sorry\n\ntheorem large_lt : 37 \u003C large := sorry\n```\nAll of the holes must then be put into the `definition_names` field in `configuration.json`:\n```\n{\n    \"challenge_module\": \"Challenge\",\n    \"solution_module\": \"Solution\",\n    \"theorem_names\": [\"large_lt\"],\n    \"definition_names\": [\"large\"],\n    \"permitted_axioms\": [\"propext\", \"Quot.sound\", \"Classical.choice\"],\n    \"enable_nanoda\": false\n}\n```\nFor all `definition_names`, comparator ensures that in the solution:\n- the name, type, universe levels and safety level match\n- the constant does not (transitively) refer to non-permitted axioms\n- the constant type checks\n\nThus, the following solution would be accepted:\n```lean\ndef large : Nat := 38\n\ntheorem large_lt : 37 \u003C large := by decide\n```\n\n## Development\n\nThe `scripts/fake-landrun.sh` can be used to replace Landrun in development if you are not on a Linux system that supports landrun.\n\nThe following commands, starting from the root directory of a fresh git checkout, will build and run `comparator` on one of the test examples:\n\n```sh\nlake build lean4export comparator\n\ncd tests/projects/simple_mismatch\n\ncat > lakefile.toml \u003C\u003CEOF\nname = \"comparatortest\"\nversion = \"0.1.0\"\n\n[[lean_lib]]\nname = \"Solution\"\n\n[[lean_lib]]\nname = \"Challenge\"\nEOF\n\nCOMPARATOR_LANDRUN=$(realpath ../../../scripts/fake-landrun.sh) COMPARATOR_LEAN4EXPORT=$(realpath ../../../.lake/packages/lean4export/.lake/build/bin/lean4export) lake env ../../../.lake/build/bin/comparator config.json\n```\n\nThe following commands, starting from the root directory of a fresh git checkout, will build and run the tests:\n\n```sh\nlake build lean4export comparator\nCOMPARATOR_LANDRUN=$(realpath scripts/fake-landrun.sh) COMPARATOR_LEAN4EXPORT=$(realpath .lake/packages/lean4export/.lake/build/bin/lean4export) lean --run runtests.lean\n```\n\nReplace the `landrun` and `lean4export` arguments as needed, or place the binaries in `PATH`.\n\n## Internals\nWe generally adopt a policy of not loading olean files as they just get mmaped into our address\nspace and then dereferenced and are as such a potential point of attack for sophisticated adversaries.\n\nThe comparator performs the following steps to ensure these properties:\n1. Build `Challenge` using `lake` in a `landrun` sandbox that has:\n   - read access to the entire file system and write access to `/dev`\n   - write access to the `.lake` directory of the project\n2. Run `lean4export` on the produced `Challenge.olean` in a `landrun` sandbox that has:\n   - read access to the entire file system and write access to `/dev`\n3. Repeat the same build sandboxed and export sandboxed steps with `Solution`\n4. Verify that all declarations used in the statement of all relevant theorems in `Challenge`\n   are the same as in the `Solution` environment.\n   This always includes the declarations from `Init` with special meaning to the kernel. Both `Challenge`\n   and `Solution` therefore need to import the default prelude.\n5. Verify that the body of all relevant theorems in the `Solution` environment only uses axioms\n   listed in `permitted_axioms`\n6. Replay the `Solution` environment into the Lean kernel. Doing this within the same process as the\n   comparator should be safe as the worst thing that can happen at this point is an exploit that\n   makes the kernel accept when it should reject and that same exploit should also be applicable\n   from within an external process.\n\nNote that as `Challenge` is trusted, both the sandbox and lean4export step for `Challenge` are not\nnecessary to the best of our knowledge. We still adopt these rather free measures as additional\nparanoia in case an adversary comes up with a means of attack anyway.\n\n## Acknowledgement\nComparator was originally developed by Lean FRO, with feedback from the AIMO team, in support of the\nAIMO series of competitions and their goal of enabling trustworthy LLM Lean evaluation on Kaggle.\n",1783784666994]