[{"data":1,"prerenderedAt":4},["ShallowReactive",2],{"l4Qza2Zisq":3},"# Lean 4 Gradescope Autograder\n\nThis project provides a Lean 4 autograder that works with [Gradescope](https://gradescope-autograders.readthedocs.io/en/latest/). \nIt checks that students have provided proof terms with the correct type or have created equal `Expr`s up to definitional equality. \nThe autograder can check theorems, functions, propositions, and instances. It *cannot* grade inductive types or structures.\n\nGrading verdicts are independently verified using [Comparator](https://github.com/leanprover/comparator): rather than trusting\nits own in-process elaboration of a submission, the autograder rebuilds the relevant declarations in a sandboxed subprocess\n(via [landrun](https://github.com/Zouuup/landrun)), re-serializes them through `lean4export`, and replays them through the\nLean kernel. To run the autograder locally (`--local`/`--test`), `landrun` and `lean4export` must be available -- either on\n`PATH`, or pointed to via the `COMPARATOR_LANDRUN`/`COMPARATOR_LEAN4EXPORT` environment variables. For local development\nwithout a real sandbox, Comparator's own `scripts/fake-landrun.sh` shim can stand in for `landrun` (no sandboxing, so don't\nuse it against untrusted submissions).\n\n## Setup \n\nMore detailed instructions can be found in the [Lean autograder shell](https://github.com/robertylewis/lean4_autograder), but here is a brief overview.\n\nFirst, set up a course repository on GitHub. Add this project, autograder-main, as a lake dependency. \nThen, use [autograder shell](https://github.com/robertylewis/lean4_autograder) to create the zip file that actually gets uploaded to Gradescope.\nThis project is meant to work with MathLib assignments, so for a good Gradescope performance, their container must have at least 2.0 CPU and 3.0GB RAM,\nand preferably the maximum resources allowed by Gradescope.\nOtherwise, you'll get inscrutable errors when it runs out of memory. \nAfter this is all set up, students will only need to submit a file to Gradescope.\n\nFor an example, see the repository [fpv2023](https://github.com/BrownCS1951x/fpv2023). \nThe lakefile of this project imports this autograder project.\nTo create a Gradescope assignment for HW1, edit the configuration file in the autograder shell \nto point at `Homeworks/Homework1.lean` in that repository, run the `make_autograder.sh` script, \nand upload the resulting zip file to Gradescope.\nStudents would then submit *only* their `Homework1.lean` file to Gradescope.\n\n## Autograding \n\nThe primary feature of the current autograder checks that *proofs* are complete. \nAn experimental feature checks the correctness of definitions.\n\n### Checking proofs\n\nThe attribute `@[autogradedProof pts]` is used to denote exercises where the student should complete a proof of a theorem statement provided by the instructor.\nThe autograder awards `pts` number of points if the student's proof is complete.\nThe solution/stencil file needs the theorem statement, but does not need a reference proof of the theorem.\n\nFor example, suppose that the stencil distributed to students contains the following code:\n```lean\n@[autogradedProof 1]\ntheorem th3 (h : ¬q → ¬p) : (p → q) := sorry\n```\nIf a student submits an assignment that replaces `sorry` with a valid proof, the autograder will grant one point.\n\nBy default, the autograder allows all and only the axioms defined in Lean core.\nExtra axioms can be allowed globally by tagging the axiom with the `@[legalAxiom]` attribute.\nYou can locally specify the axioms allowed in a solution by using the `validAxioms` attribute on that problem.\nThe autograder will only award points for the following if a student's solution does not use `Classical.choice`:\n```lean \n@[autogradedProof 1, validAxioms #[Quot.sound, propext, funext]]\ntheorem EM_of_DN_good : (∀ p : Prop, ¬¬p → p) → (∀ p : Prop, p ∨ ¬p) :=\n  sorry\n```\n\n### Checking definitions\n\n**This feature is experimental.** It should not be relied on yet.\n\nThe attribute `@[autogradedDef pts]` applies to functions, propositions, and instances, i.e. declarations whose type is not a `Prop`.\nThe correct declaration body must be provided in the solution file.\nThe autograder will try to prove that the student's definition is equal to the solution definition using `Eq.refl`, `HEq.refl`, and various tactics.\nA default list of tactics for the assignment can be set using `@[defaultTactics #[]]` over the `setDefaultTactics` function.\nIndividual problems can override the default list of tactics using the `@[validTactics #[]]` attribute. \n\nFor example, the autograder would award 2 points for a definition of `reverse` that is equal to the solution definition below.\nIt will only use `rfl` to prove the equality.\n\n```lean\n@[defaultTactics #[rfl, simp]] \ndef setDefaultTactics := () \n\n@[autogradedDef 2, validTactics #[rfl]]\ndef reverse {α : Type} : List α → List α\n  | List.nil        => List.nil\n  | List.cons x xs  => List.append (reverse xs) [x]\n```\n\n## Testing the Autograder\n\nTo run the autograder locally, build the project and run the autograder with the `--local` flag.\nThis will print the results of the autograder to the console instead of producing a JSON file.\n\n```lean\nlake exe autograder --local path/to/submission.lean path/to/solutions.lean\n```\n\n### Building a test suite\n\nMore comprehensive testing can be desirable especially during assignment development.\nTo test the autograder, build the project and run the autograder with the `--test` flag.\nThis will check the submission sheet for the `[@autograderTest status name]` attribute.\n\nThe `autograderTest` attribute is used to test multiple possible submissions to a problem\nwithout editing the master solutions file. The `status` parameter is the expected status \nof the test and should either be `passes` or `fails`. The `name` parameter is the name of \nthe problem in the solutions files. \n\nHere is an example of a solution and test file for the `reverse` function.\n\n```lean\n-- Solutions file\n@[autogradedDef 1, validTactics #[custom_simp]]\ndef reverse {α : Type} : List α → List α\n  | List.nil        => List.nil\n  | List.cons x xs  => List.append (reverse xs) [x]\n```\n\n```lean\n-- Test file\n-- Exactly the same\n@[autograderTest passes `reverse]\ndef reverse {α : Type} : List α → List α\n  | List.nil        => List.nil\n  | List.cons x xs  => List.append (reverse xs) [x]\n\n-- Different but correct implementation\n@[autograderTest passes `reverse]\ndef reverse2 {α : Type} (lst: List α) : List α := \n  lst.foldr (fun x acc => acc ++ [x]) []\n\n-- Wrong implementation\n@[autograderTest fails `reverse]\ndef reverse3 {α : Type} : List α → List α\n  | List.nil        => List.nil\n  | List.cons x xs  => List.append [x] (reverse3 xs)\n```\n",1786977007455]