Two peers are better than one: Aggregating peer reviews for computing assignments is surprisingly accurate

Scientific peer review, open source software development, wikis, and other domains use distributed review to improve quality of created content by providing feedback to the work's creator. Distributed review is used to assess or improve the quality of a work (e.g., an article). However, it can also provide learning benefits to the participants in the review process. We developed an online review system for beginning computer programming students; it gathers multiple anonymous peer reviews to give students feedback on their programming work. We deployed the system in an introductory programming class and evaluated it in a controlled study. We find that: peer reviews are accurate compared to an accepted evaluation standard, that students prefer reviews from other students with less experience than themselves, and that participating in a peer review process results in better learning outcomes.