Difference in learning among students doing penand-paper homework compared to web-based homework in an introductory statistics course

Abstract

A repeated crossover experiment comparing learning among students handing in pen-and-paper homework (PPH) with students handing in web-based homework (WBH) has been conducted. The system used in the experiments, the tutor-web, has been used to deliver homework problems to thousands of students in mathematics and statistics over several years. Since 2011 experimental changes have been made regarding how the system allocates items to students, how grading is done and the type of feedback provided. The experiment described here was conducted annually from 2011 to 2014. Approximately 100 students in an introductory statistics course participated each year. The main goals were to determine whether the above mentioned changes had an impact on learning as measured by test scores in addition to comparing learning among students doing PPH with students handing in WBH. The difference in learning between students doing WBH compared to PPH, measured by test scores, increased significantly from 2011 to 2014 with an effect size of 0.634. This is a strong indication that the changes made in the tutor-web have a positive impact on learning. Using the data from 2014 a significant difference in learning between WBH and PPH for 2014 was detected with an effect size of 0.416 supporting the use of WBH as a learning tool.