The epitome of educational research is the random controlled study in which students are randomly assigned to a control or an experimental group. These can be implemented in online homework by giving the groups different homework assignments and then a common assessment (which might be the same three identical questions at the end of both assignments). Such random studies provide more trustworthy evidence that is less likely subject to systematic errors – especially ones unrealized by the experimenter.
DuP01 Inductive Influence of Related Quantitative and Conceptual Problems Philip Dukes and David E. Pritchard Physics Education Research Conference 2001 Part of the PER Conference Invited Paper series Rochester, New York: July 25-26, 2001 By simply reversing the order of two problems for one group, we showed that problems involving numerical or symbolic helped to similar problems as well as related conceptual problems that were administered subsequently; however doing conceptual problems first didn’t help do subsequent related quantitative ones.
WPP07 Time to Completion of Web-Based Physics Problems with Tutoring, Warnakulasooriya, R., Palazzo, D. J. and Pritchard D. E., Journal of the Experimental Analysis of Behavior 88, 103-113, 2007.
For symbolic problems, the distribution function for duration while problem was open vs. time showed three peaks: real time solvers (~ 10 min), delayed solvers (returned the following day), and quick solvers (generally < 1 minute) – a group we suspected of copying their assignments.
CCP16 Researching for better instructional methods using AB experiments in MOOCs: results and challenges Zhongzhou Chen, Christopher Chudzicki, Daniel Palumbo, Giora Alexandron, Youn-Jeng Choi, Qian Zhou, David E. Pritchard Research and Practice in Technology Enhanced Learning: December 2016, 11:9 DOI: 10.1186/s41039-016-0034-4 (See earlier work by Chudzicki et. al. Learning at Scale 2015 and Z Chen at EDM16 )
We showed that students learn faster from exercises that involve a kinesthetic component (drag and dropping) than from the same exercise expressed in a multiple choice format.
CDC17 To draw or not to draw? Examining the necessity of problem diagrams using massive open online course experiments Z Chen, N Demirci, YJ Choi, DE Pritchard Physical Review Physics Education Research 13 (1), 010110
A definitive study (of over 8000 problems) showed that adding a diagram to a physics problem helped students get it correct on their first try, but not after several attempts. This may be because 50% of students draw a diagram even if one is present in the problem, and only 10% more draw one when none is present.