When I began my research partnership with Dr. Autry at Cal Poly Pomona, we did so with the aim of investigating a question of teaching and learning: how might the addition of formative practice in the etextbook impact student outcomes? The cognitive psychology textbook used in his course included free AI-generated practice questions in the ereader that appeared alongside the textbook content at frequent intervals. This partnership resulted in a study published at Learning at Scale, “Scaling the doer effect: A replication analysis using AI-generated questions.” This study offers a holistic view into the relationship between faculty course policies on student engagement with formative practice, the impact of formative practice on exam scores, and the doer effect learning science principle.

In the fall of 2022, Dr. Autry assigned 4% of the course grade to completing the practice by the end of the semester. However, we tracked student engagement and saw that a large group of students rushed to do the practice at the end of the term—and given the exams were not cumulative—this was a largely unhelpful strategy. So, in the spring of 2023, Dr. Autry made the simple change to make the practice for relevant exams due before each exam. This dramatically reduced irrelevant doing.

When reviewing student engagement data from both semesters, we realized that this course in particular satisfied the requirements for a doer effect analysis. The doer effect, established by Koedinger and colleagues at Carnegie Mellon’s Open Learning Initiative, is the learning science principle that doing practice while reading has six times the effect size on learning compared to just reading. To replicate this doer effect analysis, we need a natural learning context that includes 100+ students, high-stakes exams as a measure of learning, and variation of student formative practice usage. In addition to the other requirements, the variation in usage is not possible to orchestrate. These cognitive psychology courses met all the requirements and so we sought a post hoc IRB to investigate both the practical impact on exam scores as well as completing the doer effect analysis.

First, we did some simple analyses of aggregated exam score data and found very promising results—the spring semester had a statistically significant increase in scores by a mean of 2 points per exam (which corresponds to approximately 0.23 standard deviations). Is that a meaningful increase? Yes, absolutely. Considering that this is free, AI-generated formative practice, an increase in exam scores of any amount is beneficial. But even more meaningful was the exam score increases for students at the 25th and 50th percentile. Students at these quartiles had exam score increases that moved them from the D/C border to a solid C, and a high C to low B.

Helping struggling students achieve a move to the next letter grade could have meaningful impacts on their overall academic success. Students may struggle with the content itself or with effective study habits, so the simple yet highly effective strategy of formative practice is especially beneficial for these students. Of all the research our team has done over the years, this finding is the most meaningful to me; helping these students is why we do what we do.

We were also able to complete a correlational doer effect analysis and the results were consistent with prior research, confirming that doing practice was related to increased exam scores. Replicating the doer effect is always beneficial for the research community—as replication research is necessary to ensure the learning science research is reproducible and generalizable and also given the difficulty of meeting the conditions required to do the doer effect analysis in the first place.

Yet what makes this analysis more important is that it is the first done using AI-generated practice. This both confirms the effectiveness of AI-generated practice for student use and supports the scale at which we are deploying these questions. Providing formative practice alongside textbook content for millions of students gives all those students the opportunity to take advantage of this simple yet effective learning method.