New Research from Learning Scientists at VitalSource Shows AI-Generated Practice Questions Positively Impact Student Outcomes

September 9, 2025, RALEIGH, NC – VitalSource, a leading education technology solutions provider, announced its learning scientists have published a research paper, Scaling the Doer Effect: A Replication Analysis Using AI-Generated Questions, which explores whether AI-generated practice questions could produce the same benefits as traditional human-authored questions.

This study focused on the Doer Effect, which is the learning science principle that proves students who actively engage in practice questions while reading have better learning gains. The research was conducted during a cognitive psychology course at Cal Poly Pomona in Fall 2022 and Spring 2023 semesters. Using an AI system, practice questions were generated from the course textbook and integrated into the digital learning experience. Over 200 students participated, with different policies for completing the practice questions applied in each semester.

By both increasing exam scores and replicating the Doer Effect analysis, this study validates the effectiveness of AI-generated practice questions as embedded formative practice. A key differentiator from ad hoc generative AI tools, the questions in this study were systematically created and integrated into students’ course materials through a structured AI framework ensuring relevance, quality, and alignment with the textbook content. This approach allows educators to provide content-specific study tools directly within students’ primary learning environment without the time and cost of manual question development. Significant findings from the study include:

• Engagement Matters: When students were required to complete questions before exams, they engaged with the material more effectively. They focused on relevant content, reducing the proportion of irrelevant practice attempts from 40.3% to 6.8%.
• Improved Exam Scores: Exam performance improved with the requirement of the completion of questions. Average scores increased with significant gains seen among lower-performing students.
• The Doer Effect Confirmed: The Doer Effect analysis shows a clear relationship between completing practice study questions and higher exam scores.

“This study is the first to demonstrate that the Doer Effect extends to AI-generated questions,” said Benny Johnson, Senior Director of Learning Science at VitalSource. “A 2% improvement in exam scores is a significant statistical gain especially when it consistently benefits students who need extra support. Thoughtful implementation of learning tools makes a noticeable difference in student outcomes, especially those students who need extra support.”

This is the first Doer Effect study using automatically generated questions as the practice. The results validate the utility of these questions as formative practice for students. This research finds that the doer effect as a learning science principle has been supported using AI-generated questions. The findings highlight a practical path forward for institutions seeking to responsibly harness AI to support teaching and learning at scale.

The full study can be accessed here.

# # #

About VitalSource

VitalSource Technologies, LLC has powered digital content delivery across the higher education ecosystem for over 30 years. Winner of the 2024 CODiE award for Most Innovative Ed Tech company, VitalSource delivers more than 28 million learning materials globally each year. With unmatched scale, the VitalSource Learning Delivery Network is trusted by thousands of content providers and institutions worldwide to deliver impactful learning experiences, fuel affordable access programs, and drive a learning advantage for every student. Learn more at https://get.vitalsource.com.