My colleague Benny Johnson and I recently presented a paper for a special session on Adaptive Instructional Systems during the 22nd International Conference on Human-Computer Interactions. At Acrobatiq by VitalSource, Benny, director of research and development, and I, learning science specialist, develop data-driven courseware designed to help students learn. Acrobatiq originated from Carnegie Mellon University’s Open Learning Initiative (OLI) and our platform was created using cutting-edge cognitive and learning science principles. As a research-based organization, we believe it is important to contribute our own data analysis and findings to the educational technology and learning science fields.
This particular research project was very exciting for me because we were essentially looking to answer the question: does our approach to adaptivity positively impact learners? I’ll cover the highlights of this study, but you can find the full details of the paper, The impact of adaptive activities in Acrobatiq courseware: Investigating the efficacy of formative adaptive activities on learning estimates and summative assessment scores, here.
So, what did we study? We began the research process by narrowing down our overall goal into the following two research questions:
1. Do adaptive activities increase learning estimates for students?
2. Do adaptive activities increase student scores on summative assessments?
If you are unfamiliar with our platform or courseware, here are a few terminology definitions that will help you better understand these research questions and the results to follow:
-Adaptive activities are the personal practice activities delivered after a module of learning content to help students prepare for a module quiz.
-A learning estimate is a predictive measure generated by Acrobatiq’s Analytics Engine for each student on each learning objective, based on formative practice and broken into three categories (low, medium, and high).
-Formative practice are questions integrated into the lesson content that provide immediate feedback and repeated attempts without producing a grade.
-Summative assessments are the end-of-module quizzes that produce a score in the gradebook.
In essence, Acrobatiq evaluates how well a student is learning as they read through the lesson content and answer practice questions. At the end of each module of lessons, students take an adaptive activity which offers personalized, scaffolded questions against each learning objective from the preceding lessons. The goal of the adaptive activity is to prepare students for end-of-module (graded) quizzes.
What did our research find?
Acrobatiq’s adaptive activities increased learning estimates for 64% of total student interactions. Results showed that 25% of students who had a low learning estimate before the adaptive activity were able to increase their estimates to medium or high. Of the students in the medium category, nearly 40% were able to increase their learning estimates to the high category after completing the adaptive activity.
Students who were able to increase their learning estimates through the adaptive practice had higher mean scores on summative assessments than students who maintained or lowered their learning estimates. By completing the adaptive practice, students who increased their learning estimate also increased their summative assessment scores in a statistically significant way.
Analysis also showed a correlation between the amount of overall practice students did and summative scores. Students who did more practice generally had higher summative scores. This is consistent with the Doer Effect research from OLI, and we look forward to researching this further in future studies.
Finally, what does this mean for students?
-Acrobatiq’s personalized, scaffolded practice helps students increase their mastery of topics as well as their learning estimates.
-Adaptive activities helped prepare students for better performance on summative assessments. Increasing scores on quizzes has the potential to positively impact students’ overall course grades.
-By focusing on data-driven, continuously interactive design, adaptive courseware is better able to serve students at all levels in their learning process and improve learning outcomes.
