Collecting, analyzing, and using data from courseware are some of the most important components of an adaptive learning initiative. Faculty can use data throughout the semester to reach out to individual students who are struggling, adjust teaching, and assess student progress.
And administrators can use student learning and course outcome data to measure student success rates and identify courses that are bottlenecks for student progression.
But course learning data is less useful when it’s a blunt instrument. What if you want to employ the data to close the equity gap for particular populations of underserved students — low-income and first-generation students and students of color? How do you disaggregate student data to reveal opportunities for improvement?
Julie Neisler, Quantitative Researcher, Learning Sciences Research at Digital Promise, has been analyzing data from recent adaptive learning pilots conducted through a partnership between Every Learner Everywhere and 10 Lighthouse institutions (the first colleges and universities served by the network that are producing insight and data about implementing digital technologies). Neisler advises the following steps to extract and use insight from the data generated by adaptive learning courseware.
1. Gather early, gather often
To begin, Neisler recommends collecting existing pre-enrollment data such as academic achievement, race/ethnicity, and full-time versus part-time enrollment. She also suggests determining whether a student is Pell eligible (a proxy for coming from a low-income household) or a first-generation college student.
To do this during the pilot, Neisler worked closely with the institutional research offices on the Lighthouse campuses.
2. Compare subgroup and individual data to course data
Next, compare how individual students and groups of students do compared to the entire enrollment in the course. Which students are getting a lower passing grade, failing, withdrawing, or repeating the class?
To help lighthouse institutions answer this question, Neisler worked with Institutional Research offices to pull data on average course grades for all students as well as for groups such as Latinx and African American students, and students eligible for Pell grants. These analyses identified foundational courses with the largest gaps in course outcomes for different groups of students.
Compare success rates to changes made in your courseware. Did you make a significant change in how students move through the course? Did it have a positive impact on learning outcomes?
3. Listen for qualitative signals
You need quantitative and qualitative information to understand the whole story. Neisler says, “Look at the full picture. It’s not just the data at the end of the course, but the feedback you get from the students.”
She discovered in talking to students that they experienced increased confidence with adaptive learning courseware. The positive reinforcement the software provides after each practice activity in a course encouraged and supported them.
Qualitative information can reveal where a lower academic grade doesn’t necessarily indicate an academic problem. It might tell you that some students don’t have access to technology or that they have less prior experience with online learning technology than their peers.
Qualitative feedback might reveal that the course materials or instructor comments send subtle signals about “who belongs in this course” that discourage students from nondominant groups.
If any of those is the case, adding learning modules with more academic content to close an assumed achievement gap would be aiming at the wrong problem.
If you see large discrepancies in course outcomes for different groups of students, build in opportunities to collect qualitative data in future iterations of a course through focus groups, surveys, or observations.
4. Create targeted interventions to address gaps
If a student never had an online component to a course before, they may need extra support in an online or blended course. Likewise, you may need to address how to improve student access to technology.
“We know Pell-eligible students are less likely to have the new computers and high-speed internet connections necessary to access online courseware,” Neisler says.
“We know that institutions have resources, computer labs, and wi-fi. But if you only come to campus once a week, and you have homework assignments every couple of days, how do you get to those resources?”
5. Use data to create future improvements
Neisler says, “There’s no one thing we know will close the equity gap,” but acknowledging and better supporting the students where there are gaps is a good start. Good data analysis can inform that process.
“Break the data down into groups and examine where some didn’t do so well,” she says. “Then try to figure out what went wrong. Was it a lack of access to technology? Maybe a group didn’t come from the best high school. Maybe modules need to be revised to give more practice.”
The data you have now is your benchmark for improvement in closing equity gaps in the next semester. Neisler says it takes about three years for changes in a course to compound into a positive impact in student outcomes.
Resources for analyzing your data
Prior to the pilot project for Lighthouse institutions, Digital Promise conducted a readiness assessment that asked about success rates and how minority, Pell, and part-time students compared to peers. Neisler says it wasn’t typical to have that information: “Colleges would say, ‘We can figure that out, but that’s not something that we’ve ever looked at before.’”
Adaptive learning technology generates learning data that can put an end to unexamined assumptions. It can show if Pell students have a pass rate significantly lower than the overall rate in an essential course, for example. It can even show if particular modules or quizzes are the source of differences in course outcomes.
That allows your institution to explore and develop targeted interventions such as more technology resources or culturally responsive pedagogy.
These are some of the key components that will form the backbone of templates and guides for institutions to analyze and use their data. “We’re trying to spark this urgency that for your classes to do well, you need to disaggregate that data for the students who need the most support to see how they are faring.”
For more adaptive learning resources, visit Solve by Every Learner Everywhere.
Pamela Baker is a freelance writer specializing in education technology and higher education.