Disaggregating Learning Data to Support Equity Efforts: Resources for College and University Instructors

The data generated by adaptive learning software and other digital learning technologies will only be a useful tool in equity efforts in higher education if it can be analyzed to explore results for particular learners such as Black, Latino, Indigenous, and poverty-affected students. Disaggregating learning data helps prevent the tendency to center “colorblind” interpretations of the experiences of all students and highlights the particular experiences of minoritized learners.

Disaggregated learning data can be a powerful tool for identifying and confronting the barriers to access and progress faced by particular populations of students. It has the potential to influence how faculty work with individual students, how they adjust during a semester, and how they revise a course in subsequent semesters. Learning data is also a valuable resource for departments evaluating how effective a course with many sections and instructors is and where curricular or program-level adjustments should be considered.

Disaggregating student data to reveal opportunities for improvement for particular student populations, however, isn’t a common practice. The National Research Center for Distance Education and Technological Advancements (DETA) and WCET – The WICHE Cooperative for Educational Technologies recently conducted a research review of outcomes from education technology use in higher education and found only 17 high-quality studies that broke out data by race or ethnicity.

College and university faculty and administrators can still work to disaggregate data from digital learning technologies within their own courses and programs, however. Resources like Planning for a Blended Future: A Research-Driven Guide for Educators and Improving Critical Courses Using Digital Learning & Evidence-based Pedagogy provide practical advice and resources for aligning equity efforts with data-driven teaching practices.

Julie Neisler, Quantitative Researcher, Learning Sciences Research at Digital Promise, analyzed data from the adaptive learning pilots conducted in 2019 through a partnership between Every Learner Everywhere and 10 Lighthouse Institutions (the first colleges and universities served by the network that are producing insight and data about implementing digital technologies). Neisler advises the following steps to gather, extract, and utilize insight from learning data that centers the experiences of minoritized students.

1. Gather early, gather often

To begin, Neisler recommends collecting existing pre-enrollment data such as academic achievement, race/ethnicity, and full-time versus part-time enrollment. She also suggests determining whether a student is Pell eligible (a proxy for coming from a poverty-affected household) or a first-generation college student.

To do this during the pilot, Neisler worked closely with the institutional research offices on the Lighthouse campuses.

2. Compare subgroup and individual data to course data

Next, compare how individual students and groups of students do compared to the entire enrollment in the course. Which students are getting a lower passing grade, failing, withdrawing, or repeating the class?

To help Lighthouse Institutions answer this question, Neisler worked with institutional research offices to pull data on average course grades for all students as well as for groups such as Latino and Black students, and students eligible for Pell grants. These analyses identified gateway courses with the largest gaps in course outcomes for different groups of students.

Compare success rates to changes made in your courseware. Did you make a significant change in how students move through the course? Did it have a positive impact on learning outcomes?

3. Listen for qualitative signals

You need quantitative and qualitative information to understand the whole story. Neisler says, “Look at the full picture. It’s not just the data at the end of the course, but the feedback you get from the students.”

She discovered in talking to students that they experienced increased confidence with adaptive learning courseware. The positive reinforcement the software provides after each practice activity in a course encouraged and supported them.

Qualitative information can reveal where a lower academic grade doesn’t necessarily indicate an academic problem. It might tell you that some students don’t have access to technology or that they have less prior experience with online learning technology than their peers.

Qualitative feedback might reveal that the course materials or instructor comments send subtle signals about “who belongs in this course” that discourage students from nondominant groups.

If any of those is the case, adding learning modules with more academic content to close an assumed achievement gap would be aiming at the wrong problem.

If you see large discrepancies in course outcomes for different groups of students, build in opportunities to collect qualitative data in future iterations of a course through focus groups, surveys, or observations.

4. Create targeted interventions to address equity gaps

If a student never had an online component to a course before, they may need extra support in an online or blended course. Likewise, you may need to address how to improve student access to technology.

“We know Pell-eligible students are less likely to have the new computers and high-speed internet connections necessary to access online courseware,” Neisler says.

“We know that institutions have resources, computer labs, and wifi. But if you only come to campus once a week, and you have homework assignments every couple of days, how do you get to those resources?”

5. Use data to find and confront the next obstacle to equity

Neisler says, “There’s no one thing we know will close the equity gap,” but acknowledging and better supporting the students where there are gaps is a good start. Good data analysis can inform that process.

“Break the data down into groups and examine where some didn’t do so well,” she says. “Then try to figure out what went wrong. Was it a lack of access to technology? Maybe a group didn’t come from the best high school. Maybe modules need to be revised to give more practice.”

The data you have now is your benchmark for improvement in closing equity gaps in the next semester. Neisler says it takes about three years for changes in a course to compound into a positive impact in student outcomes.

Related reading: For a helpful discussion of continuous improvement, see pg. 10 of Optimizing High-Quality Digital Learning Experiences: A Playbook for Faculty

Resources for disaggregating and analyzing your data

Prior to the pilot project for Lighthouse Institutions, Digital Promise conducted a readiness assessment that asked about success rates and how minoritized, poverty-affected, and part-time students compared to peers. Neisler says it wasn’t typical to have that information: “Colleges would say, ‘We can figure that out, but that’s not something that we’ve ever looked at before.’”

Adaptive learning technology generates learning data that can put an end to unexamined assumptions and that decenters the so-called “average student.” It can show if Latino students have a pass rate significantly lower than the overall rate in an essential course, for example, and it can highlight which particular modules or quizzes might be the inflection point where a revised classroom practice would create a more equitable learning experience. That allows your institution to explore and develop targeted interventions such as more technology resources or culturally responsive teaching.

“We’re trying to spark this urgency that for your classes to do well, you need to disaggregate that data for the students who need the most support to see how they are faring,” Neisler says. The practices above can get a college on the path to break down learning data in a way that supports improved outcomes for minoritized and poverty-affected students.

Visit our library of comprehensive Pillar Resources

Originally published March 2020. Updated July 2021 with additional information and references.