See All Posts
At the start of the COVID-19 pandemic in spring 2020, what had been a limited part of higher education — using a range of online and remote teaching modalities — suddenly became a common necessity, and interest in digital learning courseware and other tools grew accordingly. Some courses, including a set of pilot projects supported by Every Learner Everywhere, were using adaptive learning technologies already, but how those courses helped to close equity gaps for racially and ethnically minoritized, poverty-affected, and first-generation students wasn’t well documented.
Tracking the pilot projects at 12 colleges and universities, referred to as Lighthouse institutions, from before and during the pandemic has provided several lessons so far. Some are detailed in a new report from Digital Promise, the Every Learner Everywhere network partner focused on research and data analysis support. Supporting Student Success at the Course Level: Lessons from Change Efforts During a Pandemic documents findings such as:
- faculty using adaptive courseware were more likely to use interactive learning techniques, project-based learning, and student-centered pedagogies;
- courses in which faculty had been using adaptive courseware before the start of the pandemic were more resilient during the COVID shutdown; and
- differences in academic achievement for racially minoritized students began to narrow in the third term after implementing adaptive learning initiatives.
Supporting Student Success at the Course Level: Lessons from Change Efforts During a Pandemic is based on student-level data for 193 adaptive courseware implementations in 62 different courses experienced by 26,400 students, of whom 11,223 were taking the course at a two-year college and 15,177 at a four-year institution. It also draws on a series of surveys with instructors and students.
Supporting Student Success at the Course Level: Lessons from Change Efforts During a Pandemic includes sections on pre-pilot baseline data, how data was collected and analyzed, how the pilot projects were supported with technical assistance, and the measurable impact the pilot projects have shown. The report concludes with a discussion of significant lessons learned. A summary of those lessons is below.
1. Treat improvement of teaching and learning as a team sport
Interviews with instructors showed that faculty using adaptive courseware before the pandemic had fewer challenges than their colleagues in transitioning to remote instruction. However, solo practitioners cannot change the direction of large multi-section gateway courses, which often present barriers to equity for minoritized, poverty-affected, and first-generation students.
That’s why instructional design and the implementation of adaptive courseware should be done in a team setting, says Barbara Means, Executive Director of Learning Sciences Research at Digital Promise. “It’s very hard to create change if you’re just working faculty member by faculty member,” she explains.
“We’ve found that having groups of people work together helps keep the work going through the rough spots and sets you up to spread changes more widely. Each person brings their own networks and their own contacts, and the more you have, the more likely it is that other people are going to notice this work.”
2. Consider using incentives and enlisting external facilitators
Digital Promise found that the Lighthouse institutions benefited from engaging with external agents. Outside facilitators, consulting experts, and funders can provide a vision for change, set a cadence, facilitate professional learning activities, and provide concrete incentives such as stipends and professional development programs.
Incentives and external support to center equity bolstered the institutions’ interest in continuous, widespread improvement. Working as a group to examine the initial data generated from adaptive courseware pushed faculty toward an interest in mainstreaming its use to further refine instructional design.
“The external facilitator can take on some of the management and scheduling requirements so the project uses the time of the department chair, the dean, the provost, faculty, and staff really well,” Means says. “If you’ve provided tangible incentives and you have an agreement that says ‘this side will do this, that side will do that,’ that helps people focus on the work and keep at it in the uproar of any academic semester, especially one where a pandemic hits.”
3. Collect and analyze early indicators and course outcome data
Adaptive courseware has the potential to empower faculty to measure student progress and make effective adjustments during the term, and positive course outcomes by one instructor can push others to adapt a similar course framework, creating a collective sense of accountability.
But continuous improvement demands collecting data and having a plan to use it. Means recommends instructors seek qualitative data directly from students in the form of anonymous surveys that ask about barriers to learning, what helps promote learning, and what the students like or dislike about the software they’re using.
“That input can be very enlightening for instructors and if students see that instructors are actually using student input and modifying their behavior based on it, then students are happy to give it,” Means says. “But they have to see that the responses they give go somewhere.”
Disaggregating data in large gateway courses by section enables program-level improvements. If the average grade in one section of the same course is 2.3 while another is 3.7, that data allows faculty to work together to investigate the disparity and work toward consistent learning and assessment.
4. Stick with the improvement effort long enough for departments and programs to internalize the process
Digital Promise’s report found the proportion of faculty who feel very well prepared to implement adaptive courseware rose from 29 percent in the first term of implementation to 55 percent in the third term. The Digital Promise evaluation also showed that equity gaps start to narrow in the third term after a course redesign to use adaptive learning.
Over time the impact of these equity-centered course design efforts began to impact institutions more broadly. Anecdotally, the report notes, some colleges began to institutionalize the sharing of data. At another institution, successful implementations were built into learning management systems’ course shells for all sections of a course to use. And at another, the success in one foreign language course is being duplicated by the faculty of other languages.
“The important thing is to make time in the academic year to look at the data together and think about the improvement implications,” Means says. “It takes a while for people to get comfortable with that process and build trust. And it also takes time in some cases to have the data literacy to look at data representations and interpret them easily.”
5. Attend to specific equity issues identified through student feedback and data analysis
One impact of the COVID-19 pandemic during these pilot projects was to illuminate the necessity of implementing digital learning technologies with attention to the needs of specific student populations. A seemingly “impartial” initiative dedicated to all students risks extending or amplifying existing equity gaps.
The hypothesis that “all our students need support” doesn’t stand up to the data, according to Means. Disaggregated data may illustrate opportunity gaps that digital learning may not necessarily address or may even exacerbate.
She says this is another area where an external change agent can be helpful by showing the data and catalyzing change in situations where colleagues may avoid tough equity conversations with one another.
6. Attend to scaling within departments and institutions
Finally, the report suggested that pilot projects and individual success stories alone aren’t enough to dramatically close equity gaps for minoritized, first-generation, and poverty-impacted students. Pilot projects need to be managed with an eye toward how they will be scaled for broader impact, which isn’t always easy in postsecondary education.
“I think the secret to success is when you form these teams, there needs to be a champion for it who has the connections to help scale it,” Means says. “Their professional goals and the scaling of the intervention are aligned. Then you’re more likely to get something that really lasts and expands.”
Download Supporting Student Success at the Course Level: Lessons from Change Efforts During a Pandemic