How to Measure Educational Impact in the Digital Learning Environment
As educational institutions increasingly move to offer digital and hybrid courses, educators can take advantage of advanced analytics and assessment features to enhance the way they measure educational impact.
Click here for a guide on measuring the impact of your educational initiatives using 5 levels of data.
Using a learning management system (LMS), metrics such as performance against learning objectives, formative assessment results over time, and engagement data can be tracked automatically, which all provide glimpses into educational impact. But just tracking these metrics without first designing a plan or rubric isn't going to provide you with the insight you're looking for.
This is where Guskey's 5 Levels of Data come in.
Measuring Impact Digitally with the 5 Levels of Data
Educational reform expert Thomas R. Guskey teaches that data should form the core of any course assessment. His 5 Levels of Data builds upon Kirkpatrick's evaluation model to offer course leaders a complete guide to collecting data on their course's success.
Guskey's model suggests course leaders collect data on their course across five levels, each one building upon the last:
- Level 1: Participant Reaction—What did the student think of the learning experience?
- Level 2: Participant Learning—What did the student learn?
- Level 3: Organizational Support/Change—How did organizational attributes contribute to success?
- Level 4: New Knowledge/Skills—Did the learning make a difference to the student's professional practice?
- Level 5: Student Learning Outcomes—What was the overall impact on the students?
You can read more about Thomas Guskey's five levels of data in our white paper 5 Levels of Data for Measuring Educational Impact.
The Importance of Testing Against Traditional Models
For many organizations who are switching to digital learning environments, it can be tempting to switch everything over at once. However, this would result in missing valuable data.
A best practice is to run both traditional and digital learning environments (randomizing students between the two) so that the data can be compared. This will help organizations build an evidence-based roadmap for implementing this transition at scale and supporting their faculty effectively throughout the process.
Designing Digital Learning Strategies in Reverse
Just as with any learning system, education professionals must design their digital learning strategy according to the learning outcomes they want to achieve. Guskey explains that in the planning stages, educators must look at the levels in reverse:
- Level 5: Student Learning Outcomes—What data reflects these outcomes?
- Level 4: New Knowledge/Skills—What policies and procedures will enable these outcomes?
- Level 3: Organizational Support/Change—How should the organization support these policies?
- Level 2: Participant Learning—What skills do the teachers need to implement the digital learning environment effectively?
- Level 1: Participant Reaction—What experiences will enable the students to learn the material best?
Applying the 5 Levels of Data to Digital Learning
Guskey provides plenty of information about how the five levels of data can be collected, but his information is focused on a face-to-face learning environment. Let's take a more detailed look at each level of data and how it can be measured in a digital or hybrid learning environment.
Level 1: Participant Reaction
Level 1 is your typical end-of-course feedback questionnaire. Guskey suggests you collect data on whether students liked the experience, how the course leader performed, and if the course was well-planned.
For digital learning environments, questions should also be asked about how easy it was to access course materials, how the experience compared to a classroom experience, and if there were any particular areas of the course that would have been better-taught face-to-face.
Level 2: Participant Learning
Level 2 looks at what students learn. This is typically measured through summative assessments, which in the case of digital learning can be conducted online, but you can also assess student learning via observation, assignments, discussions, and other digital materials. It can be useful to conduct a formative assessment at the start of the course to create a baseline against which future performance can be measured.
Together, level 1 and 2 are the most basic levels of data and the easiest to measure for digital learning. Many course leaders stop at these two levels and miss out on the critical insight from the following three.
Level 3: Organizational Support/Change
Level 3 shifts the focus from the students to the organizational characteristics that may boost or hinder learning success. Even when the teaching itself is done right, these organizational elements can significantly reduce the effectiveness of a course.
This level is especially important for digital learning because the way these organizational attributes worked for face-to-face training may no longer be relevant to digital learning. Questions, which can be covered by either interviews (of both course participants and administrators) or questionnaires, should aim to answer:
- Did the online learning promote changes in line with the course objectives?
- How was individual success recognized?
- Were students granted sufficient access to course leaders?
- How were students with a differing range of technical abilities accommodated?
Level 4: New Knowledge/Skills
Level 4 seeks to answer whether the learning made a difference in the student's career. This often takes time to answer, and the student may need to be evaluated several times to get the right data. Guskey comments that non-obtrusive direct observations work best.
One way to collect this data is by tracking student performance against standards and clearly establishing the metric for success, or mastery of desired skills. Mastery can be defined as the student showing proficiency in a skill a two, three, or even four times, so a lucky first try isn't confused with learning.
Level 5: Student Learning Outcomes
Level 5 seeks to answer what impact the course had, whether it was beneficial to the students, and if the student learning outcomes were achieved. The easiest way to achieve this is by looking at results from exams and tests, but it isn't the best. Instead, educators should assess student's progress throughout the course, including both formative and summative assessments, portfolios, and growth over time.