Welcome to another edition of Learning Leaps, where I’ll be sharing lessons learned from my first 16 months as a product manager at Pluralsight. Since transitioning into Product Management, one of the biggest lessons I’ve had to learn was about how to use data to make decisions related to learning products. Thats why for this week’s Learning Leaps, I’ll be taking a deeper look into how data can be used to make informed training decisions.
Data in learning (a curious past)
After spending the past 9 years designing and delivering learning experiences, I was no stranger to using data to improve learning. When I first started out in the industry, this data came in the form of “smile sheets” or feedback forms that was presented to learners at the end of a training program. This form would asks learners about their perception and reaction to the learning program that they just attended.
In general, these sheets are a great way to let learners know that we respected their opinions and wanted to know about their overall satisfaction with the learning experience. As my time in the industry went on, I found myself getting more and more frustrated with these forms because satisfaction with learning does not equal learning.
Over time, some of the companies I worked for moved beyond the simple collection of satisfaction scores and measured things like assessment scores or behavioral change. But overall, this was few and far between.
At the same time, there was an increased growth in technology platforms entering the learning industry. These platforms housed digital learning experiences and began to provide practitioners with more insights into how audiences were interacting with their experiences. It was an exciting time to be in learning!
My relationship with learning data underwent another change when I moved into learning experience design, and more recently my PM role at Pluralsight. So I decided to put together a few tips that I’ve learned over the past year that I hope will inspire others while they’re thinking about how data can help inform their learning interventions.
Tips on how to use data to make informed training decisions
Learning interventions are experiments
First and foremost, at their very core, learning interventions are experiments.
Whether it’s instructor led training, elearning, or another solution; a learning intervention is a proposed solution to help solve a performance problem inside of an organization. Ideally, it’s been proposed because there is some type of current behavior taking place that is not aligning with the desired behavior of the organization. After some type of analysis, a practitioner has determined that learning might be a potential solution to help encourage the outcomes the organization is looking for.
As learning practitioners, this means we cannot guarantee that learning will indeed be successful to deliver those outcomes. However, if we take an experimental approach to designing learning experiences we may be able to increase the likelihood that it might succeed. Taking an experimental approach means taking the time to identify the outcomes you’re hoping to implement, clarifying our hypothesis and assumptions, and determining how you’ll measure success. If you don’t determine these things from the outset, you will never be able to identify whether you’ve been successful or not.
Measure enough data to make a decision
Once you have a better idea of the outcomes you’re looking to drive, you’ll then be able to use data to help guide any decisions related to your interventions.
You can use data while you’re in discovery or identifying the problems you’re looking to solve for customers. Typically this type of data might include qualitative research like conducting surveys or observations.
You can also collect evaluative data after your solution has been released into the wild to help you assess how well it’s solving the audiences problem. Based on your overall goals, these could mean measuring things like drop-off rate, learning impact, retention rates, and more.
Whatever your reasons for measurement are, it’s important to remember that the goal of measurement in learning is to gather enough data to make a decision. It is not to collect ALL of the data you possibly can to be absolutely certain of something. If you’ve gathered enough data to make a decision and you keep collecting data, then you’ve gone too far.
Storytelling with data is a skill
Once you’ve collected some type of data is where the fun part comes in. Storytelling! Take the time to think about the story you’re looking to tell to your stakeholders. Ask yourself things like:
- What were my expectations going into this?
- Am I surprised by what the data is telling me?
- What do my stakeholders care about?
- What decisions am I trying to influence?
- Whats the best way to convey this to others?
Have this information feed into the way that you craft your story and any decisions you make based off of the data you’ve collected. Remember that anyone can collect metrics and simply report them but it takes skills to turn that data into a story that truly resonates with others.
Do you have any tips for others on how to use data to make informed training decisions? Post them in the comments below!
Be sure to check out next week’s Learning Leaps where I’ll be diving into how to use empathy to uncover the needs of your learners.