Learning Leaps Part 5: Incorporating learning strategy while building products

Welcome to another edition of Learning Leaps, where I’ll be sharing lessons learned from my first 16 months as a product manager at Pluralsight. 

Since transitioning into Product Management, one of the biggest lessons I’ve had to learn was about the difference between learning strategy and product strategy. Thats why for this final edition of Learning Leaps, I’ll be taking a deeper look into the need for learning strategy to be incorporated in technology products.

Whats in a strategy anyway?

Many organizations, especially those in the technology space, focus on the importance of product strategies to help drive their decision making. For those unfamiliar, a product strategy can be defined as a set of decisions and priorities that a company focuses on in order to achieve a shared vision that wins with the users of its products (thanks to my good friend Jess Kadar for this concise definition!).

During my time in the industry, I’ve sat through my fair share of product strategy sessions. These meetings would usually include some sort of discussion about the company mission and vision, OKRs, and even product roadmaps. These sessions were great exercises for the company to illustrate why they were focusing on specific areas of investment. The outcome of the meetings would often empower product teams to drive the specific areas that they oversaw. Despite this, the more I sat in on these discussions, the more I noticed learning strategy not being considered during the creation of technology products (even when creating learning products).

For those unfamiliar, learning strategy can be defined as a set of decisions, techniques, procedures, and processes that learning creators can use to promote desired learning outcomes. As a learning strategist, I’m often thinking about the learning outcomes I’m looking to drive and the best way to deliver them. In other words, I consider things like learner goals and objectives so that I can determine the types of content or activities to help influence a learners knowledge, skills, and behavior.

Learning and product strategy are different yet intertwined

Over the years, the general theme I noticed in the industry is that product strategy and design are often driven by market and customer needs while learning strategy is driven by theory and research (Note: this is an oversimplification and I’m happy to dive deeper with anyone who may be discussing more of the intricacies).

Theres a few problems with this approach. First, as Clark Quinn mentions in Where is the Learning Science in Technology Products?, there is a documented disconnect between what learners think is good for their learning and what actually works. This is the same way that customers will ask for things that they think they want but it still may not get to the root cause of what they actually need.

Second, this approach leads me to think about the ancient qualitative vs. quantitative research debate that happens in product and UX communities. It should not be one or the other but rather a blending of the two or a mixed-methods approach.

What i’m recommending is that learning strategy should be considered an equally as important apart of a business as product strategy. If you look into the market, the best learning and educational products have a solid foundation of learning strategy. To put it succinctly, if you’re building a product to help people learn, you have to know how they learn.

Balancing customer needs with learning science

So whats the best way to balance your customer needs with learning strategy? Heres a few tips to get you started:

Connect with people who have learning expertise

If you don’t have a background in learning, thats okay! You can connect with folks who already have some inside of your organization. These individuals might be sitting on your learning and development, UX, or even product teams. They’ll be familiar with learning theories and models that could help to influence the success of your learning product. Use them!

Incorporate learning research while making product decisions

As a PM with a background in learning, I often lead and conduct my own learning research alongside the product discovery process. This often means leading mixed method research which include surveys, conducting user interviews, or prototype testing. At the same time if I have a specific research question in mind, I’ll look at existing learning research to see what it was says. I’ll incorporate this research into any synthesis and consider it when making any product decisions with my team.

Build out a center of learning research

Depending upon the size of your learning product and organization, it might be worth investing in a center of learning research. Many of the larger learning companies in the industry such as Pearson, edX, and Houghton Mifflin Harcourt have done just this. These centers often focus on staying up to date with the latest learning research and techniques in addition to conducting efficacy studies among customers to prove that learning outcomes are indeed happening.

Do you have any tips on how technology platforms can incorporate more learning science into the experiences they’re building? Post them in the comments below!

Learning Leaps Part 4: Using empathy to uncover the needs of your learners

Welcome to another edition of Learning Leaps, where I’ll be sharing lessons learned from my first 16 months as a product manager at Pluralsight.

While transitioning into a Product Manager role, one of the biggest responsibilities of my role became leading my team through a human-centered design process in order to deliver meaningful learning experiences to our customers.

That’s why for this weeks Learning Leaps, I’ll be taking a deeper look into how learning practitioners can use empathy to help uncover the needs of their learners.

A human-centered approach to learning

After spending the past 9 years designing and delivering learning experiences, I was no stranger to the need to connect to learners in order identify their needs. Historically, I spent a lot of time using the ADDIE framework (analyze, design, develop, implement, and evaluate) to create the learning that I was creating. This all came to a head during 2016, when I transitioned to the user experience team at the company I was working for.

My time on the team exposed me to using a human-centered approach to solving problems. For those unfamiliar, human-centered design is an approach that focuses on building a deep empathy with the people you’re creating for, using brainstorming and ideation to identify possible solutions, sharing your ideas with your audience to get their feedback, and then eventually pushing your solution out into the world.

By incorporating this approach into the way that I was already working, I was able to iterate quicker and truly connect to the needs of the learners I was serving. Since moving into Product Management at Pluralsight, I’ve seen the true power of human-centered design in action. Part of my daily role is leading a team through the human-centered design process in order to deliver impactful experiences to our learners. I’ve become such a huge advocate for this framework that I wanted to share some tips and tricks for how other practitioners can incorporate this framework into how they design and deliver learning experiences.

Tips to use empathy while creating learning experiences

Identify your purpose for connecting with learners

As I shared during the last Learning Leaps article, learning interventions are proposed solutions to help solve performance problems inside of an organization. Thats why before you jump into meeting directly with learners, you’ll want to first take a step back to determine the outcomes you’re hoping for.

Some questions you may want to ask yourself are:

  • What is the current performance taking place? What is the ideal performance the organization is looking for?
  • What are my learners backgrounds?
  • What am I looking to find out by connecting with learners?
  • What decisions am I looking to make with the information that I find out?
  • How many learners do I need to connect with in order to get an accurate picture of what is happening?

Determine the best way to connect with your learners

Once you have your intentions nailed down, you can start to identify questions that you’d like to ask learners and determine the best way to connect with them.

Where I am in the discovery process, will usually impact the type of method I might use to gain empathy with my learners. For example, if i’m trying to identify the problems I’m looking to solve for customers, I’ll usually use some type of qualitative research method like sending out surveys, conducting interviews, or focus groups. If i’m trying to gain a better idea of how a learning solution is already performing, I might use some type of quantitative measure.

Below are some things I keep in mind when choosing the best way to connect with learners:

SurveysSurveys can sometimes get a bad rap because of overuse and poorly written questions.

They’re a great tool if you’re looking to collect data from a large group of people or identify trends over time.

I’ll often send out a survey to identify general themes from my audience and help narrow down participants that I may want to interview in person.
InterviewsInterviews are a great way to meet directly with learners and ask them about their needs, motivations, and preferences.

It can take some time to recruit, conduct, and synthesize information from interviews so be sure you schedule enough time in your process.
Focus GroupsFocus groups can be a helpful tool if you’re looking to facilitate a group discussion around the area you’re exploring.

As learning practitioners, we often conduct focus groups without even realizing it. It could be in the form of a group discussion or debrief at the end of a course or training session.

In any case, they’re a helpful way of getting feedback from multiple learners at the same time. Just watch out for any bias that this may cause!

Regardless of the method you choose, connecting with your learners doesn’t have to be a labor-intensive process. Connecting with only 5-7 learners can help you to notice themes and trends that may be occurring.

Practice mindful communication

Once you’re able to connect directly with learners, remember to practice mindful communication. This might include being mindful of asking non-leading questions, asking open ended questions, embracing silence, and keeping your reactions neutral when learners are responding to you.

Synthesis always takes longer than you think it will

Finally, once you’re done speaking with your learners, you’ll want to synthesize your findings into a way that is manageable for you and your team. I find that incorporating stakeholders in your synthesis process is a great way to get buy-in from them when you’re designing solutions later on.

One general rule of thumb when conducting synthesis is that it always takes longer than you think it’s going to. I’ll usually try to schedule a block of time on my teams calendar for the session.

Some questions you may want during the session might include:

  • What were our expectations going into this?
  • Are we surprised by what learners told us?
  • How does what we learned impact decisions related to our experience?
  • Whats the best way to communicate what we learned to others?

Do you have any tips for others on how to use empathy to meet the needs of their learners? Post them in the comments below!

Be sure to check out the final edition of Learning Leaps where we’ll be diving the need for learning science in technology products offered today.

Learning Leaps Part 3: Using data to make informed training decisions

Welcome to another edition of Learning Leaps, where I’ll be sharing lessons learned from my first 16 months as a product manager at Pluralsight. Since transitioning into Product Management, one of the biggest lessons I’ve had to learn was about how to use data to make decisions related to learning products. Thats why for this week’s Learning Leaps, I’ll be taking a deeper look into how data can be used to make informed training decisions.

Data in learning (a curious past)

After spending the past 9 years designing and delivering learning experiences, I was no stranger to using data to improve learning. When I first started out in the industry, this data came in the form of “smile sheets” or feedback forms that was presented to learners at the end of a training program. This form would asks learners about their perception and reaction to the learning program that they just attended.

In general, these sheets are a great way to let learners know that we respected their opinions and wanted to know about their overall satisfaction with the learning experience. As my time in the industry went on, I found myself getting more and more frustrated with these forms because satisfaction with learning does not equal learning.

Over time, some of the companies I worked for moved beyond the simple collection of satisfaction scores and measured things like assessment scores or behavioral change. But overall, this was few and far between.

At the same time, there was an increased growth in technology platforms entering the learning industry. These platforms housed digital learning experiences and began to provide practitioners with more insights into how audiences were interacting with their experiences. It was an exciting time to be in learning!

My relationship with learning data underwent another change when I moved into learning experience design, and more recently my PM role at Pluralsight. So I decided to put together a few tips that I’ve learned over the past year that I hope will inspire others while they’re thinking about how data can help inform their learning interventions.

Tips on how to use data to make informed training decisions

Learning interventions are experiments

First and foremost, at their very core, learning interventions are experiments.

Whether it’s instructor led training, elearning, or another solution; a learning intervention is a proposed solution to help solve a performance problem inside of an organization. Ideally, it’s been proposed because there is some type of current behavior taking place that is not aligning with the desired behavior of the organization. After some type of analysis, a practitioner has determined that learning might be a potential solution to help encourage the outcomes the organization is looking for.

As learning practitioners, this means we cannot guarantee that learning will indeed be successful to deliver those outcomes. However, if we take an experimental approach to designing learning experiences we may be able to increase the likelihood that it might succeed. Taking an experimental approach means taking the time to identify the outcomes you’re hoping to implement, clarifying our hypothesis and assumptions, and determining how you’ll measure success. If you don’t determine these things from the outset, you will never be able to identify whether you’ve been successful or not.

Measure enough data to make a decision

Once you have a better idea of the outcomes you’re looking to drive, you’ll then be able to use data to help guide any decisions related to your interventions.

You can use data while you’re in discovery or identifying the problems you’re looking to solve for customers. Typically this type of data might include qualitative research like conducting surveys or observations.

You can also collect evaluative data after your solution has been released into the wild to help you assess how well it’s solving the audiences problem. Based on your overall goals, these could mean measuring things like drop-off rate, learning impact, retention rates, and more.

Whatever your reasons for measurement are, it’s important to remember that the goal of measurement in learning is to gather enough data to make a decision. It is not to collect ALL of the data you possibly can to be absolutely certain of something. If you’ve gathered enough data to make a decision and you keep collecting data, then you’ve gone too far.

Storytelling with data is a skill

Once you’ve collected some type of data is where the fun part comes in. Storytelling! Take the time to think about the story you’re looking to tell to your stakeholders. Ask yourself things like:

  • What were my expectations going into this?
  • Am I surprised by what the data is telling me?
  • What do my stakeholders care about?
  • What decisions am I trying to influence?
  • Whats the best way to convey this to others?

Have this information feed into the way that you craft your story and any decisions you make based off of the data you’ve collected. Remember that anyone can collect metrics and simply report them but it takes skills to turn that data into a story that truly resonates with others.

Do you have any tips for others on how to use data to make informed training decisions? Post them in the comments below!

Be sure to check out next week’s Learning Leaps where I’ll be diving into how to use empathy to uncover the needs of your learners.

Learning Leaps Part 2: What collaboration looks like when creating learning products

Welcome to another edition of Learning Leaps, where I’ll be sharing lessons learned from my first 16 months as a product manager at Pluralsight. While transitioning into a Product Manager role, one of the biggest lessons I’ve had to learn was about facilitating effective collaboration while creating learning experiences. That’s why for this weeks Learning Leaps, I’ll be taking a deeper look to identify what collaboration looks like and provide some tips to help get you started!

Moving from IC to Product Manager

After spending the past 8 years designing and delivering learning experiences, I was no stranger to collaboration on the job. In all of my previous roles, I was always an individual contributor on a cross-functional team. I had enjoyed this type of role and had done well with my approach to collaboration.

Soon after transitioning into my PM role at Pluralsight, I had a huge wakeup call that my existing approach to collaboration needed to change and FAST!

Product Management is collaboration. As a Product Manager, you are driving the product you’re responsible for. This means that you’re frequently coordinating and collaborating with all of the stakeholders who touch your product. So rather than being an individual contributor on a cross-functional team, you’re the one driving the initiatives and overall decision making related to your product line.

This type of collaboration is a skill that is learned and perfected by many Product Managers over time. So I decided to put together a few tricks that I’ve learned over the past year that I hope will help others while they’re collaborating on learning experiences.

Tips for effective collaboration when creating learning experiences and products

Identify and build trust with your stakeholders

A week before joining the team at Pluralsight, I fell down the stairs and broke my ankle. This put a huge damper on the onboarding plan I had created for myself. Since I am the absolute epitome of an introvert, I knew I was going to have to make an extra effort to meet everyone who I would be working with.

Over my first month on the job, I ended up having 30+ virtual 1:1 sessions. I did them in order of priority; starting with my immediate team including developers and UX designer. I put together questions that would help me learn more about their experiences inside of the company and in their roles. By taking the time to learn more about my team and stakeholders, I was able to gain empathy about the challenges they had to overcome on a daily basis. This gave me insight on things I could do to help make their lives easier in our work together.

After the official meet and greets were done, I made sure to put reoccurring meetings with stakeholders on the calendar so I’d never have to think twice about who to talk to and when. To this day, I’m still discovering people that would be great to connect with or touch the products and initiatives i’m working on.

Define the outcomes you’re looking to drive

Once you have a better idea of who you’ll be working with, you’ll want to identify the outcomes you’re looking to drive in your work together. Whenever I kick off a new project or initiative, I’ll usually schedule a meeting for everyone to come together and chat about the outcomes we’re hoping for and why. This alignment meeting makes sure that everyone starts out with the same context and helps us to be more effective in our work together.

Define roles for everyone on the team

When working on a cross functional team, it’s important to remember that everyone brings unique strengths and perspectives to the table. It is through your work together you’re able to deliver better outcomes than one person would be able to individually. Collaboration works best if everyone has an idea of what is being expected of them. Whoever the project leader is (in my case since I was the PM, it was me), will want to make sure everyones roles are clear from the very beginning so there is no confusion as you deliver on your mission.

Communicate early and often

Once you have your stakeholder group figured out, you’ll want to figure out the best way to work together. Since Pluralsight has multiple offices around the world, much of our work is done asynchronously. Depending upon the size of the project I’m working on, i’ll often spin up a slack channel for everyone to communicate and share insights with one another.

I’ll try to limit scheduled meetings for major project milestones like brainstorming, sharing user research synthesis, or discussing priorities for a coming year or quarter. As a Product Manager, I’m usually deep in the weeds of the problems I’m involved with, while others on the team may jump in and out as their schedule allows. Because of this, i’ll also try to over-communicate as much as possible to ensure others can follow along with things as they’re unfolding.

Show your work!

Prior to joining Pluralsight, I had always worked inside companies that were smaller in size. For context, when I joined The Predictive Index, the company had around 30 employees and when I left it had reached 100. Pluralsight on the other hand, is around 1400+ people currently. It goes without being said, collaborating inside of a 100 person company is drastically different than collaborating inside of a 1400 person company.

At a smaller size company, collaboration meant coordinating with maybe 8-10 stakeholders. During a recent project I was working on I had over 30+ stakeholders I had to coordinate with.

While chatting with my teammate Patrick, he noted how important showing your work was inside of larger organizations. He compared it to showing your work during math class. I vividly remember my math teacher trying to drill into my head the need to document my work as I progressed through problems (as a child I hated this activity).

I use a similar approach today for my projects inside of Pluralsight. I use project documents to highlight the outcomes were looking to drive, hypothesis we have, links to designs, experiment ideas, and decisions about future product strategy. This approach allows others to follow along with decisions and how they were made. As much as I despise forcing myself to slow down and document my work in this way, I’ve found that it’s made me more strategic about what i’m communicating and why. It’s also made it easier for me to get buy-in, influence, and manage others.

Seek context with intention

One of the core values that Pluralsight encourages amongst all of it’s employees is to seek context with intention. I’m a huge believer that as children we were driven by our curiosity and as we entered adulthood it was beaten out of us little by little. Thats why this value is my absolute favorite and I utilize it all of the time when working with others.

When working teams, it is not uncommon for everyone to have different viewpoints because of the vantage point they have in their role. I’ll frequently question why someone has a particular opinion, ask why they did something a certain way, what their thought process were and why. By asking questions, i’m able to learn more about the constraints and possibilities of a project.

Do you have any tips for others on how to collaborate more effectively? Post them in the comments below!

The next Learning Leaps will resume in 2020, where we’ll be dialing in on how to use data to make informed training decisions.

Introducing Learning Leaps: Lessons learned while creating learning products

A little over a year ago, I decided to take a huge leap in my career. After spending nearly 7 years in the learning industry, I decided to make the transition into a formal Product Management role.

The decision to move into Product Management was strategic on my part. I had spent the past 8 years designing and delivering learning experiences that were offered as a product or service for the audiences I serving. As time went on, I found myself adopting an iterative approach to learning that was rooted in learning experience design and research. I identified the overlap between the instructional design and product development processes and how they were both focused on identifying and solving problems. Over time, I desired more and found myself inching towards the world of user experience and product management.

So during August 2018, I decided to finally take the leap and switch roles. With the transition, I knew I wanted to focus on gaining the skills and expertise that wouldn’t have been afforded to me in a traditional learning role. These included:

  • deeper knowledge and experience with the inner workings of the business
  • setting the vision and strategic direction of a learning product
  • making data-informed decisions including conducting discovery research and creating hypothesis tests for experimentation
  • and inspiring and leading others

I’ve been in Product Management for 16 months now and I can say that it’s been one of the most difficult and rewarding experiences of my life. I’ve learned a ton personally and professionally and my overall approach to learning has changed.

With the transition, I’ve discovered that one of my favorite parts of the role is speaking directly with technology and learning leaders, in addition to learners themselves. I’ve conducted hundreds of research interviews and have heard stories from organizations about their desire to create a culture of learning, how they’re preparing employees for their roles, and promoting ongoing skill development. I’ve also heard learners talk about their motivations for continuous life long learning including their need to provide for their families and their desires to move up in their careers.

All in all, these conversations have reinvigorated my passion for learning. They have also exposed to me the opportunities we have as practitioners, learning providers, and the industry as a whole.

So I’ve decided to take the leap once again and bundle some of the lessons I’ve learned over the past 16 months into a series called Learning Leaps. Each week, I’ll be sharing about a topic I’ve encountered during my transition. My goal is to give back to the community that has given me this opportunity in the first place. Thank you for coming along this journey with me and I hope you find the lessons shared both inspiring and helpful.

With gratitude,

Roberta

The Introvert’s Guide to Creating Learning Products: The First 60 days

I say it all the time, but I cannot believe how fast time flies! I started my new role as Product Manager of Pluralsight’s newest product, Role IQ, over 60 days ago!

The move to product management was a very meticulous decision on my part. It took over 6 months of research, networking, interviewing, and a lot of introspection before finally deciding to take the leap to an official PM role. Now with 60 days in the bag, I’m so happy that I decided to make the move. Sometimes I wonder why I didn’t do it sooner, but then I remember that things always happen in the time they’re meant to.

 

Off to a rough start

The transition to Product Manager didn’t happen as smoothly as I would’ve liked. A week before my start date, I was scheduled to head into the office to meet some of my new team members. On the way out of the house, I ended up falling down 6 stairs. After falling, I lifted up my leg and noticed my ankle facing the opposite direction. OUCH! A trip to the ER and a dozen X-rays later, I found out my prognosis: a fractured ankle and  3 torn ligaments.

Image from iOS.jpg
Left: Immediately after the fall, Right: 2 weeks post-op cast removal

 

The first 60 days in a new role would be a challenge for almost anyone. But throw a broken ankle into the mix and things get taken to a whole other level. To say that I’ve grown personally and professionally while working and recovering over the past 60 days is an understatement. I wanted to take some time to share some lessons I’ve learned while starting my journey creating learning products:

 

Get your hands on the product!

One of the things I did within my first week, was complete a product teardown for Role IQ. A product teardown is when you investigate and reverse engineer the thinking and experience behind a product. This activity gave me a first hand look into the existing functionality of my product before I got too familiar with it’s ins and outs. It also allowed me to open up a more productive dialogue about the product with my team from the very beginning.

 

Start building relationships with all of your stakeholders

I am the absolute epitome of an introvert, so I knew going into my new role that I would have to make an extra effort to meet everyone. Over the first month, I ended up have 30+ virtual 1:1 sessions. I did them based on order of priority starting with my immediate team including developers and UX designer. I then starting meeting with other key stakeholders including other PMs, product marketing, support, and customer success. Once all the initial meet and greets we’re done, I made sure to put reoccurring meetings with stakeholders on my calendar so I’d never have to think twice about who to talk to and when. Due to the size of Pluralsight, I’m still discovering people that would be great connect with about my product. Thats why forming relationships early with folks is so important; whoever you meet with will likely refer you to others.

 

Get aquatinted with your OKRs and performance metrics

My second week on the job was the first week of Q4 and I was lucky enough to have perfectly crafted OKRs (objectives and key results) all ready to go. After reviewing the OKRs with my manager and getting my mind wrapped around them, I set up some time with my team. I held a deep dive session with the Role IQ team to discuss what we planned on accomplishing for the quarter and brainstorm some possible approaches to problems. The time spent discussing the OKRs was invaluable. It made each of us aware of how the product would be measured and what success would look like.

 

Talk to the customer ASAP

Almost all companies these days tout about the importance of “voice of the customer”, but not all practice it. At the end of the day product management is all about solving problems for your customers. How can anyone solve their customer’s problems if they’re not talking to them? Thats why I made it a point to kick off customer calls as soon as possible. Since my product encompasses B2B and B2C markets, that meant hopping on calls with learning and tech leaders inside of enterprise organizations, as well as connecting with the learners themselves.

I followed Pluralsight’s Directed Discovery process which included doing voice of the customer (VOC) exploration calls. I also did some customer confirmation testing (CCT) which included looking at qualitative and quantitative feedback from customers who were already interacting with the product.

In addition to conducting my own research, I listened to customer recordings that took place before I inherited the product. I also sat in on client calls that others we’re conducting. I can honestly say, there really is nothing like hearing feedback directly from the customer. Some of the best insights I’ve heard, have come straight from these sessions and they’ve immediately impacted the future of the product.

 

Have your first win

I knew going into my new role that I wanted to have my first win as soon as possible to prove that I was bringing value to the team. One of my favorite moments was leading my product into an Open Beta where managers and admins could opt into our experience via a banner in the UI. It took a ton of wrangling for it to go live – including our team  finishing up some amazing work on analytics features, working collaboratively with another product team, and leading demos for our product marketing, sales, and presales team. It was the moment that really proved to me that I could be a Product Manager, I was doing it, remotely, and with a broken ankle to-boot!

 

Ask Questions

One of the most powerful things a product manager can do is ask questions. If you hear someone talking about a process or procedure that you’ve never heard of – ask a question. If someone says why they built or do something a certain way – ask a question. You will learn so much about whats going on, how things work, and how someone thinks the way they do about something. Ask anyone and everything – it’s important to soak up as much knowledge as possible.

 

Be patient and take care of yourself

One of the most important things I’ve learned since getting injured is the power of patience. For the past few years, I’ve ran as fast as humanly possible to every goal that I set out for myself. I’ve had mentors and friends tell me to slow down, have more fun, and make time for myself but I never listened. The injury forced me to slow down and be patient with myself and my body. I’ve learned to listen to the signs that I need rest and not feel guilty about sitting on the couch and sleeping on the weekends. At the end of the day, it’s completely impacted my working style and made me a well rounded product manager.