Evaluation of Virtual Training Program

Click here to view the analysis presentation.

Project Background

The Predictive Index recruited a research group of masters-level students to investigate why it’s virtual training program was receiving low satisfaction ratings and identify potential improvements to the program.
My Role

As project lead, I led three other masters-level students through research and analysis of the program.

About The Predictive Index: The Predictive Index is a talent optimization company that  businesses use to hire, inspire, and retain talent inside of their organizations. A part of their product offering are instructor-led training classes that introduce the methodology and concepts behind their workplace assessment tools.

The Challenge

After nine months of piloting a virtual training program, the learning team noticed satisfaction scores of the virtual program were lower than the in-person program.

The Opportunity

The Predictive Index partnered with a group of master’s students from Boise State University to investigate why the virtual program had lower satisfaction scores ratings and identify potential improvements to the program. 

Key Research Questions and Areas: The analysis focused on four areas, including, program design and delivery, learner application, completion rate, and sales/marketing with the fundamental questions being:

  • Is the program meeting its intended outcomes?
  • What improvements could be made to the virtual training programs?

A Mixed Methods Approach

The team used a mixed-methods approach that included learner surveys, semi-structured interviews, ad-hoc data analysis, literature review, and observations.

Learner Surveys: The project team distributed a web-based survey to learners two months after their completion of attending the virtual training.

Semi-structured interviews: I conducted interviews with key stakeholders inside of the organization to gain a better idea of the current state of the program as well as the intended business goals. Based on feedback from the interviews, I developed a graphic to show the inputs and results of the program.

To get a better picture of the training program, the project team reviewed level 1 reaction data as well as course enrollment and completion data.

Literature Review: The team conducted a literature review of 20 academic articles to identify best practices used in the delivery and design of virtual training sessions. The findings were used to create an observation checklist that was used to evaluate the current state of the program.

Observations: Using the observation checklist, the project team observed four virtual training sessions and rated various elements of the program design and delivery.

Data Synthesis: Following data collection, the project team created rubrics, scored data, and created a dimensional triangulation rubric. The team drew evidence-based conclusions and recommendations using a synthesis rubric.

Overall Themes and Opportunities

The team determined that the overall quality of the virtual program was high. Overall, the program is headed in the right direction, but some gaps should be addressed.

The team prioritized the following recommendations based on the importance weighting of the individual dimensions:

Research DimensionsRecommendations
Program design and delivery
Determine the overall strategy of the virtual training program. For example, is it just an online version of the existing instructor-led training, or does it have another mission and purpose?
Learner application
Develop a level 3 (application) survey that is delivered to learners after a specified period (3 months) after completing the program to determine whether they have applied what they learned when they return to the workplace.
Completion rate
Develop a customer-facing website that makes self-registration for learners an easy one-stop-shop. The site should contain a calendar of training events where customers can see availability and register for events.
Sales and marketing strategy
Adapt the marketing strategy to indicate the learning outcomes learners can expect to receive based on their attendance.

Furever App for Cat Shelter Volunteers

Click here to view the full project.

Project Description
The Furever App is a team project that I worked on during my time in Boise State’s Organizational Performance and Workplace Learning (OPWL) masters program. 
My Role
As a contributing Learning Experience Designer on this project, I worked on analysis, design, and prototype development along with three other students.

The Challenge

Volunteers at the Idaho Humane Society struggle feeling confident while matching cats from the shelter to their forever homes. Current adoption rates of cats in the shelter are at 73%, and it is not uncommon for adopted cats to be returned at a later date. The shelter hopes to increase adoption rates to 85%. 

The Opportunity

The project team has the opportunity to create support resources to help volunteers match cats to their forever homes.

Mixed Methods Approach

Analysis: Our team sifted through over 200 pages of research consisting of volunteer interviews, surveys, and observations. We also conducted interviews to identify common issues among animal shelter volunteers.

Mission Statement: Based off of our research, the team conducted a brainstorm to generate possible opportunities to help the shelter volunteers. From there, we developed a mission/vision statement to guide our project.

CatVolunteer_WildIdeas
Learning Experience Statement for Cattery Volunteers at Idaho Humane Society

Empathy Map: We created an empathy map based on the research of our learners. This activity allowed us to consider who are learners were, what they’re doing, seeing, saying, hearing, thinking, and feeling.

CatVolunteer_EmpathyMap
Low Fidelity Empathy Map for Idaho Humane Society Cattery Volunteers

Learner Persona: Next, we created a learner personas for a segment of our volunteer group. This activity gave us a better understanding of the backgrounds, goals, and characteristics of our learners.

CatVolunteer_Persona
Higher Fidelity Learner Persona for Sharon

Journey Map: Next, we created a learner experience journey map for our learner persona. The journey map provided us with a better understanding of the different phases of learning volunteers go through and the tasks they performed. The journey map was used to identify where potential solutions could be incorporated.

CatVolunteer_JourneyMap
Higher Fidelity Journey Map for Sharon

The Furever Home Web-Based App

The outcome of our project was a low fidelity prototype of The Furever Home Web-Based App. The app would support the volunteers throughout the adoption intake process and enable them to provide effective, efficient, and customer-centered intakes.

Volunteers can use the app on their mobile phone or tablets. The app incorporates volunteer tools such as an intake questionnaire, job aids on cat and animal behavior, animal care and safety information, and customer service. The app also incorporates a digital badging component that tracks how many successful adoptions a volunteer has assisted with and encourages them to mentor and coach other volunteers in the shelter.

CatVolunteer_Prototype
Higher Fidelity Prototype of The Furever Home Based Web App

SaaS Learning Strategy

Click here to view an overview of the learning strategy.

Project Description

During Fall 2017, The Predictive Index was looking to expand its product offerings to include in-product digital learning experiences.
My Role

As Lead Learning Experience Designer, I conducted discovery and strategy research to identify potential opportunity areas related to this product space.

About The Predictive Index: The Predictive Index is a talent optimization company that businesses use to hire, inspire, and retain talent inside of their organizations. 

Approach

I conducted customer interviews to gain insight into users’ motivations, pain points, and identify opportunity areas for the product. I also led brainstorming sessions with internal stakeholders to determine where learning experience design fits within the whole product vision.

Outcome

The result was the creation of a mission statement as well as a rollout strategy for how The Predictive Index intended to deliver delightful digital learning experiences to their customers. As of early 2018, multiple learning experiences have been live and contributing to the whole product vision of the company (SaaS + eLearning + Workshops + Consulting).

LXD_KnowDoFeel
Learning Experience Brainstorm (Know, Do, Feel)
LXD_MissionStatement
Learning Experience Mission Statement