Digital Upskilling:

Problems and Prospects

Challenge

* Note: Specifics details of this project have been omitted for proprietary reasons.

How can we empower employees to consistently engage in upskilling so they can expand their career potential while they are nevertheless immersed in their current roles and responsibilities?

For a study on learner engagement, my team sought to understand the challenges of sustained upskilling for the working professionals using our learning experience platform (LXP). As a research team for a product that is a relatively new player in the digital upskilling ecosystem, we regularly conducted user interviews with both internal clients and users of competing platform (e.g,, Coursera, Udemy, Degreed, etc.). However, we had yet to conduct a long-term study of our users’ engagement within our platform.

What obstacles, both internal to the product and otherwise, stood in the way of our learners’ upskilling endeavors? What features of the platform were, and were not, successful in helping our users to incorporate learning into their working lives? And what modifications, small and large, might best encourage sustained engagement in both the short and long term?

These questions motivated our research, whose goal was to generate ideas for fostering motivation and sustained learning practices in ways that apply directly to our learners’ working lives.

Process

We began by developing a research plan aimed at understanding the challenges to creating sustained learning practices and the opportunities for encouraging these practices within our platform.

We arrived at a two-stage qualitative approach.

  1. Explore people’s learning practices via remote diary studies using dScout. Participants (n=16) logged their learning experiences on our platform over 21 days, detailing their learning successes, obstacles, and evolving rapport with the platform.

  2. Document, subsequent the diary study, a taxonomy of data touch points via remote focus groups using Zoom and Mural, in which the same participants (n=16) would discuss their experience over the prior 3-week period. 

Methodology

Diary studies are a useful tool to gather longitudinal data that trace people’s evolving attitudes and behaviors over time. To facilitate this process, our participants were asked to commit to a course or credential program over the 21-day period. They were also invited, though not required, to complete any additional learning they wished in the form of articles, videos, podcasts, etc. Participants were asked to complete a series of tasks every 3-day interval:  

  • At least one segment of their required learning

  • A video log via dScout, recording high and low experiences

  • A written log on a excel file to provide any supplementary data, screen shots, or links

Diary Study

Participants recorded brief videos describing their experiences across the learning process, from initial login and navigation to content consumption and evaluation. Participants engaged with both internally developed and 3-party sourced courses and credentials. Participants’ selection of supplement material ranged from articles by CNN, TechCrunch, and USA Today; videos by TED and BizSkills; and podcasts by Getting Smart and Pluralsight.

In total, 224 submissions were collected, offering us a trove of data to analyze.

At the culmination of our diary studies, we divided our participants in groups of 4 to conduct our retrospective focus groups. These were structured around three exercises on the Mural boards.

First, participants were asked to document 15 aspects of their diary study experience on pre-arranged sticky notes. After performing the exercise individually, everyone shared what they wrote. It was an opportunity to reflect on the manifold aspects of online learning.  

Focus Groups

Second, participants copied their sticky notes and arranged them in the bullseye labeled “satisfaction with product.” The center signified the most satisfying aspects; each concentric ring outward represented decreasing satisfaction. The exercise invited participants to consider what aspects of their learning experience were most conducive and most prohibitive to learning.

Finally, participants were asked to rank a series of 8 platform features in terms of their effectiveness in getting them to return to the platform regularly:

  1. Clear indicators of progression towards mastery of a skill or completion of one’s learning plan.

  2. Notifications such as required learning due dates, course progression, etc.

  3. Personalized suggestions related to new courses, skills paths, etc.

  4. The ability to set learning goals with reminders to complete them.

  5. Features supporting community learning, e.g., the ability to create collections, to learn within a team, provide ratings/reviews, etc.

  6. The ability to share learning achievements with one’s leaders and on external platforms.

  7. The ability to collaborate with your direct manager on your learning goals/timelines.

  8. Gamification elements like learning streaks, learning badges, company leaderboards, etc.

Participants were asked to individually divide the features into two groups: 4 that are of primary importance for facilitating learning (with a weighted value of 2.0); and 4 that are of secondary importance (with a weighted value of 1.0). The exercise offered the research team insight into learners’ views of what constituted the most essentials aspect of learner engagement.

All together, our focus groups provided 368 data touchpoints for us to analyze.

Insights

Our diary study produced a plethora of insights into our users’ learning habits, challenges, and strategies. Yet three key findings stood out:

  • Flow of work: the study demonstrated that learning habits are largely dictated by work patterns, which can vary widely, and that key to stimulating sustained engagement is making learning a more seamless aspect of learners’ working lives. Outside of making learning mandatory, this is best achieved through content delivery. Participants reported that completing their learning goals within the study’s allotted time intervals was largely dependent on the type of content they consumed. Articles, videos, and podcasts, for example, suited most users better for shorter spurts than more involved courses. This suggested that learners should have more input on how these intervals, and the format in which content paired with them, are constituted.

  • Content Relevancy: a crucial insight resulting from the study was that a central feature of our platform needed to be rethought. What differentiates it from competitors is its concept of a learning plan that is tailored to a learner’s professional role. The intention is to provide learners with a ready-made curriculum that addresses the exigencies of their current position. However, no matter how closely their assigned learning plans aligned with their day-to-day responsibilities, our participants invariably found a significant portion of their learning plans to misaddress their actual learning needs. What the current learning plans failed to adequately address, it turns out, was usually learners’ long-term ambitions, which could diverge from their current roles in a myriad of ways. This suggested a move away from linear learning plans to more loosely aligned skills pathways, which once again learners should have more input in configuring.

  • Social learning: there was broad consensus among our participants that social learning, and the socialization of learning, can help sustain long-term goals. First, learners expressed the desire for access to subject matter experts through whom they could address outstanding questions and consolidate prior learning. Second, over 2/3 of our participants suggested that group channels in which learning insights, struggles, and achievements could be shared would significantly impact regular engagement. Finally, learners expressed the importance of being able to share their learning accomplishments with their supervisors and peers as crucial to maintaining motivation. These findings indicated the need to look more closely at how we can incorporate social opportunities as integral aspects of learning pathways.

During our focus groups, the results of our Mural board exercises largely mirrored and reinforced our diary study findings. Discrepancy arose, however, when our participants engaged in the final feature-ranking exercise, particularly around the subject of social learning.

Feature Priority Ranking

n=16; 1st priority = 2.0,. 2nd priority = 1.0

While participants stressed the importance of social interaction in their diary study feedback, these same participants relegated it to secondary importance in the feature ranking exercise. What could account for these apparently conflicting assessments?

Upon further questioning, the answer appeared to be twofold. First, participants saw sociality to play a fundamentally supportive or complementary role in stimulating learner motivation. In other words, its effectiveness was dependent on the prior establishment of more core features centered on the individual learner’s goals. Second, not all social features were of equal value; some, like a community leaderboard, could even be discouraging or anxiety provoking to some participants.

Results

Our study produced foundational insights into long-term learner engagement as well as tactical recommendations for product development opportunities. These included increased personalization of the homepage and a movement from role-based to skill-based learning plans.

The project was also the impetus for several more targeted foundational studies, including ones tackling learning completion and accomplishment, goal setting, and personalized learning.

*That’s all for now! Specifics details of this project have been omitted for proprietary reasons.