Tracking is not Learning

SCORM and the LMS are the Achilles Heel of training. Tracking data has become synonymous with measurement. This week I got an email from a vendor promoting the tracking capabilities of the product. It made realize how often tracking data is used to misrepresent training success.

Many people think that metrics pulled out of an LMS indicate the success of training programs. They see tracking metrics as “performance measurement”. Tracking is not measurement and is no indication that learning took place, or that learning will transfer to job performance. Relying on tracking data shows our collective weakness in measuring training effectiveness.

Reliance on tracking data also means we aren’t asking the right questions about our training programs. How many times are you asked the following questions about training:

  • How many learners completed the eLearning course?
  • How many people attended training?

These are important stats, but they have little, if any, correlation to job performance. Just look at how effective the Secret Service ethics training was prior to the debacle in Columbia. Everyone completed training, but it was clearly not effective.

Reliance on tracking data also means instructional designers aren’t being honest about real measurement and effectiveness. It’s easy to hide behind tracking numbers, and often those numbers can give the impression that training programs are effective and valuable. It certainly sounds better to say that 90% of learners completed training than to say you have no idea whether or not they learned anything.

Yes, we can design training courses to measure learning through activities and assessments (not just quizzes, real assessment). In those cases, the tracking metrics do provide value. If the course has rigorous assessment, then your completion stat is an indicator of learning. But how many courses have you developed that truly assess learning? How many have you taken online or attended? Instructional designers are often between a rock and hard place – there is an expectation from management (or customers) that people complete training, so we are under pressure to ensure they do. It’s our job as course developers to make sure that completion is not the most important metric.

And we won’t necessarily find the answer in Kirkpatrick’s Level 3 evaluation. That’s a great idea, but impractical in many organizations. If you can do Level 3 evaluations, then do them. We really need to look more seriously at the types of integrated assessments we do in training and how learners can measure their own success. We can’t be afraid to let people fail the assessment, and shouldn’t punish those that do.

But instructional designers and course developers need to start at the beginning and ensure that the people asking for reports on training success understand what the data means. We also have to ask the right questions before we start developing training:

  • Why is this training important to the organization.
  • What criteria will be used by management to determine success.

If the answer to the second question is something like “everyone will complete the training” then you better go back to the first question and dig deeper into the problem.

In the long run, reliance on tracking data and lack of learning assessment will come back to bite us. If you figure out quickly that your training isn’t effective, then you can make adjustments before it’s too late. Achilles was a mighty warrior, but in the end he was defeated by his one vulnerability. Don’t let assessment and measurement be yours.

And we’re still looking for that LMS…

It’s been a long and busy 10 months since my last blog post with lots of ups and downs, celebrations and sorrows, accomplishments and let downs. Some of these I’ll get around to telling because I want to share what I’ve learned, good and bad. One of the things I would put in the “let downs” list is I have not locked down an LMS. To be honest, I didn’t spend that much time looking for an LMS in the last year; it’s just been a busy year.

I thought I had found an LMS with ClickCourse, but I realized I needed something that can manage classroom and online training for both customers and employees. Instead of looking at vendors, we spent a lot of time scoping and documenting requirements, and when all was said and done we had a long list of requirements. Coming up with the requirements forced us to look at what we really do as training developers and managers, and in many cases caused us to question why we did certain things. It helped focus on the main goals of an LMS and how an LMS will help us do our jobs better. With that renewed focus we are in a better place to make a decision, a decision based on business goals.

That’s an important point, because it applies to what we do as course developers and where we should always start – the goal. Every training and learning initiative should have a clear goal. It sounds obvious, but it something that’s easy to forget and brush aside.  A lot of courses get created because something new comes along or some policy gets handed down, but that doesn’t mean the training is divorced from business goals. If you’re training isn’t clearly tied to a business goal, you need to go back to square one and figure it out. If there is no goal, don’t waste your time on training.

The goal should be tangible, tied to business outcomes, and communicated to stakeholders. Communicating how the training ties directly to a business goal can be the key to the success of your training, and the initiative it supports. In many cases it’s pretty easy, which makes it easy to overlook.

Many businesses implement new systems or applications to improve efficiency, then roll out training to all employees. The training goal shouldn’t be training the new system, it should be improving efficiency and maximizing return on investment of the new system. In most cases, you don’t want to teach all aspects of a new system to every employee, you want to teach each group the specific processes and features they need. The overall goal will be supported by the training objectives for each group.

Yes, this is training design basics. In the block diagram for ADDIE defining goals is the first step, but one thing I’ve learned over the last year is that it doesn’t hurt to be reminded of the basics and not take the obvious for granted.

In my search for an LMS I’ve clearly defined my goals. Those goals guided the process of defining requirements and prioritizing the requirements. I’m still looking, but have a clear picture of what I need and how it will help reach business goals.

Community Makes the Difference at DevLearn09

I’m still decompressing from DevLearn09. As expected, the eLearning Guild put on an outstanding event. I knew it would be too much to take in, and it was. It was hard to decide which sessions to attend, so I missed several I wanted to see because of scheduling conflicts.

The Twitter activity was also overwhelming. In every session I attended there were people tweeting about it. It’s hard to pay attention and tweet at the same time, but we want to share the small nuggets of learning. Even though I missed sessions, I got real-time reports (positive and negative) on the sessions I missed.

This was good and bad. The downside was a couple of sessions I attended didn’t meet my expectations, so seeing tweets about how fabulous others were made me a little jealous.  The good thing was that I benefited from others sharing the highlights of sessions I missed. I wish there was a way to see all the presentations I missed.

Twitter also became the de facto method of networking. Almost everyone I met included their Twitter name as part of the introductions. Mark Chrisman (@badsquare) put his Twitter name on his name badge, great idea. Michelle Lentz (@writetechnology) put it best:

My favorite recurring line this wk: “oh! I follow you on twitter!” Instant friendships. #dl09

And it really was instant friendship. We were all there for a common purpose and had similar interests, so we already had a lot to talk about. Twitter accelerated the conversation because if you just met someone you follow or who follows you, then you already knew a lot about the person.  It felt like a reunion and conversation flowed easily and freely. The networking and relationships made DevLearn even more enriching and rewarding than it already was.

The technology helps us for connections, the events help us grow relationships. This makes the community stronger. It encourages us to share and be involved, even if it’s only 140 characters at a time. The sense of community, and wanting to give back to the community, was palpable. I’ve been to other conferences, but none of them have the sense of community that DevLearn has. Twitter is just one tool that helps us connect, but it really is the people that make all the difference.

Here are some of the people I connected with, they are all great people and worth following. Thanks to B.J. Schone for the idea.

I’ve got a lot more to say about DevLearn09, but the most important part of the conference was the people and community we are a part of, so I wanted to get this out first.

eLearning Podcasts

I love podcasts. I listen to one everyday – they’ve completely replaced the radio for me, and music during my short drives to work. I still listen to music while I work, but whenever I get the chance I listen to podcasts.

Unfortunately, I have yet to find a podcast about eLearning that I really like. I’m not talking about using podcasts to deliver eLearning, there is plenty of that. I’m talking about a podcast for eLearning and/or training professionals. I’m using eLearning here very generically to cover just about everything to do with designing/developing/delivering/shaping content/activities/adventures to help people learn or do something. I tweeted the question last week and the only response was from James Kingsley (@onEnterFrame) saying “let me know if you find a good one pls.”

Syberworks, the LMS vendor has a podcast that I tried but couldn’t really get into. (I’m naturally skeptical of vendor podcasts. I fully recognize I might be missing something here.)

I recently discovered Captivating! the Adobe Captivate podcast, but haven’t listened yet. I’m not a Captivate user and most of the episodes seem to be short tutorial videos. And videos are kinda hard to watch while driving.

Based on my Google searches, there seems to be void waiting to be filled. We need a regular, ongoing eLearning podcast. If you know of one, please tell me, and others. If you want to start one, let me know. I’m game.

Missing the Point of Twitter

I recently read an NPR commentary about Twitter in which the author says he won’t use Twitter because he thinks people should keep their lives private and not broadcast every mundane event to world. Unfortunately, the author is missing the point, and missing it badly.

The power of Twitter is not in telling the world that I’m having a turkey sandwich for lunch. The power is in learning from other people. Twitter is an ongoing conversation about what is happening in the world around us. It’s a stream of consciousness medium that you can dip into whenever you want, or ignore for as long as you want. It’s me as an individual learning from the collective tweets of those I follow, and being able to contribute to that collective experience.

Yes, there are a lot of people tweeting away about every nuisance of their life, and that does get old, fast. But the cool thing about this stream is that you don’t have to follow everyone. You get to choose who you want to listen to – you can filter out the noise. It’s not a broadcast to the world, it’s a selective tuning in to the people and organizations you want to hear from.

With the increase of marketing, spammers, and blatant self-promotion on Twitter you have to choose carefully who you follow, and potentially who you block. I do not automatically follow everyone who follows me. I’m not trying to collect followers, I’m trying to make meaningful connections. There has to be a connection, or I won’t follow.  I also don’t feel bad about un-following people that add too much noise to the Twitterstream. Author Matthew Wayne Selznick (@mwsmedia) summed it up pretty nicely with this tweet:

“Sigh. Even the tweetstream of one of my favorite blogs, @WritetoDone, is mostly linkballast. Communicate! Be human — at least mostly!” (link)

The key to Twitter success is not having thousands, or millions, of followers. It’s following the right people and building connections. It’s who you follow, not who follows you.

Teaching the Social Web

Yesterday my wife got a message on Facebook from a friend asking for advice on how to handle a young teenage girl’s use of MySpace. I instantly went into parent mode and told her how we deal with our teenager. The overall theme of my response was trust and communication. If you build trust and communicate openly, you can avoid a lot of problems and pitfalls.

Then I got to work and started thinking about social learning, the workplace, and generation gaps. Today we think and talk about how to implement and manage social learning in the workplace. I’ve seen discussions and blog posts about appropriate use of social tools like Facebook, Twitter, LinkedIn and FriendFeed in the workplace. People talk about the same themes I did in my response: Trust and Communication.

  • Do we trust people to be responsible in the workplace?
  • What types of communication are appropriate?
  • Are usage guidelines effectively defined?

The talk is often reactive, e.g.  people have the tools and are using them, so we need to put some guidelines and policies in place. It’s shifting to be more proactive, e.g. how do we build social learning tools our employees can use effectively?

We need to be proactive and think about our future workforce. My 4th grader just finished a research report. From my perspective the goal of the report was to prepare kids for more in-depth research and reporting as they get older – valuable skills in the professional workplace. There was a direct correlation between the 4th grade report and the work I do daily. Schools have built-in programs that prepare kids not only for the next level of education, but also for life as a working adult. But does that preparation adequately address the rapid shift of social interactions to the online world? Are kids being prepared to use social tools as resources?

At a certain level and in some cases, yes. But are the foundations for responsible and effective use being laid at the right age? I’m not sure. If kids, and I mean pre-teen kids, can learn to research online using government web sites and Wikipedia, then shouldn’t they also learn about how to use social sites for learning? At what age should they start learning how to be an online citizen?

Parents need to be responsible and discuss appropriate social use of the web with their kids, but do schools bear some responsibility for preparing kids for appropriate social learning? What does that preparation look like? Most professional jobs expect entry level workers to know certian tools like Office. When do we start demanding that workers also demonstrate effective use of Twitter or LinkedIn or Ning or blogging? It’s a question worth asking of your local school board.

Access = Learning

Learning is about access to information. The more information people have available to them, the more likely they are to learn. Sounds pretty obvious, right? After all, Google has turned into the greatest job support/learning tool ever created because it gives us instant access to the information we need, when we need it. Why should we treat learning in our organizations any differently?

Learning resources should be freely available to the people that need them without forcing them to jump through hoops.  The learning landscape is moving beyond the concept of a course. Times are changing, people are growing up with Google, Digg, Facebook,, RSS feeds, and Twitter. The way we provide information and training must match the way people consume it.

Remember the newspaper? It got delivered to your door every morning and you sat down to read it over breakfast. That’s how you learned what was going on in the world. How many people under the age of 40 still do that? How many ever did that? The idea that courses are the pinnacle of eLearning is as archaic as the idea of getting news only from the morning paper. Newspapers are scrambling to change their business model, and so should course developers.

Why not just let learners view the resources they need? Web server statistics will give us a lot of information about what resources people are using. We can build simple tracking or feedback mechanisms without a lot of overhead.

If we want people to learn we need to be less concerned about how many pages they view or which quizzes they pass, and more concerned about providing the right information in the right way. Access to information does equate to learning. We know that 80% of learning is informal. We should focus at least 80% of our effort in that direction.