Tracking is not Learning

SCORM and the LMS are the Achilles Heel of training. Tracking data has become synonymous with measurement. This week I got an email from a vendor promoting the tracking capabilities of the product. It made realize how often tracking data is used to misrepresent training success.

Many people think that metrics pulled out of an LMS indicate the success of training programs. They see tracking metrics as “performance measurement”. Tracking is not measurement and is no indication that learning took place, or that learning will transfer to job performance. Relying on tracking data shows our collective weakness in measuring training effectiveness.

Reliance on tracking data also means we aren’t asking the right questions about our training programs. How many times are you asked the following questions about training:

  • How many learners completed the eLearning course?
  • How many people attended training?

These are important stats, but they have little, if any, correlation to job performance. Just look at how effective the Secret Service ethics training was prior to the debacle in Columbia. Everyone completed training, but it was clearly not effective.

Reliance on tracking data also means instructional designers aren’t being honest about real measurement and effectiveness. It’s easy to hide behind tracking numbers, and often those numbers can give the impression that training programs are effective and valuable. It certainly sounds better to say that 90% of learners completed training than to say you have no idea whether or not they learned anything.

Yes, we can design training courses to measure learning through activities and assessments (not just quizzes, real assessment). In those cases, the tracking metrics do provide value. If the course has rigorous assessment, then your completion stat is an indicator of learning. But how many courses have you developed that truly assess learning? How many have you taken online or attended? Instructional designers are often between a rock and hard place – there is an expectation from management (or customers) that people complete training, so we are under pressure to ensure they do. It’s our job as course developers to make sure that completion is not the most important metric.

And we won’t necessarily find the answer in Kirkpatrick’s Level 3 evaluation. That’s a great idea, but impractical in many organizations. If you can do Level 3 evaluations, then do them. We really need to look more seriously at the types of integrated assessments we do in training and how learners can measure their own success. We can’t be afraid to let people fail the assessment, and shouldn’t punish those that do.

But instructional designers and course developers need to start at the beginning and ensure that the people asking for reports on training success understand what the data means. We also have to ask the right questions before we start developing training:

  • Why is this training important to the organization.
  • What criteria will be used by management to determine success.

If the answer to the second question is something like “everyone will complete the training” then you better go back to the first question and dig deeper into the problem.

In the long run, reliance on tracking data and lack of learning assessment will come back to bite us. If you figure out quickly that your training isn’t effective, then you can make adjustments before it’s too late. Achilles was a mighty warrior, but in the end he was defeated by his one vulnerability. Don’t let assessment and measurement be yours.

And we’re still looking for that LMS…

It’s been a long and busy 10 months since my last blog post with lots of ups and downs, celebrations and sorrows, accomplishments and let downs. Some of these I’ll get around to telling because I want to share what I’ve learned, good and bad. One of the things I would put in the “let downs” list is I have not locked down an LMS. To be honest, I didn’t spend that much time looking for an LMS in the last year; it’s just been a busy year.

I thought I had found an LMS with ClickCourse, but I realized I needed something that can manage classroom and online training for both customers and employees. Instead of looking at vendors, we spent a lot of time scoping and documenting requirements, and when all was said and done we had a long list of requirements. Coming up with the requirements forced us to look at what we really do as training developers and managers, and in many cases caused us to question why we did certain things. It helped focus on the main goals of an LMS and how an LMS will help us do our jobs better. With that renewed focus we are in a better place to make a decision, a decision based on business goals.

That’s an important point, because it applies to what we do as course developers and where we should always start – the goal. Every training and learning initiative should have a clear goal. It sounds obvious, but it something that’s easy to forget and brush aside.  A lot of courses get created because something new comes along or some policy gets handed down, but that doesn’t mean the training is divorced from business goals. If you’re training isn’t clearly tied to a business goal, you need to go back to square one and figure it out. If there is no goal, don’t waste your time on training.

The goal should be tangible, tied to business outcomes, and communicated to stakeholders. Communicating how the training ties directly to a business goal can be the key to the success of your training, and the initiative it supports. In many cases it’s pretty easy, which makes it easy to overlook.

Many businesses implement new systems or applications to improve efficiency, then roll out training to all employees. The training goal shouldn’t be training the new system, it should be improving efficiency and maximizing return on investment of the new system. In most cases, you don’t want to teach all aspects of a new system to every employee, you want to teach each group the specific processes and features they need. The overall goal will be supported by the training objectives for each group.

Yes, this is training design basics. In the block diagram for ADDIE defining goals is the first step, but one thing I’ve learned over the last year is that it doesn’t hurt to be reminded of the basics and not take the obvious for granted.

In my search for an LMS I’ve clearly defined my goals. Those goals guided the process of defining requirements and prioritizing the requirements. I’m still looking, but have a clear picture of what I need and how it will help reach business goals.

Community Makes the Difference at DevLearn09

I’m still decompressing from DevLearn09. As expected, the eLearning Guild put on an outstanding event. I knew it would be too much to take in, and it was. It was hard to decide which sessions to attend, so I missed several I wanted to see because of scheduling conflicts.

The Twitter activity was also overwhelming. In every session I attended there were people tweeting about it. It’s hard to pay attention and tweet at the same time, but we want to share the small nuggets of learning. Even though I missed sessions, I got real-time reports (positive and negative) on the sessions I missed.

This was good and bad. The downside was a couple of sessions I attended didn’t meet my expectations, so seeing tweets about how fabulous others were made me a little jealous.  The good thing was that I benefited from others sharing the highlights of sessions I missed. I wish there was a way to see all the presentations I missed.

Twitter also became the de facto method of networking. Almost everyone I met included their Twitter name as part of the introductions. Mark Chrisman (@badsquare) put his Twitter name on his name badge, great idea. Michelle Lentz (@writetechnology) put it best:

My favorite recurring line this wk: “oh! I follow you on twitter!” Instant friendships. #dl09

And it really was instant friendship. We were all there for a common purpose and had similar interests, so we already had a lot to talk about. Twitter accelerated the conversation because if you just met someone you follow or who follows you, then you already knew a lot about the person.  It felt like a reunion and conversation flowed easily and freely. The networking and relationships made DevLearn even more enriching and rewarding than it already was.

The technology helps us for connections, the events help us grow relationships. This makes the community stronger. It encourages us to share and be involved, even if it’s only 140 characters at a time. The sense of community, and wanting to give back to the community, was palpable. I’ve been to other conferences, but none of them have the sense of community that DevLearn has. Twitter is just one tool that helps us connect, but it really is the people that make all the difference.

Here are some of the people I connected with, they are all great people and worth following. Thanks to B.J. Schone for the idea.

I’ve got a lot more to say about DevLearn09, but the most important part of the conference was the people and community we are a part of, so I wanted to get this out first.

eLearning Podcasts

I love podcasts. I listen to one everyday – they’ve completely replaced the radio for me, and music during my short drives to work. I still listen to music while I work, but whenever I get the chance I listen to podcasts.

Unfortunately, I have yet to find a podcast about eLearning that I really like. I’m not talking about using podcasts to deliver eLearning, there is plenty of that. I’m talking about a podcast for eLearning and/or training professionals. I’m using eLearning here very generically to cover just about everything to do with designing/developing/delivering/shaping content/activities/adventures to help people learn or do something. I tweeted the question last week and the only response was from James Kingsley (@onEnterFrame) saying “let me know if you find a good one pls.”

Syberworks, the LMS vendor has a podcast that I tried but couldn’t really get into. (I’m naturally skeptical of vendor podcasts. I fully recognize I might be missing something here.)

I recently discovered Captivating! the Adobe Captivate podcast, but haven’t listened yet. I’m not a Captivate user and most of the episodes seem to be short tutorial videos. And videos are kinda hard to watch while driving.

Based on my Google searches, there seems to be void waiting to be filled. We need a regular, ongoing eLearning podcast. If you know of one, please tell me, and others. If you want to start one, let me know. I’m game.

Missing the Point of Twitter

I recently read an NPR commentary about Twitter in which the author says he won’t use Twitter because he thinks people should keep their lives private and not broadcast every mundane event to world. Unfortunately, the author is missing the point, and missing it badly.

The power of Twitter is not in telling the world that I’m having a turkey sandwich for lunch. The power is in learning from other people. Twitter is an ongoing conversation about what is happening in the world around us. It’s a stream of consciousness medium that you can dip into whenever you want, or ignore for as long as you want. It’s me as an individual learning from the collective tweets of those I follow, and being able to contribute to that collective experience.

Yes, there are a lot of people tweeting away about every nuisance of their life, and that does get old, fast. But the cool thing about this stream is that you don’t have to follow everyone. You get to choose who you want to listen to – you can filter out the noise. It’s not a broadcast to the world, it’s a selective tuning in to the people and organizations you want to hear from.

With the increase of marketing, spammers, and blatant self-promotion on Twitter you have to choose carefully who you follow, and potentially who you block. I do not automatically follow everyone who follows me. I’m not trying to collect followers, I’m trying to make meaningful connections. There has to be a connection, or I won’t follow.  I also don’t feel bad about un-following people that add too much noise to the Twitterstream. Author Matthew Wayne Selznick (@mwsmedia) summed it up pretty nicely with this tweet:

“Sigh. Even the tweetstream of one of my favorite blogs, @WritetoDone, is mostly linkballast. Communicate! Be human — at least mostly!” (link)

The key to Twitter success is not having thousands, or millions, of followers. It’s following the right people and building connections. It’s who you follow, not who follows you.

Teaching the Social Web

Yesterday my wife got a message on Facebook from a friend asking for advice on how to handle a young teenage girl’s use of MySpace. I instantly went into parent mode and told her how we deal with our teenager. The overall theme of my response was trust and communication. If you build trust and communicate openly, you can avoid a lot of problems and pitfalls.

Then I got to work and started thinking about social learning, the workplace, and generation gaps. Today we think and talk about how to implement and manage social learning in the workplace. I’ve seen discussions and blog posts about appropriate use of social tools like Facebook, Twitter, LinkedIn and FriendFeed in the workplace. People talk about the same themes I did in my response: Trust and Communication.

  • Do we trust people to be responsible in the workplace?
  • What types of communication are appropriate?
  • Are usage guidelines effectively defined?

The talk is often reactive, e.g.  people have the tools and are using them, so we need to put some guidelines and policies in place. It’s shifting to be more proactive, e.g. how do we build social learning tools our employees can use effectively?

We need to be proactive and think about our future workforce. My 4th grader just finished a research report. From my perspective the goal of the report was to prepare kids for more in-depth research and reporting as they get older – valuable skills in the professional workplace. There was a direct correlation between the 4th grade report and the work I do daily. Schools have built-in programs that prepare kids not only for the next level of education, but also for life as a working adult. But does that preparation adequately address the rapid shift of social interactions to the online world? Are kids being prepared to use social tools as resources?

At a certain level and in some cases, yes. But are the foundations for responsible and effective use being laid at the right age? I’m not sure. If kids, and I mean pre-teen kids, can learn to research online using government web sites and Wikipedia, then shouldn’t they also learn about how to use social sites for learning? At what age should they start learning how to be an online citizen?

Parents need to be responsible and discuss appropriate social use of the web with their kids, but do schools bear some responsibility for preparing kids for appropriate social learning? What does that preparation look like? Most professional jobs expect entry level workers to know certian tools like Office. When do we start demanding that workers also demonstrate effective use of Twitter or LinkedIn or Ning or blogging? It’s a question worth asking of your local school board.

Access = Learning

Learning is about access to information. The more information people have available to them, the more likely they are to learn. Sounds pretty obvious, right? After all, Google has turned into the greatest job support/learning tool ever created because it gives us instant access to the information we need, when we need it. Why should we treat learning in our organizations any differently?

Learning resources should be freely available to the people that need them without forcing them to jump through hoops.  The learning landscape is moving beyond the concept of a course. Times are changing, people are growing up with Google, Digg, Facebook,, RSS feeds, and Twitter. The way we provide information and training must match the way people consume it.

Remember the newspaper? It got delivered to your door every morning and you sat down to read it over breakfast. That’s how you learned what was going on in the world. How many people under the age of 40 still do that? How many ever did that? The idea that courses are the pinnacle of eLearning is as archaic as the idea of getting news only from the morning paper. Newspapers are scrambling to change their business model, and so should course developers.

Why not just let learners view the resources they need? Web server statistics will give us a lot of information about what resources people are using. We can build simple tracking or feedback mechanisms without a lot of overhead.

If we want people to learn we need to be less concerned about how many pages they view or which quizzes they pass, and more concerned about providing the right information in the right way. Access to information does equate to learning. We know that 80% of learning is informal. We should focus at least 80% of our effort in that direction.

Will Social Marketing Kill Social Learning?

Twitter may be the hottest thing online right now, and is probably the fastest growing social networking tool around. Twitter is all over the news, both online and traditional. Everyone from Barack Obama to the Chihuahua next door has a Twitter account. It’s becoming as commonplace as email.

A couple of days ago ZDNet posted an article about a commercial Twitter spamming tool. Not surprisingly, I heard about the article on Twitter. Twitter spam is nothing new. If you’ve had an account for more than a couple of days, you’ve probably gotten followers who are spammers trying to get you to follow them. The article got me thinking about the negative impact social marketing  has on social learning.

Twitter seems to be changing from a fun way to connect and share into a promotional tool. I got hooked on twitter because I enjoyed reading little snippets about what people were up to. I’ve found a lot of great resources through Twitter. From that perspective, Twitter is a great social learning tool. You can find resources and even get help from your Tweeps (people you follow or who follow you).

I think social marketing could negatively impact social learning. With so many companies getting accounts and pushing their products and services, it’s becoming harder and harder to find value in individual tweets. Companies don’t seem to really understand Social Marketing. Social marketing works when Person A tweets about a product and gets Person B interested. Person B then passes it on to others, and so on.

Just because a company has a Twitter account does not mean it uses social marketing. Using a social networking tool to spam us with ads is not social marketing. At that point it is just advertising, and I get plenty of that already. Social marketing is also not viral marketing. They are very different and companies need to learn the difference.

For me the value of the resource or tweet comes from the source. If B.J. Schone or Tony Karrer posts a link about training, I’ll check it out. They are real people who contribute to the online community. Company X posting links to their own white papers do not have the same value for me and I probably won’t follow the link. More precisely, I won’t follow them to begin with.

I’ve noticed a sharp increase in the number of followers I’ve gotten over the past month or so. I usually wait a few days before checking out their profile, and about half the time the account has been suspended by the time I check it out. I don’t typically follow people unless their profile indicates we have something in common, either professionally or personally.

At least with Twitter we have control. I don’t have to follow every entity that follows me and I can block ones that really offend me. I can unfollow people who do nothing but promote themselves or the companies they work for. I can follow cool services like @php that will help me, or people like @DarthVader that make me laugh, or people like JC Hutchins that I’m a fan of. I can still take advantage of social learning, but I first have to separate the signal from the noise.

Twitter is still awesome even though corporations, spammers, and “social marketers” have jumped on the band wagon. We just have to not follow them. You should measure the value of Twitter by the quality of the people, real people, that you connect with, not by the number of followers you have.  Twitter and other social tools have the power to be great learning resources, just don’t let the spam get in the way. Now I have to go reduce my Twitter noise.

5 Best Improvements in Articulate Presenter 09

I’ve been using Articulate Presenter 09 for a couple of months now and have come up with the five things I like best about the new version. If you follow me on Twitter, you might have seen my posts about having problems with the new version. Articulate has released two updates to Presenter 09 and most of the problems I had have been cleared up, or I’ve found workarounds.  Even with the issues, I wouldn’t go back to the previous version.

So, my top 5 improvements:

1. Articulate Tab – Easily the best thing about the new version, Articulate integrates Presenter into PowerPoint 2007’s ribbon bar. All the functions are in a tab on the ribbon, making them easy to find. The button are grouped logically to improve your work flow so you don’t have to hunt around for menu options.

2. Improved Quizmaker Integration – I was never happy with the integration of Quizmaker and Presenter, but Articulate has fixed the issues. Now you can seamlessly jump from Presenter into Quizmaker and back with no errors. You also see a thumbnail of the first question (or instruction) and the quiz options on the placeholder slide. Thank you, Articulate.

3. Video – I’ve only used this for one project so far, but it was completely painless. Video needs to be in the FLV format, but Articulate supplies a conversion tool just in case your video editing software doesn’t convert to FLV. Inserting video uses the Insert Flash tool and has the same options as SWF files.

4. Improved Previewing – Now you can preview the current slide, the next three slides, or a range of slides. I use this feature a lot. It comes in handy for seeing how well a series of slides flow without having to publish the entire course.

5. Articulate Branding Removed – This is actually an option in the presentation template; the Articulate logo is still there by default. You just have to un-check the box to remove the branding. Obviously this has nothing to do with improved functionality, but the forced branding in the old version drove me crazy. No other development tool does that, nor should they.

Presenter 09 has several other features that have been improved, but I wanted to highlight the five that I like best. It’s a good tool and has definitely sped up development for me. You do need to plan carefully and make sure you don’t just put up a bunch of text slides that will put you learners to sleep. Remember, rapid development is not necessarily good development.

It’s time for upgrades, and a Poll

It looks like it’s time to open up the wallet and start purchasing software upgrades. In the past couple of weeks two important eLearning software upgrades have hit the virtual streets:

  • Adobe CS4 came out this week
  • Articulate released the Studio 09 update to all their tools a couple of weeks ago

Tools are constantly going through upgrade cycles, but these are two of the biggest names in the eLearning development tool market, so they caught my attention.

Adobe CS4

This is a huge upgrade, and the first time Adobe has released a unified upgrade for all the Adobe and former Macromedia products. They’ve also changed the bundles – thank you Adobe. The CS4 Web Premium bundle now includes:

The upgrade from most CS3 bundles is $599, even the old Web Standard which did not include Soundbooth, Photoshop or Illustrator. I think Adobe has finally gotten their bundles right making the new CS4 Web Premium bundle an essential upgrade for eLearning developers. I will be upgrading my license at work as soon as possible. The full version is $1699 if anyone wants to buy me an early Christmas present for personal use.

Articulate Studio 09

Articulate Presenter is arguably the most popular PowerPoint to Flash conversion tool available. That’s no fluke, the tool works well and streamlines development. With the right hacking, you can accomplish a lot with Presenter and make it do things it wasn’t designed to do. It’s also an easy way to create Flash animations from slides. I’ve used it for several months and haven’t been disappointed.

Studio 09 Standard includes Presenter, Quizmaker and Video Encoder. The Professional version adds Engage. I downloaded Studio 09, but haven’t tried the new Presenter yet. There were a few quirks and bugs in the old version of Presenter that I hope are fixed. When I have time, I’ll publish some courses and let you know what I think.

For Studio 09, Articualte added Articulate Video Encoder ’09. This tool converts video files to FLV for inclusion in courses. I did take some time to test out this tool and can’t say I was overly impressed. If you don’t already have the Flash Video Encoder (which comes with Flash) then this tool will be useful. It works fine, I just prefer Flash Video Encoder. One feature that Articulate Video Encoder includes that the Flash encoder doesn’t is the ability to record your webcam and convert the file to FLV. I didn’t test this feature, and probably won’t use it. I wouldn’t buy this product as a stand alone purchase, especially for $149, but it’s a nice addition to the Studio 09 bundle, especially for people who don’t have Flash.


WordPress also added a new tool recently – Polling. Let’s give it a try and see how it works.