Tracking is not Learning

SCORM and the LMS are the Achilles Heel of training. Tracking data has become synonymous with measurement. This week I got an email from a vendor promoting the tracking capabilities of the product. It made realize how often tracking data is used to misrepresent training success.

Many people think that metrics pulled out of an LMS indicate the success of training programs. They see tracking metrics as “performance measurement”. Tracking is not measurement and is no indication that learning took place, or that learning will transfer to job performance. Relying on tracking data shows our collective weakness in measuring training effectiveness.

Reliance on tracking data also means we aren’t asking the right questions about our training programs. How many times are you asked the following questions about training:

  • How many learners completed the eLearning course?
  • How many people attended training?

These are important stats, but they have little, if any, correlation to job performance. Just look at how effective the Secret Service ethics training was prior to the debacle in Columbia. Everyone completed training, but it was clearly not effective.

Reliance on tracking data also means instructional designers aren’t being honest about real measurement and effectiveness. It’s easy to hide behind tracking numbers, and often those numbers can give the impression that training programs are effective and valuable. It certainly sounds better to say that 90% of learners completed training than to say you have no idea whether or not they learned anything.

Yes, we can design training courses to measure learning through activities and assessments (not just quizzes, real assessment). In those cases, the tracking metrics do provide value. If the course has rigorous assessment, then your completion stat is an indicator of learning. But how many courses have you developed that truly assess learning? How many have you taken online or attended? Instructional designers are often between a rock and hard place – there is an expectation from management (or customers) that people complete training, so we are under pressure to ensure they do. It’s our job as course developers to make sure that completion is not the most important metric.

And we won’t necessarily find the answer in Kirkpatrick’s Level 3 evaluation. That’s a great idea, but impractical in many organizations. If you can do Level 3 evaluations, then do them. We really need to look more seriously at the types of integrated assessments we do in training and how learners can measure their own success. We can’t be afraid to let people fail the assessment, and shouldn’t punish those that do.

But instructional designers and course developers need to start at the beginning and ensure that the people asking for reports on training success understand what the data means. We also have to ask the right questions before we start developing training:

  • Why is this training important to the organization.
  • What criteria will be used by management to determine success.

If the answer to the second question is something like “everyone will complete the training” then you better go back to the first question and dig deeper into the problem.

In the long run, reliance on tracking data and lack of learning assessment will come back to bite us. If you figure out quickly that your training isn’t effective, then you can make adjustments before it’s too late. Achilles was a mighty warrior, but in the end he was defeated by his one vulnerability. Don’t let assessment and measurement be yours.

And we’re still looking for that LMS…

It’s been a long and busy 10 months since my last blog post with lots of ups and downs, celebrations and sorrows, accomplishments and let downs. Some of these I’ll get around to telling because I want to share what I’ve learned, good and bad. One of the things I would put in the “let downs” list is I have not locked down an LMS. To be honest, I didn’t spend that much time looking for an LMS in the last year; it’s just been a busy year.

I thought I had found an LMS with ClickCourse, but I realized I needed something that can manage classroom and online training for both customers and employees. Instead of looking at vendors, we spent a lot of time scoping and documenting requirements, and when all was said and done we had a long list of requirements. Coming up with the requirements forced us to look at what we really do as training developers and managers, and in many cases caused us to question why we did certain things. It helped focus on the main goals of an LMS and how an LMS will help us do our jobs better. With that renewed focus we are in a better place to make a decision, a decision based on business goals.

That’s an important point, because it applies to what we do as course developers and where we should always start – the goal. Every training and learning initiative should have a clear goal. It sounds obvious, but it something that’s easy to forget and brush aside.  A lot of courses get created because something new comes along or some policy gets handed down, but that doesn’t mean the training is divorced from business goals. If you’re training isn’t clearly tied to a business goal, you need to go back to square one and figure it out. If there is no goal, don’t waste your time on training.

The goal should be tangible, tied to business outcomes, and communicated to stakeholders. Communicating how the training ties directly to a business goal can be the key to the success of your training, and the initiative it supports. In many cases it’s pretty easy, which makes it easy to overlook.

Many businesses implement new systems or applications to improve efficiency, then roll out training to all employees. The training goal shouldn’t be training the new system, it should be improving efficiency and maximizing return on investment of the new system. In most cases, you don’t want to teach all aspects of a new system to every employee, you want to teach each group the specific processes and features they need. The overall goal will be supported by the training objectives for each group.

Yes, this is training design basics. In the block diagram for ADDIE defining goals is the first step, but one thing I’ve learned over the last year is that it doesn’t hurt to be reminded of the basics and not take the obvious for granted.

In my search for an LMS I’ve clearly defined my goals. Those goals guided the process of defining requirements and prioritizing the requirements. I’m still looking, but have a clear picture of what I need and how it will help reach business goals.

And the LMS selection winner is…

After over a year of looking and talking with several LMS vendors, I finally chose an LMS – ClickCourse from Rapid Intake.

My requirements were pretty simple with cost at the top of the list. I wanted something that could track courses reliably, integrate with our existing customer extranet, generate basic reports, and be relatively easy to use. ClickCourse met those criteria at the best price. With ClickCourse I’m not paying for a bunch features I don’t need. Inquisiq was my first runner up. It’s also very affordable and was recommended by a few people.

There are well over 100 LMS products on the market and they all do mostly the same thing – track training. True, many do a lot more and offer some impressive features for managing learning and learners, but in the end I just wanted SCORM tracking at a fair price. If money were no concern, I probably would have chosen a different product and expanded the scope of the LMS. In that case I probably would have went with Thinking Cap or SyberWorks.

Price wasn’t my only consideration. Another primary factor was the company and their customer service. If you’ve been following this blog, you may recall my past posts about Rapid Intake’s Unison and Flashform. (By the way, Unison just won the Brandon Hall Gold Award for Best Innovation in Learning Technology.) Through this blog and subsequent in-person meetings I’ve gotten to know Garin Hess on a professional level and have been impressed with his (and others at Rapid Intake) willingness to help me and answer my questions. Even though I have yet to purchase any Rapid Intake products, I’ve gotten better customer service from them than from many companies I’ve purchased products from.  I’m confident Garin and his team can help us get our LMS up and running smoothly.

Now that the decision is behind me, I’m looking forward to the implementation. I know that will be… interesting. On a related note, Tracy Hamilton has been blogging about her experiences with her LMS over at Discovery Through eLearning. I’m sure I’ll be in touch.

Finally, to all those that have helped with this process through comments and emails, thank you.

Why it’s hard to find an LMS

It’s been several weeks since my last blog post about my LMS search. The search moves ever so slowly. I can’t say I’ve learned anything new about LMSs, but I am increasingly surprised by how much they cost. I am seriously considering building my own. Why would I pay tens of thousands of dollars year after year for something I can build myself? Time is the only valid reason – I don’t really have the time.

Shopping for an LMS is nothing like shopping for eLearning development tools. For example, if you want to compare software recording tools it’s pretty easy. You go to the various web sites, download trial versions and compare prices. Shopping for an LMS is not that easy.

I’ve found three areas that make it hard to find an LMS: they don’t all post pricing online, they don’t all have trials available, and the feature sets vary greatly.

Price – As a rule of thumb, if the price is not on the web site it’s going to cost a lot. Vendors want you to contact them for the price so they can get your email address and phone number so they can make person-to-person contact. I understand that, but I prefer to contact you when I am ready. I research online because I don’t want to talk to sales people until I am ready. Putting your prices online saves me time and saves you time. If your product is out of my budget, then you don’t have to waste time calling or emailing me.

(Side Rant – don’t hide your phone number on caller ID. I will not answer the phone unless I can see who is calling, and I might just delete the voice mail without listening to it. I consider it a professional courtesy to show your phone number on caller ID.)

Trials – Several vendors do give you free trials, which I greatly appreciate. I can’t imagine why any vendor would not offer a free trial. Webinars can give an adequate overview, but I want to get in and poke around myself. I don’t make any major purchase without some kind of test drive, large software purchases should be no exception.

Feature Sets – Most LMSs cover the basics pretty well (course tracking, user management, reports, and catalog management), but they can differ greatly once you get past the basics. Don’t let this get in your way. Before you start talking to sales reps, nail down your requirements. If you know exactly what you need, you can easily weed out products that offer too much. Keep in mind that you will pay for all those features, even if you don’t use them. I’ve seen incredibly robust LMSs that have awesome features that I would love to have, except that I don’t really need all the features. If you don’t need it, don’t pay for it.  You have to find the LMS that meets your requirements without too many extras you don’t need.

And don’t be afraid to tell the sales rep that their product is too robust.  You have to be honest with them; even if they don’t get the sale they will appreciate your honesty. Tell them up front what is most important to you and don’t get distracted by bells and whistles. You have to compare products based on how well they meet your requirements, not on the features they offer.

And the adventure continues…

My current research project of finding an LMS continues, albeit slowly. I’ve said before that my needs for an LMS are pretty simple – basic tracking, nothing fancy. All the systems I’ve looked at so far do more than I need and charge accordingly.

Then last week some people who are starting up an internal training program came to me asking what I use to track online courses. I have a homegrown LMS that sort of works for me, but definitely won’t work for them. However, now all those features of commercial LMSs that I don’t need may be relevant for the new internal training. For example, with external training we have one user group – customers. We don’t have learning plans based on job roles or skills matrices. We just have courses. Internally, we do have job roles. Everyone will need certain courses, but engineers might not need customer service training.

I want to help the internal folks come up requirements and then help them make a good choice. I figure there’s a 99% chance that whatever works for them will work for me. Working together always benefits all parties.

In terms of what I’ve seen lately, GeoLearning was the last product I looked at. The product looks great – lots of features, very robust, well thought out user interface, easy to use. The demo was not so great. I didn’t actually make it all the way through because the presenter’s wireless connection kept dropping forcing him to reconnect to WebEx. After the third time, I left. Fortunately for him I’d seen enough that I forwarded the sales person’s contact info to the internal training folks. Here’s a helpful tip for giving online presentations: Use a wired Internet connection, especially if you are presenting to eLearning professionals who probably know a few things about online presentations.

I was scheduled to attend a demo of Outstart’s LMS, but project deadlines got in the way, and my workgroup took a half day off to go wine tasting. Sometimes you need to relax to gain some perspective. The perspective I gained is that riding around in a limo and sipping wine is a good way to spend the afternoon.

The Great LMS Selection Adventure

Well, lucky me finally convinced people that we need an LMS. Now I just need to figure out which one will work best for us. About a year ago I started down this same path, but the project was put on hold. I’ve been in touch with one vendor from my earlier list, but I have a sneaking suspicion the landscaped has changed in the last year.

At the eLearning Guild Annual Gathering I was surprised by how few LMS vendors were there. I figured there would be a bunch of companies promoting their products. Maybe most vendors have hit a saturation point where they are living off the support/upgrade/maintenance agreements. Based on my research, ongoing support adds up to a lot of money, which is why I’m comparing prices over five years. Some LMSs have a relatively low initial cost, but high support costs. Others have high initial cost, but low year-to-year costs. Some are just downright outrageously expensive no matter how you look at it.

I’ve started doing my homework, starting with the eLearning Guild’s Learning Management Systems 2008 report and their 382 Tips on the Selection of an LMS. Both are great resources. I’ve been coming up with requirements based on our current processes and what can save us time. Requirements are tough to define. You can base them on what you currently do, but some things always fall through the cracks. Just today I came across Tracy Hamilton’s post My 300th post is a call for LMS Help which deals with managing classroom training in a LMS. She has some interesting issues with scheduling that I would have never considered. (Thanks Tracy!)

Then you have to consider what is possible, not just what you currently do. “What if…” scenarios can hold the keys to future efficiencies. Just imagine if about 5 years ago the record labels had asked “What if we embrace this peer-to-peer stuff?” how different the music industry would be today. I don’t want to miss future opportunities.

Then there is the money to consider. Last year I told a vendor that 90% of LMSs do 90% of what I need them to, so the biggest factor in selection is cost. If an LMS saves me $20K over five years but doesn’t have a couple of minor features, I’m going to save some money. Sorry, but that’s reality. Money matters most to me in this decision.

In my current search I’ve looked at seriously at Inquisiq, talked with the folks at SyberWorks, and looked into ClickCourse.  I have a list with a few others that I’m going to look into. So far, I like Inquisic because they put their prices on their web site and allow you to quickly setup up a trial, without spamming you. ClickCourse is in my budget, but I’m not sure it will handle the classroom training. I’ll be posting more as look at other vendors.

At this point I don’t want vendors contacting me. If you’re not a salesperson and you want to recommend an LMS, please leave a comment or email me directly. My email is in the right nav bar. I give a lot of weight to peer recommendations. On second thought, if you’re a salesperson with an LMS that costs less than $10,000 for the first year for 2000+ learners, you can contact me. I like hosted solutions because my IT group is busy with Office 2007 upgrades.

Is Moodle your LMS?

Last week I attended an eLearning Guild webinar about their recent release of Learning Management Systems (LMS) Report. The webinar was as much about their data analysis/display tools as it was about the LMS report. The data tools were very cool and allowed you to splice up the data in a thousand different ways. It would be great to have a dynamic tool like that for evaluation data. I could probably make the results say whatever I wanted. The coolest thing is that the data was live. The tool pulled the data from the actual online data base, so if people were answering the survey questions during the webinar, the results would change to reflect the new responses. The data parsing software is from Tableau Software, if you’re interested in checking them out.

Anyway, enough about the toys, on to the report findings. Unfortunately, I didn’t shell out the $1300 for the entire report (I think that was the cost they mentioned in the webinar). That’s just not in my budget, but I did read through the synopsis, which is available on the eLearning Guild web site. You might have to be a member to view it. Here’s the direct link to the PDF.

If you follow the LMS market, then the report won’t surprise you much. The big name players are all there with their market shares. What did surprise me was how much of an impact Moodle had on the survey. Based on this survey, Moodle enjoys a significant market share, even in corporate environments. For organizations with fewer than 5000 people and fewer than 5000 learner, Moodle is the most used LMS. That’s significant. Even if the eLearning Guild’s data has a wide margin of error, Moodle still has a large install base in smaller organizations. My theory is that these organizations don’t have the budget to spend on commercial systems, so they choose the most economical product – Moodle. Moodle also offers a robust feature set that rivals many commercial products, so not only do you save money, you’re likely getting most, or all, of the features you need. The synopsis also discussed return on investment, cost per learner, and satisfaction. Needless to say, Moodle scored very high in these categories.

So, what does this mean to the average eLearning practitioner? If you’re shopping for an LMS then you definitely need to look into Moodle. It has matured to a point were it legitimately competes with, and often beats, commercial products. While the software itself is freely available, you have to consider installation, configuration, and maintenance. Those are not free, and Moodle itself doesn’t offer any of those services. Moodle isn’t a company, but there are people out there that offer Moodle install/setup/config services. Moodle is open source, and thus has a huge support base that you can tap into to answer questions and get help. You can also customize it to meet your specific needs.

More importantly, because of Moodle’s success other open source tools could start to be used more widely across all organizations. Corporations and government agencies tend to be a little fearful of open source products, maybe because they don’t come with support. The truth is they do come with support, but that support comes from the community. The more people that belong to the community, the better the support becomes. That is one of Moodle’s strengths. Now that a lot of people use it, you can find help online and there are many books available as well. Wouldn’t it be cool if there were an authoring tool that had the same open source community? Your eLearning budget could focus on making better training, not upgrading tools every year. Software is a huge cost that can quickly get out of hand, not to mention keeping track of licenses.

I look forward to the day when my software budget is $0.