So you know you want an LMS. Good. This article isn’t here to tell you why you need one, or why you should use one, this is a guide for anyone who already knows they want an LMS, just wants a guide for what to check for.
Over the years I have spent many hours testing content and trying various different Learning Management Systems, and have even done some LMS (like) design work with Articulate Online. Over that time period I have had the opportunity to learn a lot about what does work well, and what doesn’t work well in a lot of systems, so based on my knowledge on the subject, here is my list 10 things to consider before choosing in an LMS.
1. What LMS Standard does it use (AICC, SCORM 1.2, or SCORM 2004)?
There are 3 LMS standards of significance these days, AICC, SCORM 1.2 and SCORM 2004.
A little history on the standards…
The AICC standard was developed by the aviation industry in 1993 and has evolved a bit over the years, but remains pretty similar to its roots.
The SCORM 1.2 standard was released in 2001 and is the most popular standard (in terms of usage) to date, and was actually developed by the department of defense. It was at least in part designed based on some of the limitations of the AICC standard. Although some consider to be an improvement over AICC, it still had many shortcomings.
SCORM 2004 was released in 2004 (shocker huh?) and in this specification they addressed some of the shortcommings and ambiguities of SCORM 1.2.
In the years that I have worked with Learning Management Systems issues have come up where the LMS expects one thing, but the content expects another. A big reason this has happened is that the early standards like SCORM 1.2 have left a lot open to interpretation. For example, certain versions of Saba will expressly ignore the status that the content sends to the LMS if the content exits with a “suspend” mode. With our content we expressly exit with a “suspend” mode when a user exits a course, which will normally allow a user to relaunch the content. So when you launch and complete content in Saba we expressly tell the content that the status is “completed”, but Saba ignores this because the status is set to “suspend”. Both sides are technically doing what is right, but still we fail. Find a solution to the Saba problem here.
There are other places where the earlier standards were just flat out missing something that should have been included. In the SCORM 1.2 standard there is no option for the content to communicate with the Learning Management System what the question was. It allows you to send an “interaction id”, but this is generally assumed to be an ID, and not question text. So the standard allows the content to communicate what type of question it was (t/f, MC, MR, etc), what the correct response was, and what the learner response was, but no way for us to communicate what the question was. In SCORM 2004 they improved this part of the standard by allowing us to pass the question “description” which allows us to communicate the question text.
Some of the older standards have some shortcommings based on the age of the standard. Back when AICC and SCORM 1.2 were developed, we weren’t in the same place that we are today in terms of computer processing power, bandwidth, etc. This caused the standards to be limited in what they allowed the content to send. For example, the SCORM 1.2 standard only allows content to send 256 characters to the LMS for a learner response to a question. So, if you have an essay question that you ask your users, any response they give over 256 characters would violate the SCORM 1.2 standard. Again, this is something that has been improved in SCORM 2004, where they raised the limit to around 4000 characters.
So as the standards have evolved, generally they have been improved along the way, with each newer version addressing shortcomings, ambiguities and limitations of the previous veresions. So make sure you consider what standard or standards (many support more than one) the LMS supports before making your purchase decision. If your LMS supports more than one, or all, GREAT!
Other things to note when thinking about standards:
- Most major content authoring tools support AICC, SCORM 1.2 and SCORM 2004
- If you often use custom developed content, make sure your content developer also can work with the standard of your LMS choice
- Beware of vendors that only partially implement a standard. Both the ADL (who controls the SCORM standard) and the Aviation Industry for CBT Committee offer certification for LMS vendors. If an LMS claims AICC, SCORM 1.2 and SCORM 2004 support, see if they have been certified as compliant.
2. What technologies are required by the LMS?
Ok, I admit it, I have had a love hate relationship with Java…the hate part stems for a Java class I took in college and 2 years I spent working at Sun Microsystems. The love part is that Java can do some really cool things cross platform (i.e. Screenr).
Java definitely has its place, but LMS environments is not one of them. See post and comments here for more info.
Every once in awhile I get a report that our content doesn’t work in some LMS running a specific version of Java (example 1, example 2). Why it works in every browser, and non-Java based LMS, but has issues with specific Java versions is a serious pain in the ass to figure out.
I could rant for another 2 pages on everything I think is wrong with Java in a LMS environment, but that is not the point of the article. The point is, look at what is required by the LMS to view content. Does the LMS work without any additional plug-ins or odd system requirements? Also check to make sure your LMS system requirements match the systems your users will be using. Do they support Safari browser on a Mac, do they support Firefox? Do they support IE6 and later?
3. How easy is it to get content up on the LMS?
One of the problems with many Learning Management Systems is that getting content up to the LMS takes a monumental effort. I kid you not, in some Learning Management Versions it is a 15 step process of getting content up to the LMS. Seeing that getting content up to an LMS was often a difficult task, the AICC board along with several eLearning experts developed the PENS standard (Packet Exchange Notification System). The idea was that the content authoring tool like Articulate Presenter or Captivate would notify the LMS that some content was available, and the LMS would then go pick up the content from the authoring tool. This sounds great, and is great, but the problem is that most content vendors and Learning Management Systems are slow to adopt new standards. It is kinda like the problem with clean fuels for cars, gas stations won’t get green fuel until cars start using them. PENS is an improvement, but it hasn’t been adopted by most LMS vendors or content vendors, so it is important to see how easy it is to upload content to the LMS.
Use Scripted Demos
Map out common things that you are going to need to do with the LMS, and ask your LMS Sales Representative to walk you through those tasks.
Some ideas for scripted demos:
- Upload content to the LMS
- Enroll a student
- Run reports on tracked content
These are day to day tasks that you are going to want to do, so you should know how easy or how difficult this will be before purchasing.
As a bonus, ask your sales rep to take content you have and upload it to the LMS and prove that it works.
4. Focus on ease of use of the features you need, not on the number of features the LMS offers.
We use to have a bugbase that we used at Articulate that was truely aweful. It was slow to load, took forever to log a bug, had poor text handling and would always throw exceptions, it was just a terrible bugbase for actually logging and tracking bugs, and those should be the core features. Every year the bugbase would release a new version and they would keep piling on new features that I didn’t care about (CRM Management, Case Management, external bug reporting system), without making the core features work. The company that made the bugbase was about competing on the number of features, not on making the core features work. On the complete opposite side of the spectrum look at the iPhone. A phone that has really high customer satisfaction year after year. The iPhone doesn’t win all feature comparisons with other phones, yet the iPhone continues to sell really well. The iPhone sells well not because it has more features than any other cell phone, but because the features they did build they built well.
5. Beware of the “home grown” LMS.
Too many support cases and forum posts begin with “Our in house developed LMS…”. As I mentioned above, the SCORM standard leaves a lot open for interpretation. It is naive to think that because there is a documented standard creating a SCORM based LMS will be easy. There are many companies that (like Rustici Software) have made an entire business out of doing SCORM implementation, and if building an LMS and doing the SCORM implementation was easy, they wouldn’t be in business.
6. Hosted or Licensed?
Do you need to license the LMS and host it on your own servers, or is it alright for it to be hosted by the LMS company?
There are benefits and drawbacks to both approaches.
Benefits of licensed
- You own the software, and likely won’t have to pay re-occurring fees for anything but maintenance (if you choose to get it).
- You control the security - if security is a big issue for you, licensing the software might be better than a hosted solution because you can control access to the LMS (through a corporate firewall, etc).
Benefits of hosted
- Usually less expensive - If you outright buy the software you will likely be faced with a bigger upfront cost.
- Don’t have to manage the servers - We for example host Articulate Online and Screenr at Rackspace. Why do we do this instead of owning our own servers? Well, Rackspace is the premier hosting provider in the world. They are better at setting up servers, better at dealing with security, better at managing load balancing, fire walling, etc. than we are. We handle the software part, which is a core competency of ours, and let Rackspace host the servers, which is a core competency of theirs.
- Updates are usually included for free – If you own your own server you may or may not be entitled to free upgrades. We have many customers that are stuck using AICC because they licensed their server years ago, and no longer get free updates. If you are on a hosted solution you most likely are always on the most up to date version of the software.
8. Is Open Source a possible solution?
I am not advocating this solution as the most painless, easiest option. There are going to be a lot of implementation costs, time cost, hardware cost associated with an Open Source solution to your LMS needs, but Open Source should definitetly be considered. The benefit of Open Source is the software is free. The downside is that most Open Source Learning Managment Systems that I have used can be buggy, and hard to deploy. I have only personally used Moodle and Dokeos (have used Moodle for about 4 years), and they have gotten better over time, but they still have some issues.
In addition to Moodle and Dokeos there are at least 10 other open source LMS options.
On another note, just because it isn’t open source, doesn’t mean they will have good support. I will refrain by calling out any vendors by name.
Also there are some vendors that do open source hosting and support for the fraction of the cost of a traditional licensed LMS vendor. Some names in this field are:
9. What kind of reports are available?
Rustici Software (SCORM.COM) points out in SCORM for RFPs for LMSs that:
SCORM can provide a wealth of data about what a learner does in a course, but it’s up to the LMS to keep the data and use it. You’ll want to know what kind of reports there are, how data is tracked and what the learner sees at a minimum.
Really good point here. Some Learning Management Systems do a really good job with capturing all the data, but don’t do the greatest job on reporting the data. For example, Moodle does a good job at capturing all the SCORM data, and will do a great job of reporting status and score, but gets lazy with the reporting of the interaction data (how people answered each question). If you want to see this in action, check out this post I did on uploading and tracking in Moodle.
A big part of the success of Articulate Online has been the richness of our reports (see this Articulate Online report presentation for more info). With SCORM 2004 reporting, the LMS should be able to generate reports that are just as rich as what Articulate Online offers, but few LMS actually provide such elegant reports.
A good idea is to list out what type of information you will need to gather in your LMS, and then ask the LMS sales representative how to get that data.
For example, you might need to:
- Know who has taken a quiz
- How that person scored
- What questions are people missing consistently
- What is your most popular contet
Make sure your LMS is capable of generating the data you need.
10. How well do they work with content vendors?
During our beta cycles I occasionally reach out to LMS vendors to ask if I can test our content in their LMS just to make sure that everything works ok. Most LMS vendors a really cool about this, and allow us to test in a trial or sandbox account. But occasionally some of the bigger vendors ask for a substantial amount of money ($2000-$5000) just for us to test content in their LMS. Seriously, and then they claim the charge is “nominal”. Anyway, ask your LMS vendor if they partner with any content vendors, or if the authoring tool you use has been tested in their environment.
SCORM.COM has a really good post on What to ask about SCORM in an RFP that has 7 questions anyone should ask of the LMS when getting a request for proposal and (at least) 27 questions to ask about your LMS vendor about SCORM. Highly recommend checking it out.
Do you have any suggestions that you think should be added to the list? Do you hate your LMS vendor? Tell me about it in the comments.
Note: Someone pointed out this list only contains 9 items (I skipped 7). Going to keep it titled a list of 10 though, because 10 reads better than 9 Sorry to short change you by 1.