Public LMS Evaluations

Many universities are currently, or have recently, reviewed their enterprise LMS including my own employer. Unfortunately we haven’t made our review process public but many universities have been generous enough to provide information about their reviews. The following list are some of the publicly available LMS reviews that I have found. These were really useful in developing and informing our own review and I hope that by bringing them together in one place it will be useful for others.

I have provided links to each institutions review documents and a quick summary of their process. I have included the eventual choice of LMS but of course this is a very small sample and shouldn’t be taken as being indicative of any particular trend in LMS selection. The list is not presented in any particular order.

If you know of any other publicly available LMS review documents please let me know and I will add them.

University of Canterbury, New Zealand

Information Site: http://uctl.canterbury.ac.nz/learn-moodle/review-process-and-documentation

Time Period: October 2007 to October 2008

Summary: An institutional merger meant that there were two different LMSs to support; BlackBoard and Interact. There was a desire to move to a single product. A short listing process was used to identify Moodle as a potential alternative to BlackBoard. A subsequent process was used to select between Moodle and BlackBoard.

The process was as follows:

The Steering Group used a decision process based on the following criteria: usability (by teachers and students, informed by findings from the trial); operational (reliability, external support, flexibility and features); strategic (future proofing, adaptation, stability of provider and marketing); cost (transition and long term); and risk. The relative importance of each criterion was agreed.

Selection: Moodle

North Carolina Community College System

Information Site: http://oscmoodlereport.wordpress.com/

Time Period: May 2008 to August 2009

Summary: This very detailed report describes the assessment of Moodle as an alternative to BlackBoard as the NCCCS LMS.

This conclusion was reached by a team of NCCCS staff using three independent research techniques – each of which verified the findings. The three techniques included functionality comparisons, end-of-term survey results by both instructors and students, and case studies of four NCCCS institutions that have migrated from Blackboard to Moodle.

Selection: Moodle

University of North Carolina – Charlotte

Information Site: http://www.lmseval.uncc.edu/index.php?option=com_docman&task=doc_download&gid=39

Time Period: January of 2008 to May 2009

Summary: The comparison was between BlackBoard and Moodle. It focused on:

  • pedagogical considerations
  • compliance considerations
  • financial considerations
  • open source considerations
  • The pedagogical evaluation is particularly interesting.

Selection: Moodle

University of Notre Dame

Information Site:http://www.ebuford.com/CMS%20REPORT%20FINAL%20PUBLIC%20REPORT%20V1.pdf

Time Period: May 2007 – February 2008

Summary: The following is taken from the review report executive summary:

The comprehensive CMS evaluation process consisted of a series of functional and technical activities including vendor presentations, semester-long pilots of two alternative course management systems, formal faculty and student usability studies, student surveys and faculty focus groups. In addition in-depth technical analyses were conducted to evaluate how each CMS would integrate into the university‟s student information system, how extensible and flexible each system was for future development, and how well each CMS matched the vision for CMS use at the University of Notre Dame.

The following systems were evaluated: Angel Learning Management Suite, Blackboard Vista, Sakai.

Selection: BlackBoard Vista

University of Northern Texas

Information Site: http://web3.unt.edu/lmseval/index.cfm

Time Period: March 13, 2007 and April 30, 2008

Summary: This is a very comprehensive LMS review site with lots of detail. Four products were evaluated; Angel 7.2, Desire2Learn, BlackBoard Vista 4.2 and Sakai. I really liked the way that the evaluation was distributed amongst ‘Core Implementation Teams’ so that all of the stakeholders can contribute in their area of expertise.

There were five stages to the evaluation:

  1. Initial research
  2. Identification of base criteria
  3. Weighting of the criteria
  4. Identification of stakeholders
  5. Development of proposal
  6. Selection: BlackBoard

McMaster University

Information Site: http://www.ltrc.mcmaster.ca/lmseval/

Time Period: December 2006 to April 2008

Summary: The university was looking for a replacement for WebCT CE 4.1. The evaluation was organised in 8 stages including:

  1. RFI and product qualification using over 450 checkpoints
  2. Summative evaluation from the stakeholder evaluation committee.
  3. Selection of qualifying systems
  4. RFP to a selected shortlist of vendors and a software trial
  5. Evaluation of faculty/student feedback.
  6. Detailed recommendation by the evaluation committee including cost analysis.
  7. Pre-deployment of the LMS.
  8. Implementation, training and migration.
  9. The following systems were evaluated:

Learning Environment (Desire2Learn), eLearning Server (Intrafinity), Vista (Blackboard), Moodle (Open Knowledge Technologies) , and FirstClass (Palantir Information Systems)

Another, very comprehensive LMS review process.

Selection: BlackBoard

Australian National University

Information Site: http://information.anu.edu.au/daisy/infoservices/1804.html

Time Period: 2007 to November 2008

Summary: Consultation and a review process lead to a the identification of a set of requirements. This, in turn lead to an RFI from providers, a refinement of the statement of requirements and then an RFP.

Selection: Moodle

Louisiana State University

Added: 2009-09-22 thanks to Michael Feldstein

Information Site: http://appl027.lsu.edu/itsweb/cmsweb.nsf/index

Time Period: 2006 to Fall, 2007

Summary: The evaluation was as follows:

Five CMS were selected for formal evaluation: Angel, Blackboard, Desire2Learn, Moodle, and Sakai. Responses to the Request for Information were obtained from all five (Appendix F).The three commercially licensed Course Management Systems (Angel, Blackboard, and Desire2Learn) were set up for hands‐on evaluations by their respective vendors. The two open source candidates (Moodle and Sakai) were hosted internally in a virtual server environment within the Frey Computing Services Center. A common demonstration course was set up on each CMS to evaluate the systems’ features. Ninety volunteer evaluators were given instructor and student accounts to evaluate each system against a common rubric (Appendix G). Evaluators were split into two groups and each group was given three systems to evaluate, with Blackboard being the common system evaluated. Sixty‐one completed evaluations were
returned.

The University community was invited to live demonstrations of each of the five candidate CMS. Over a period of two weeks in early June 2007, each system was presented to the campus community. Each vendor was allotted time for four (4) fifty (50) minute presentations and an open lab for a period of eight (8) hours. The vendor presenting for Sakai was Unicon. LSU ITS personnel presented the Moodle CMS, due to the simplicity of implementing a working evaluation system. The open lab hours were allocated to promote hands‐on experimentation
with each system.

Selection: Moodle

Central Queensland University

Added: 2009-11-30

Information Site: http://appl027.lsu.edu/itsweb/cmsweb.nsf/index

Time Period: 2006 to Fall, 2007

Summary: The evaluation was as follows:

To inform our own input into the broader process, CDDU will

  • Select a small number of courses
  • Develop a description of the models/abstractions embodied in the courses and in the relevant LMSes (both the LMS they are currently in and will be moved to)
  • Port the courses
  • Identify common practice in existing online courses.
  • Summarise and disseminate our findings with each system.

The following systems were evaluated:

  • Moodle
  • BlackBoard
  • Sakai

Selection: Moodle

Queensland University of Technology

Added: 2009-11-30 thanks to @heyyeah (see comments)

Information site: http://www.tils.qut.edu.au/initiatives/learningmana/OLTReview.jsp

Time Period: Oct 2005 – June 2006

Summary: Review of commercial and open source LMS to a highly customised in house LMS. At the time of the decision in 2006, it was felt that the open source LMS did not match the functionality of the selected LMS.

Selection: Blackboard

University of Minnesota

Added 2010-05-20

Information site: http://webct.umn.edu/ and  http://www.oit.umn.edu/cms-search/index.htm

Time Period: Oct 1009 – March 2010

Summary: Compared BlackBoard V9 and Moodle to see which provided a better successor to WebCT Vista.

Selection: Moodle

Other Resources

The following resources are also useful when considering a review of the LMS:

EDUCAUSE – Selecting and Implementing a Course Management System for Your Campus

Electronic Journal of e-Learning – Learning Objects and Virtual Learning Environments Technical Evaluation Criteria

LMS Comparison across Ontario Universities

Summary

These are just a brief selection of publically available LMS review sites and documents along with a few other resources. Of course none of these resources address the question about whether we need to have an institutional LMS/CMS/VLE. That is a different question and one that I will talk about more in future posts.

In the meantime please feel free to contribute your resources in the comments.

34 thoughts on “Public LMS Evaluations”

  1. Hi Mark,

    Thanks for this great list. One request. Could you change the title for UNC to say University of North Carolina – Charlotte? That is the evaluation report to which you are pointing. Our friends down on Tobacco Road are piloting Sakai and are usually referred to as UNC or UNC-CH (UNC – Chapel Hill). Usually when someone just refers to UNC they mean UNC-CH, Tar Heel blue. 🙂

    Thanks,
    Neal

  2. Hey Mark – this information was extremely valuable to me. I am currently conducting an evaluation for a small for-profit university and based upon my research I am looking at pretty much the same LMS vendors as many within the studies you have provided. Would you know where in any of these studies or anywhere else for that matter I can find a side by side comparison of the most important product features and funtionalities for Blackboard, Desire2Learn, Moodle-Comcourse, Sakaim etc.

    1. Hi Rick,
      Sorry for the delay in replying and I’m glad you found this post useful. I believe there are some such comparisons carried out by commercial organisations that are available for a fee (normally a considerable fee). Having said that I do think that these are of limited use because it really comes down to what is the best fit for the organisation. The core functionality of the major LMS offerings is similar but there are differences at the edges that may sway your decision. Things like the degree of customisability of the system, integration issues, degrees of openness, vendor reliability, hosting strategy. This point was well made at a forum I attended last Friday at which several Australian universities described their LMS evaluation processes and the key criteria they used for selection. I have a post planned reporting on this forum that I hope to publish soon.
      Cheers

      Mark

Leave a Reply

Your email address will not be published. Required fields are marked *