Quality QA
Licensed CC by benwatts

Over the last couple of months I have been asked to help a university (that will remain nameless) in its transition to a newer version of of its Learning Management System (LMS). As part of this I have had to access many LMS course spaces to check that content has migrated successfully and that that things are working as they should.

It has been a profoundly depressing experience. I knew it would be and you’ll appreciate why I knew if you look at my current full time occupation.

Let me begin begin by saying that there are a few dirty little secrets about online learning at traditional universities. Here are two:

1. Not many courses have any form of content online whatsoever (even when the university promotes a policy of minimum online presence).

2. When a course does have online content it is invariably rubbish.

Here are some things that typify the courses I have seen recently:

  • Poorly structured sites with no narrative, no instruction, no guidance for the student.
  • Un-capitalised headings and grammatically incorrect sentences in mixed and garishly coloured fonts.
  • HTML using deprecated tags and with strange symbols that are presumably the remnants of copying and pasting from MS Word.
  • Links to PDF, DOCX and PPT files that are often way too large with no indication of the size or file type before the user clicks on the link.
  • Discussion forums that are not used or are used in very strange ways indeed.
  • Student blogs where the student is expected to print their blog out before submitting it for assessment. Although it should be said that I’m impressed that course was using blogs at all because I reckon only maybe 2-3% of course sites would have a blog set up.

But worst of all perhaps is the content itself. I had the misfortune to visit one site that was being used to teach web design. You’d think that this site would be really good but sadly no. It displayed many of the features I’ve listed above but worst of all the content of this site was at least three years old and made reference, in many cases, to information that was 8 or 9 years old. From this site I learnt that Alta Vista was the third biggest search engine and Netscape Navigator 4 was a major browser. If this was a human anatomy course then older content might be fine but a course like web design needs to be updated almost monthly the pace of change is so rapid.

To put the icing on the cake, this course was a wholly online course being offered to students as part of an Information Systems degree. I kid you not.

All of this brings me to the title of this post and the observation that there appears to be no quality assurance applied to elearning at most universities. The only surprising thing about this observation is how little it is spoken about. Senior academic managers appear to be totally unconcerned.  I suspect they don’t know and probably don’t want to know. If they knew about it then they would have to do something about it and then all of a sudden you have to deal with academic staff who cry ‘academic freedom’ at the drop of a hat and we all know how much fun that is.

What has happened is that academics have been allowed to continue into the online environment their thousand year old practice of engaging in a secret communion with students that happens in the classroom . I remember, as a young course leader at a university in the UK, being asked a question by a very respected Professor and external examiner (now there’s a novel thing). He said “Mark, how do you know that the lecturers are teaching what you want them to teach in the classroom?”. I, of course, had no idea so I mumbled something about outcomes and assessment. I could have said to my teaching staff that I thought it would have been a good idea to have some peer observation of their teaching but that would have gone down like lead balloon.

And there is the problem. Academics don’t like non students (dare I say, people who are not beholden to them and therefore are more likely to be critical) in their classroom and they don’t like them in the online course spaces.

Of course a really brave Vice Chancellor or President would say, right we are going to have all of our content as open courseware in five years time and the first thing we are going to do is allow every member of the university, staff and student, to be able to see the content in every course. Not sensitive data but just the learning content itself. We’re going to identify and reward the staff members working on the best courses so that everyone can see good practice. Finally we are going to provide course development resources to help you transition your content to an engaging online format and to ensure it is only made available when it has been through an appropriate QA process to meet some mutually agreed standards.

Such academic leaders are few and far between. Many (most) senior academic managers don’t understand online learning at all. I remember being at a meeting of Deans of Faculty who all thought that uploading Powerpoint files was the same as blended learning.

In the meantime student expectations increase continually as they engage with high quality content from many other providers as part of their daily online activities. Something is going to give and it may be sooner than we think.

29 thoughts on “eLearning at Universities: A Quality Assurance Free Zone?

  1. Hi Mark.

    I’ve been Unit Coordinator for less than 6 months and right now am brushing up my Blackboard units ready for release to the students. I have the choice between making things bad for me (ie. spend extra hours and time away from my family trying to remediate what is essentially a bad system) or bad for the students (ie. leave outdated content presented in a way that is better than most online units, but still far from usable, clear or pleasant for the students).

    I am not surprised by your assessment of the units you audited, but I wish it was otherwise. I have not completed formal training in education and am new to this game – but after spending a couple of days at my university’s teaching course for new academics, I am very surprised and saddened to find that I am ahead of most of my colleagues in providing effective online learning. Something is very wrong with in our university system when this is so.

    Right now I am updating a Blackboard unit that is about to be duplicated so that it can be offered via Open University. So I will have content in two separate Blackboard instances being run at the same time. I have inherited unit materials that are all uploaded WORD docs. Once the unit has been copied, if I want to make a change – yup – change the WORD doc and upload it to two separate instances and hope like heck that some eager student is not using the outdated material because they have already downloaded it. So – my choice at the moment is 1) whether to convert all the WORD docs to HTML and upload them to a WordPress instance and make it visible in frame in Blackboard ( a lot of setup, but then pretty breezy) 2) Leave it as in (more time with my family – except every time it needs updating…).

    One sixth of my students are internal and the rest are all external or Open University students, so effective online learning is not just a nice extra for my job, but an essential core…

    Reply
    • Thanks for the comment Kathryn. I can tell you that your reply has disturbed some well known and respected educational technologists who have commented to me privately about it. I think the last sentence of your second paragraph is particularly telling.

      If I was in your position and lecturing again I would be inclined to gradually move content from your two sites to a public blog and I would license it using a Creative Commons Attribution license (http://creativecommons.org/). I would check with your University copyright office but if they didn’t let you do this then I would question whether you were actually working for a university.

      I do believe that, in time, academic staff that do this will be recognised and rewarded. Maybe not by their own institutions in the first instance but by others and those rewards will be tangible. It is worthwhile looking at the toolkit page for the open courseware consortium (http://www.ocwconsortium.org/community/toolkit) to see what the benefits are for the individual over and above the joy of altruism.

      Reply
  2. It seems a convenient truth that despite years of personal, focused study to finally be accepted as an academic that there is no requirement to have any teaching experience or qualifications. Yet they judge us – unable to teach us.

    It’s clear many are not interested in it – from personal experience – they rarely reply to emails, offer no feedback during the course and seem to spend lots of time away at conferences or whatever they do.

    They charge us the same, regardless of the teachers ability. It’s a total lottery.

    But if you want the paper, you keep quiet, upload your essays and comply. We teach ourselves, we use Facebook and other places. I’ve given up trying to work harder – its a vacuum. What’s the point. Then there’s the marking – totally random. We know they use the bell curve and only hand out so many HD lollies – it’s retarded. What’s the point, the odds are the most people will get Ds and Cs anyway.

    Reply
    • Thanks for the comment Wilma. I was particularly interested in your point about teaching yourselves. I think that this will happen more and more and, in fact, should be encouraged. I don’t think a lot of stuff at universities need be taught at all. I do, however, think that academic staff should be able to provide a framework for student learning, provide feedback and guidance and be able to assess learning rigorously, consistently and authentically.

      Reply
  3. Not surprised by your findings Mark. eLearning as well as learning in the traditional face to face environment is a ‘quality assurance free zone’. When I introduced some eLearning standards and a review process for all courses with online presence at my institution some academics threatened to resign others organized meetings in protest and made submissions to the academic board to stop such standards and process being approved. Interstengly enough the main argument was ‘ we do not have such standards and processes applied to our classroom teaching why should we have them for the online or blended courses’. It took a lot of energy and time spent in workshops and consultations but after 3 months the standards and quality review process were approved by academic board and are part of the institutional quality managment. A year and half later I see some academics start to appreciate the positive feedback they receive from their students and a programme area with significant number of courses with online presence is planing to review all existing courses and redevelp them to meet our standards.

    We are still a long way from having all courses with online presence meeting the basic web standards and presenting content that is up to date but I’m hopeful as we have started on that journey.

    Reply
    • That’s excellent Vasi. Your institution will be much better placed for the future because of all that hard work. It is my expectation that those institutions that can put these sorts of QA systems into place will be the winners over the next 10 to 15 years.

      Reply
  4. 10 or 15 years ago I taught within an institution that had a traditional print-based distance education operation. A very strong central division was responsible for the quality control of all print-material that went out. But there were at least two main problems.

    First, it encouraged consistency to some fairly arbitrary bureaucratic standard that eventually got taken to extremes so that the forest was lost for the trees. e.g. the print standard specified that certain bits of text should be 11.5 font, a decision based on very specific research. It also led to the situation where the standard suggested that all quotes in text should be double quotes. The central division got to the stage of automatically changing text that didn’t meet the standard and subsequently really screwing up some printed notes for a Prolog programming course.

    Second, it didn’t address the out of date material. One of the best course – in terms of meeting the standards – was on one computer hardware that was focused on the 8086/8088 architecture in the late 1990s.

    The presence or absence of academic qualifications or experience in education/teaching was also not a significant factor. Some of the worst material was coming from folk within the Faculty of Education. Something that is still happening today.

    Some of the best material I saw was coming from recent graduates with no formal educational qualifications. What they did have was that they cared about their students and were prepared to go above and beyond in terms fo their teaching. Each and everyone one of these folk are no longer working within the university sector and with the ERA cloud would probably never get back in.

    IMHO, formal QA systems and requiring formal education qualifications of academic staff always struck me as examples of Level 2 attempts at improving learning and teaching. They were options that allowed senior management to run around doing things, holding meetings and generating formal, objective statistics to show what they were doing rather than actually addressing the real, fundamental issue. i.e. that the university environment does not encourage or enable quality teaching.

    Reply
  5. Thanks David. I completely agree with your reply. Particularly the last paragraph. I would not want to see a rigid, autocratic QA system imposed that insisted that every online course is presented in exactly the same way. Mostly what I am talking about is a) presenting information in a professional way and b) making sure that the content is current. This is just common sense and, as you point out, is not related to qualifications or, indeed, experience. Unfortunately, as the saying goes, common sense ain’t so common.

    PS on a tangential note, being sans PhD I know I wouldn’t get another lecturing position at a university. An impediment that doesn’t seem to prevent many senior academic managers from both continuing up the ladder and dictating university professional entry requirements.

    Reply
  6. Mark, this is an interesting, unsurprising, and yet saddening post.

    I teach in the same content space as Kathryn, at a different Australian institution. Like Kathryn, I’m a new academic, heading into my second year of teaching full time, after only one semester as a sessional staff member.

    I see three underpinning issues here.

    Firstly, like Wilma says, there is no requirement for teaching academics to have teaching qualifications. We are under immense pressure (and get significant support) to get ourselves qualified as researchers, but that is only one part of what we do. Being a good researcher does not necessarily mean you’re a good teacher. In fact… No, I won’t finish that sentence! This is a multi-faceted job: we teach; we research; we provide service to our institutions and our professions. We should be equally as qualified to carry out the teaching and the service. I feel I’ve got the latter covered, but I do feel at a disadvantage as a teacher for not having a teaching qualification. Luckily for me (and my students), I had the benefit of participating in an early career academic development program in my first year as an academic, and this really helped me develop my teaching skill set. At my institution, mandatory teaching qualifications are on the radar, and, in my opinion, so they should be. My intention is to get some traction with my PhD and then to embark on a Grad Cert in Higher Ed. It’s a start, at least.

    Yes, we should be teaching ourselves about emerging tech and how tech can be used in our teaching. And yes, I’m proactive and I have alerts set up for interesting content on pedagogy and technology applications in blended learning (plus a pile of textbooks on my desk). But there is fundamental theory of education that you need a structured learning environment to get familiar with – or at least I do. And I think we need to be certified as teachers. Don’t our students deserve that? I just do not understand why we have to be qualified to research, but not qualified to teach. What does that say about how the system values the teaching aspect of our jobs? I love research, don’t get me wrong. But I also love teaching, and at the end of the day, I’m turning out graduates that need to be able to do a job. I better make bloody sure I know how to teach them to do it. And knowing the content inside out and back the front is not enough.

    Secondly, find 10 courses that have moved from a face-to-face mode to an online learning mode or a dual mode cohort environment, and you can bet your bottom dollar that 9 of them didn’t have any funding (or anywhere near enough) to do it. Online learning is different to face-to-face learning. Redesigning courses for the online environment takes time and teaching in this environment requires new skills, which can sometimes mean retraining. Add in the complexity of a course where you have a combined face-to-face and online cohort (as both Kathryn and I do), and you can double the amount of work involved. Virtually every learning activity needs to be designed twice. If universities don’t resource their teachers to step back, take stock, and holistically redesign their courses for the online environment, then we’re going to end up with shitty course sites, shitty content, and shitty learning activities. I’ve been fortunate to have had a faculty funded teaching and learning grant to undertake research this year to develop a whole-of-course framework to underpin our approach to what we are calling blended learning for a dual mode cohort. We’re lucky. Others aren’t. But I should also add that we’re committed to teaching and learning and T&L research, and to being evidence based educators. And that means we spend a lot of our own time on course development, grants or no grants.

    Thirdly, we are at the mercy of crappy tools. Kathryn and I have been tweeting in solidarity these past couple of weeks as we both hand code every piece of content for our BB sites. I migrate nothing from my existing sites because the spaces I teach in (web content management, emerging tech for service delivery, and even good old library collections management) are far too dynamic. Every bit of content gets reviewed. Every bit of content gets hand coded. Last semester, I ran my course site on an install of WordPress with BuddyPress and my students blogged through the semester on their own blogs. It worked like a dream. The WYSIWYG editor works. I can schedule content for publication. Embeds work. Students can collaborate. Everything works! Six months without BB made me forget what a pain it is to work with. My site setup time has increased exponentially compared to the time I spent on my WP+BP site, and that included the time it took me to figure out what plug ins I needed, how best to administrate the site, and rethinking our usual learning site design. Recently, I’ve been working with Moodle for a side project, and I have to say, I dislike it just as much as BB. If we’re going to build decent sites, we need decent tools.

    Reply
  7. Pingback: Posts vía RSS » Blog Archive » eLearning at Universities: A Quality Assurance Free Zone? | Mark Smithers

  8. Thank you for your excellent reply Kate. In fact I’ve noticed that the comments to this post, including yours, are being referred to quite a lot in my Twitter stream.

    I agree with most of what you say although I think we could have an interesting debate about the use of the word ‘teaching’. I think that there are some things that do need to be taught in the traditional sense in universities but I also think that a lot of what we ‘teach’ can be learned by the students themselves in better ways.

    The point you make about funding for transitioning courses into online or blended delivery is an important one. Fifteen years ago I was a subject matter expert for a course (unit/subject) being developed for traditional print based distance education delivery. The budget for development was roughly $50K. It seems as though all of that ability to provide resources for online course development has just disappeared.

    Finally, with regard to tools. If you are happy running your course using WP then why not continue? Other universities are seriously considering similar things. Have a look at the work of @jimgroom at University Mary Washington.

    Reply
  9. Hi Mark

    Depressingly, I concur with many of your remarks. I have access to a few VLEs across the HE sector and although I can find some excellent provision, these are few and far between – there’s a long way to go in terms of e-learing in this sector! One specific example is a handbook from a local HEI that states that e-learning will be supplied by tutors emailing students!

    However, over the past few years I see signs of improvement such as academics being promoted partly on the basis of their approach to e-learning – proactively critically questioning their colleagues approach to online learning – taking much of the e-learning debate on-board (challenging outdated content and references and encouraging the creation of an on line environment that is content rich, engaging, has suggested / required activities, uses scaffolded online discussions etc.) Further, I have recently discovered that senior management are scrutinising academic proposals for new / updated courses and taking particular interest in the e-learning aspects – rejecting applications that are too bland and insisting on a tailored approach to TEL, adapted to the objectives of individual units. Also, I hear that institutions are embedding ‘good’ e-learning practice in course documentation and are giving TEL units real teeth to attack poor practice.

    Echoing Kate’s second point I found during my masters research that teaching staff aren’t given the time or resources (by management or their peers) to develop or update materials. Also, echoing Kate’s last point, some of the VLE/LMS systems can be slow and cumbersome – reminding me of the earlier word processors on the computer systems of the 1980s (such as Display Write on the PC or Text Editor on the IBM 360 mainframe: http://www.youtube.com/watch?v=LjMJ99MuFSg). If learning systems will improve as much as word processors have over the last 30 years then e-learning is still in the stone age.

    My vision is that technology will make teaching provision far more visible, students will be attracted to the best teaching, and institutions will be forced to create qualifications out of units from a variety of sources, both educational and from experience, perhaps a development of my notion of the ‘microversity’ (http://dbcallaghan.blogspot.com/2010/01/microversity.html). However, there are some very powerful vested interests that will manage to maintain the status quo for a while yet (ask anyone who’s been trying to use APL routes in the UK HE sector recently, let alone the poor soles tasked with getting the Bologna Process implemented here).

    Finally, a question: What will be the best form of qualification for this century:
    1) A degree from a university
    2) A portfolio of evidence demonstrating that you do X, Y or Z?
    … and what type of organisation is best placed to provide that?

    Reply
    • Hi David,

      Thanks for your thoughtful comments. I am glad you have seen signs of improvement. I unfortunately haven’t. That is not to say that there isn’t some exciting and innovative work going on. There is but it is nowhere near being mainstreamed at the moment. It will take a fundamental shift in the attitude of many university academic staff that says that they are the masters of what they ‘teach’ and they will ‘teach’ it their way. The vast majority are (or should be) subject matter experts first and foremost not delivery experts. They will have to get used to working in a team to deliver it effectively digitally.

      I totally agree with your penultimate paragraph and I hope that will happen.

      With regards to your final question I have to say option 2. In many ways that is the case already. My portfolio of evidence up until a few years ago was my resume. Now it’s my resume and this blog and my twitter stream and my comments and all of those little things that go to make up a social, digital reputation.

      I also think that a first degree at 18 will become far less important (please let it be so). I make the observation that in my case that I have a first degree in Building Construction. That degree hasn’t been a requirement for the job I was in at the time for at least the last 11 years. I now have a reasonably senior position managing ed tech systems for a very large university. Did I really need my first degree?

      Cheers

      Mark

      Reply
  10. Hi Mark

    Just a quick follow up: I’m not using WP this semester because my use of it in the past has been *ahem*… shall we say unsanctioned? The unit I taught last semester was about using social media for service delivery and therefore I had a strong case for using social media to deliver it. I feel that my justification for using WP for other units is on shakier ground, in that my main reason for wanting something other than BB is that I dislike it immensely. Technically, there’s nothing I want to do in BB this semester that can’t be done there, although it will take more time to achieve it and it won’t look as slick. I guess, as a new academic (and one on a contract!), I’m trying to choose my battles wisely (not something I’ve ever been good at in the past – I’d generally rather fight *every* battle). Also, I’m conscious that if I don’t ask the university for the tools I want, they’ll never know I need them. So for my next WP based course site, I’ll be logging a job asking for a university hosted instance. We shall see…

    Oh, and BTW, I probably do agree with you on the ‘students can learn in other ways’ front. I suspect a lot of my ‘teaching’ is probably not what the academy would see as teaching. My ‘lectures’ tend to be discussions and my set readings tend to be *gasp* blog posts and news stories. I think in a lot of ways, though, I’m still really forming my whole philosophy of teaching. So let’s have that discussion another day!

    K

    Reply
  11. Hey all. I am a recent graduate, and through my studies have become interested in educational technologies and the advances within the industry. Invariably, I was forced to take at least some online courses, and I wholeheartedly agree with the view that there a general lack of content, or if there is some sort of scraps of content within a platform, it is either cumbersome to access and review, lack grammatical and structural composition, and forums/boards are generally misused, if used at all. I would be interested in getting more involved in elearning (teaching is a passion, as is computer technology. A merging of the two would be perfect). Do you think attending an e learning
    conference (linked is the E-Learning Summit’s list of speakers/sessions. It is in Washington D.C.) would serve as a good framework for breaking into the field as a developer/instructor? Thanks.

    Reply
  12. Pingback: So, What do we do about the poor content in higher online education? « Ask, So What?

  13. Pingback: Idea Organizing: So, What can be done about poor quality content? « Ask, So What?

  14. Pingback: Learning technologies – should the tail wag the dog? | Join the dots

  15. Pingback: The dissonance between the constructivist paradigm and institutional e-learning « The Weblog of (a) David Jones

  16. Hi. I’d like to share a bit of good news. I’m from Victoria, BC Canada and I’ve worked for 8 years at a small but progressive university delivering professional and applied graduate degrees – Royal Raods University (http://www.royalroads.ca) I’m an instructional designer and it is my job and unit’s mandate to ensure course quality. We’ve set up a set of quality standards and conduct quality checks as part of the course development process. Here is more info on our practice: http://faculty.myrru.royalroads.ca/ctet/establishing-quality-review-online-courses

    I’m glad to say that I don’t come across bad course web sites – well, I get to fix them. And our faculty knows what we expect to be a good online course.

    Reply
  17. Aside from training people to teach effectively is the whole question of training people to use IT effectively in teaching and learning. Any IT training teachers and lecturers get tends to be training that was designed for secretaries, yet even with the basic tools of Word and PowerPoint (or their equivalents) you can create engaging, interactive, collaborative resources and experiences. More to the point, effective and imaginative e-learning can save significant time for tutors as well as learners but only when you get the hang of what works well and that requires time to be trained by people whose technical skill is allied to teaching skills. For a flavour of the types of easy interactive resources even Word can produce, have a look at the JISC TechDis ITQ for Accessible IT practice where the focus is on inclusive, creative use of everyday tools.
    If we train tutors with ECDL and CLAIT style courses we can’t really complain if the things they end up doing with IT are unimaginative.

    Reply
  18. Teacher, let me tell you that if this is your reality, our here in Colombia is not so far from you, and I think that here, for sure, it`s worst that the level you refer, because over here universities are still closing, hiding their coursesand their contents in order to not to give away their “know how”… the problem these universities are going to have is when students learn how to learn… although it problably will delay a little bit more!!
    Thanks for your report!

    Reply
  19. Mark,
    Pick two students in your class randomly and identify 10 differences between them with respect to teaching and learning. If you can do that you will know
    1. Why LMS fail in a college environment,
    2. Why LMS succeed in business environment, finally
    why teaching is a secret commune.

    Cheers,

    Reply
  20. Pingback: eLearning at Universities: A Quality Assurance Free Zone?Mark Smithers | Mark Smithers | Flexibility Enables Learning

  21. Pingback: Making e-learning tools that are more supportive – BIM, TPACK and truncated feeds | The Weblog of (a) David Jones

  22. Hi there just wanted to give you a quick heads up. The words in your article seem to be running off the screen in Safari. I’m not sure if this is a formatting issue or something to do with internet browser compatibility but I thought I’d post to let you know. The layout look great though! Hope you get the problem solved soon. Kudos

    Reply

Leave a reply

required

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>