Quality Matters Review of LIB 2210: Applying the QM Rubric for Higher Education to an Information Literacy Course

Abstract This article discusses the use of the Quality Matters Rubric for Higher Education, 6th ed. to self-review an asynchronous information literacy course. In this instance, the authors acted as reviewers and subject matter experts (SME). The Rubric proved to be a useful tool that encompasses several instructional design best practices. After the official self-review period, the QM Rubric influenced the enhancement and updates to the course. The authors recommend more use of the QM Rubric by teaching librarians, collaboration with instructors going through a QM Review, and training for online teaching in Library and Information Science graduate programs.


Introduction
Quality MattersTM (QM) is a nonprofit organization "specializing in standards, processes, and professional development for quality assurance in online and blended learning" (Quality Matters, 2018, p. 2).The Maryland Online, Inc. consortium created the QM Rubric for Higher Education (hereafter known as the "QM Rubric") based on research-supported instructional design best practices for online learning experiences.The QM Rubric reviews the design of a course, rather than delivery or academic content.Evidence of subject faculty involvement in QM professional development and use of the QM Rubric for course development and improvement is widespread.
In this article, the authors will provide context for how academic librarians have incorporated the QM Rubric into their teaching and demonstrate how they reviewed an asynchronous, credit-bearing information literacy course (Introduction to Library and Online Information Research), using the QM Rubric.This informal review helped the authors integrate online pedagogies into their teaching practices and engage students more effectively in online learning environments.During the self-review of the course, the authors found that the QM Rubric incorporated existing best practices of instructional design, such as Backward Design, Community of Inquiry, and Universal Design for Learning.Additionally, the QM Rubric influenced the continued enhancement of the course, as well as the authors' online teaching pedagogy.

Academic libraries and quality matters
Educators have published extensively about the QM Rubric's intended purpose as a quality assurance tool for improving course design in online and blended learning environments.There is also evidence that participation in QM professional development workshops impacts pedagogical practices across delivery formats (Kearns & Mancilla, 2017).
For their part, academic instruction librarians' use of the QM Rubric has been more varied, applying it to online tutorials, workshops, learning management system modules, and more.This could be explained by the fact that information literacy instruction is far more likely to be delivered in one-shot sessions or at a student's point of need, rather than in credit-bearing courses.
Some instruction librarians have used the QM Rubric while developing and improving online or hybrid credit-bearing courses (Coaplen et al., 2013;Loesch, 2011;Newby et al., 2014;Yantz & Lef, 2018).However, the most common use of the QM Rubric by teaching librarians has been developing or improving instructional materials that support students at their point-of-need.Reeves et al. (2016) detailed how they used QM Rubric in the development of six self-paced, stand-alone online information literacy learning modules at the University of Toledo.Goodsett (2017) used the QM Rubric to improve online workshops and advocated for other librarians to apply the QM Rubric to online information literacy learning objects such as research guides, tutorials, and webpages.Since then, some academic librarians applied the QM Rubric to library instruction modules (Farmer et al., 2021) and created reusable learning objects during the COVID-19 pandemic and beyond (Sheets et al., 2022).Librarians at Iowa State demonstrated an additional use of the QM Rubric by using it to develop a tool for peer evaluation of online teaching (Vega Garcia et al., 2017).
Many librarians have collaborated with instructors and instructional design teams to improve the quality of online or hybrid courses.After participating in QM training at Emporia State University, librarians joined instructional design teams, resulting in opportunities to incorporate library instruction within online courses containing research assignments (Mudd et al., 2015).
In collaboration with an instructional designer, Grand Valley State University Library's head of instruction services redesigned an institution-specific certification program for online or hybrid teaching faculty "to better meet the needs of instructional librarians and their unique single-session format of teaching" (Kenward & O'Kelly, 2017, p. 248).Instructional designers based the campus certification program, "Foundations of Online/Hybrid Course Development", on theories including the Community of Inquiry model, Universal Design for Learning, and the QM Rubric."Foundations" was required for anyone assigned to teach an online or hybrid course.During the month-long hybrid program, librarians used an adapted, condensed version of the QM Rubric to peer review learning modules created by their library faculty colleagues.
Instruction librarians have clearly demonstrated the usefulness of the QM Rubric for a variety of purposes in online information literacy instruction and collaborating with subject faculty and instructional designers.

Goals of the QM self-review
During the spring of 2019, two recently-hired librarians and the department chair in the Library Teaching & Learning Department (hereafter referred to as "the department") at Bowling Green State University (BGSU) attended a workshop (Applying the QM Rubric for Higher Education) with the goal of improving the design of LIB 2210 and reviewing the curriculum.
While some academic libraries focus on formal training or mentoring models to onboard their new hires, the department prioritized special projects that were placed on the back burner due to staffing or prioritizing other institutional strategic initiatives.This approach helped the new librarians feel more quickly integrated into the department, helped reduce the workload on senior librarians, and quickly established connections with their peers as they adjusted to the new working environment.
LIB 2210 is a seven-week information literacy course developed, in part, to address the university's growing need to offer one-credit hour courses.These courses help students stay enrolled full-time or complete their degrees in a timely manner.Previous LIB 2210 instructors were concerned about students' struggles in navigating the content in the learning management system (LMS) and successfully completing the assignments.The goal of the QM Review was to improve the navigation of the online course shell and increase student engagement and success.
One of the positive outcomes of attending the QM workshop was using an existing framework to guide the review of the LIB 2210 course.This resulted in revising the learning outcomes, adding opportunities for learner-to-learner interaction, and making general improvements that would ultimately benefit the learning experience of students.It also started discussions of exploring sustainable methods of teaching online information literacy courses targeted to high-impact programs on campus.

Self-review process
After attending the workshop, the authors used the QM Rubric to review LIB 2210.They reviewed the course chronologically: beginning at Standard 1 and ending at Standard 8.In a typical course review, the QM Reviewer is not a Subject Matter Expert (SME) and only reviews the design and functionality of the course, not its content.However, since the authors were acting as both SMEs and QM Reviewers, they also took the time to review the existing content for currency, accuracy, and clarity.Small fixes, such as updating URLs, were completed during the review process as much as possible.Large-scale recommendations, such as the creation of new course outcomes, were recommended and presented to the department for discussion and approval.

Self-review recommendations
An essential component of a QM-certified course is alignment.Alignment occurs when "critical course components work together to ensure that students achieve the desired learning outcomes" (Quality Matters, 2018, p. 5).These four critical course components are Learning Outcomes (Standard 2), Assessment (Standard 3), Instructional Materials (Standard 4), and Course Technology (Standard 6).When determining course quality, the learning outcomes (both course and module outcomes) are key.If learning outcomes are missing or not measurable, it is difficult for the review process to continue.
General Standard 2 (the foundation of alignment) reviews all learning outcomes in the course, as well as how those learning outcomes connect to learning activities.Standard 2.2: The module/unit-level learning outcomes or competencies describe outcomes that are measurable and consistent with the course-level outcomes or competencies.
Standard 2.3: Learning outcomes or competencies are stated clearly, are written from the learner's perspective, and are prominently located in the course.Standard 2.4: The relationship between learning outcomes or competencies and learning activities is clearly stated.(Quality Matters, 2018, pp. 14-18).
The LIB 2210 course as presented to the authors did not include module-level learning outcomes and some of the original course learning outcomes were not measurable.Since it was difficult to move on in the review without these components (and they were acting as SME), the authors drafted module-level outcomes for the seven modules before moving on to Standard 3 (Appendix B).
After the review was completed, the authors met to determine a plan of action for implementation.They presented an official proposal and implementation timeline to the department for approval.The significant recommendations were: 1. Update the course outcomes to align with the Association of College & Research Libraries (ACRL) Framework for Information Literacy for Higher Education (ACRL, 2015).(Appendix A) 2. Approve module-level learning outcomes created by the authors.3. Create individual module overviews for clarity and ease of use (Figure 1). 4. Replace long sections of text in LMS modules with narrated lecture slides/videos. 5. Increase the amount of learner-to-learner interaction in course.6. Create a general Q & A discussion board for students to ask questions to instructor (and each other).7. Create additional discussion-based assignments.8. Make class participation part of final grade.9. Convert some quizzes to discussions.10.Update rubric for Information Cycle assignment in Module 3. 11.Update the course textbook to align with the Framework for Information Literacy for Higher Education.
The department approved all reviewer recommendations, as well as the implementation timeline.

Success of the QM self-review process
Reviewing LIB 2210 with the QM Rubric resulted in a course template using instructional best practices that will serve as a blueprint for future online University Libraries (UL) courses.In particular, the revised LIB 2210 helped prevent the teaching burnout of librarians, many of whom were new to teaching asynchronous classes in addition to their other job duties.An updated course template with few structural issues allowed the instructor to focus on the success of their students, not fixing broken links or answering navigational LMS questions.The creation and implementation of a revised course template allowed more time for instructors to focus on providing positive learning experiences for students.It also helped the department jump-start conversations related to scaling credit-bearing courses and teaching multiple sections of LIB 2210.

Support for new LIB 2210 instructors
In Fall 2019 (the first semester after the QM Rubric self-review process), the department had the opportunity to teach seven sections of LIB 2210.Historically, the department taught one section of LIB 2210 fall semester and two sections spring semester.Although the opportunity to teach a high volume of students was exciting, teaching seven sections of the course was an extra demand on the workload of librarians in the department.The department chair called for volunteers throughout the UL to teach a section of the course during the Fall of 2019.Several of these volunteers had not previously taught a credit-bearing course online, let alone LIB 2210.To support these new instructors, the department chair and one of the authors created an Instructor Toolkit, including weekly instructor task lists (such as sending announcements and checking for technical issues), as well as assignment examples and answer keys for quizzes that needed manual grading.Since most of the new instructors were unfamiliar with the LMS and/or teaching an online credit-bearing course, it was strongly recommended for them to attend two fundamental workshops provided by the university's Center for Faculty Excellence: 1) Basics of Canvas and 2) An Introduction to Online Course Design and Teaching.The department chair acted as the point of support for the new instructors and was included in their course shells as a teacher.The Instructor Toolkit is still in use and updated regularly by one of the authors.

Course enhancement
"Continuous improvement" is one of the primary uses of the QM Rubric (Quality Matters, 2018, p. 5).After a course has been QM Certified, it is assumed that instructors will continually revise a course and it will eventually be recertified by QM.After the initial self-review process in 2019, the QM Rubric continued to influence the ongoing enhancement of LIB 2210.

Enhancing student engagement
In an asynchronous online course, learner-to-learner interaction increases active learning and student engagement with the course.However, it is important that the students recognize the interaction as meaningful and suitable to the course (as opposed to meaningless "busy work").The type of interaction missing from LIB 2210 before the QM self-review was learner-to-learner interaction (Standard 5.2).
For example, in the first week of the course, students introduced themselves on a discussion board, but they were not required to respond to any classmate's post.(Which may have been perceived as insignificant learner-learner interaction.)The instructor, however, is encouraged to reply to all student introductions to enhance teacher-learner interaction.(Students were not required to interact with each other at any time during LIB 2210.) Meanwhile, after the QM Rubric self-review took place, it became apparent that the assignment in Week 2 (Types of Information) was not successful.Students reviewed three types of articles and answered questions about them: a peer-reviewed journal, a magazine article, and a newspaper article.Although they answered most of the questions correctly, almost none of the students were able to identify the articles as scholarly or popular (the main learning objective of the assignment).
In the final revision of the assignment, students were broken into smaller groups to review a variety of source types produced by an individual scholar.They were required to post answers to questions about the sources on a discussion board then respond to one post in their small group.This assignment added learner-to-learner interaction relevant to the course material and increased student mastery of the module learning outcomes.

Questions about the sources
1. Identify the source you have chosen.2. Explain why the author is an authority/expert on the subject, connecting your justification to the week's readings and/or videos.3. Identify the primary audience for the source.If your answer is the "general public", you MUST include characteristics of the "general public" and why they would be interested in the source.4. Identify an appropriate situation or web platform/app to use this source as a citation or to share it.Refer to the Critical Thinking: Which Resources section in Chapter 5 of the textbook.Create your OWN scenario where you would use or share this source.5. Justify your selection of the scenario or platform you chose to use this source in, connecting your justification to the week's readings and/or videos.

Original week 2 learning outcomes
• Students will be able to categorize information sources as scholarly or popular (Course Outcomes 3, 4).• Students will be able to identify the uses and/or purposes of various information types (Course Outcomes 3, 4).• Students will be able to evaluate websites for authority, using Top Level Domains and site ownership (Course Outcome 3).

Revised week 2 learning outcomes
• Students will be able to identify the use(s) of various information types (Course Outcomes 1, 4, 6).• Students will be able to identify the purpose(s) of various information types (Course Outcome 3).
• Students will be able to investigate an author's authority and/or expertise on a given topic (Course Outcome 4).• Students will be able to justify their reasoning for selecting an information type for a research scenario (Courses Outcomes 1,6).

Selection of a new textbook
One of the recommendations during the QM Rubric self-review process was the selection of a new textbook.The text in use during the QM Rubric self-review was The Information Literacy User's Guide (Bernnard et al., 2014), which was based on the Seven Pillars Model of Information Literacy (Society of College, National and University Libraries, 2015).Since this was not the current information literacy model used by the department, it was recommended that the textbook be updated to one based on the ACRL's Framework for Information Literacy for Higher Education (ACRL, 2015).Although it was not possible to update the entire curriculum before the course resumed, it was the intention of the department to eventually find an updated text.
During the update of the Types of Information assignment (discussed in the Enhancing Student Engagement section above), one of the authors found the Introduction to College Research textbook by Butler et al. (2021).This text cites the ACRL Framework and includes chapters on algorithm bias, disinformation, and fact-checking.This book was presented as a replacement text for LIB 2210 and accepted by the department, with the caveat that any instructor could select the curriculum material for their individual course sections.
The selection of an updated open textbook met the following QM Standards: Standard 4.1: The instructional materials contribute to the achievement of the stated learning outcomes or competencies (Quality Matters, 2018, p. 24).
Standard 4.4: The instructional materials represent up-to-date theory and practice in the discipline (Quality Matters, 2018, p. 26).

BGSU Canvas course template
During the Spring of 2021, one of the authors was able to work with an Instructional Designer from Instructure, the company that developed and published the Canvas LMS.During this process, the LIB 2210 course template was updated to the BGSU Canvas Course Template.This was mostly an esthetic change, using accessible icons and other images that signal meaning across all BGSU courses.Theoretically, if most courses look the same in the LMS, students will be able to navigate Canvas more easily, and be able to focus on the course content instead of searching for what they need.
Additional updates were made to increase navigational clarity of the course and decrease the cognitive load of the students.For example, many of the text readings in Canvas were long pages that went "below the fold" (or scroll) on the computer screen.These pages were broken down into multipage readings that would be easier to digest for the student.These principles adhere to QM Standard 8: Accessibility and Usability Standard 8.1: Course navigation facilitates ease of use.

QM Rubric as instructional best practice
The QM Rubric brings several instructional design best practices into one assessment tool.This article highlights three instructional design best practices encompassed in the QM Rubric as they were the ones most relevant in the QM Rubric self-review and enhancement of LIB 2210: Backward Design, Universal Design for Learning, and Community of Inquiry.

Backward Design
The QM Rubric's focus on learning outcomes complements the Backward Design process, first introduced by Wiggins and McTighe in Understanding by Design.The chronological steps of Backward Design are very similar to QM Rubric's concept of alignment: 1. Identify the desired results (learning outcomes).2. Determine acceptable evidence (assessment).3. Plan learning experiences and instruction (instructional materials).(Wiggins & McTighe, 2005, p. 17) Like the concept of Backward Design, in the learning outcomes of a course are the foundation of the learning experience in the QM Rubric.Without measurable learning outcomes, learning cannot be assessed.When the learning outcomes and assessments are aligned with the instructional materials and learning activities in the course, there is an increased chance for a successful learning experience.

Universal Design for Learning
Like Universal Design for Learning (UDL), the implementation of QM Rubrics allows instructors to improve the learning experiences of all students by focusing on inclusion through varied assessments, accessibility, and usability.Accessibility and inclusion are core principles incorporated into effective instruction and are present in the QM Rubric.For example, Standard 1 promotes inclusiveness through the course welcome and introductions.Standard 3 emphasizes multiple opportunities to demonstrate competence with varied assessments (Representation in UDL) and Standard 8 enhances inclusion through accessibility and usability (CAST, 2018;Quality Matters, 2018).

Community of Inquiry
The QM Rubric General Standard 5 addresses the Learning Activities and Learner Interaction of the course.The three types of interaction in Standard 5 (learner-learner, learner-teacher, and learner-content) align with the three tenets of Community of Inquiry (CoI): social presence, cognitive presence, and teacher presence.The CoI framework "facilitate[s] the creation of communities of learners actively and collaboratively engaged in exploring, creating meaning, and confirming understanding" (Garrison, 2009, p. 352).Social presence in CoI corresponds with learner-learner interaction in the QM Rubric, cognitive presence with learner-content, and teacher-presence with learner-instructor.
Interaction between students and between the student and instructor increases engagement in the course, which increases student academic success.If students are not interacting with each other or the instructor in an online course, the CoI Framework and the QM Rubric agree that it is not the fault of the learners but the lack of instructional design of the course and planning of those interactions by the instructor that is to blame (Anderson & Garrison, 1995;Garrison et al., 1999).

Creating quality online learning environments
According to the United States Department of Education (2021), 59% of undergraduate students in 2021 were enrolled in an online course, even if they were not enrolled in a fully online program (2021).This is a 46% increase from 2012.Librarians participating in online learning is nothing new; online library tutorials have been mentioned in the literature since the 1990s (Chamberlain & Mitchell, 1996;Dewald, 1999;Donaldson, 2000) and the concept of embedding a librarian in the LMS starts to appear in the early 2000s (Cohen, 2002;Getty et al., 2000;Shank & Dewald, 2003).
However, the use of the QM Rubric by librarians, compared to other instructional design rubrics and tools, has been limited.
The QM Rubric should be used as guiding document for quality online education, regardless of the librarian's level of involvement in the learning experience.Even if librarians do not have the opportunity to teach a credit-bearing information literacy course, the QM Rubric can be applied to online one-shot library instruction and online learning objects.

Professional implications
As teaching librarians consider the effectiveness of one-shot library instruction (Bowles-Terry & Donovan, 2016;Pagowsky, 2021), the QM Rubric can be more heavily incorporated in the creation of information literacy credit-bearing courses, as well as other curriculum and LMS-based learning experiences designed in collaboration with instructors and/or instructional designers.
Some librarians were able to forge new partnerships or improve existing relationships with distance education departments and campus technology constituents (Pickens & Witte, 2015).Similarly, after receiving their QM Reviewer certification, Newby et al. (2014) developed partnerships with campus technology and instructional design constituents and new academic departments.A positive by-product of their training was finding a "niche" for themselves as information literacy "consultants" (p.36) for instructors who were redesigning their courses.Kenward and O'Kelly (2017) state that implementing programs that include librarians as QM consultants help capitalize on building relationships between libraries and cross-campus units who provide e-learning support to faculty (i.e., instructional designers, teaching and learning centers) by allowing librarians to extend their reach and expertise beyond the library.Having librarians as QM consultants creates an opportunity for librarians to demonstrate their value on campus and insert information literacy in courses and research assignments as SME (Kenward & O'Kelly, 2017).Attending QM Rubric trainings increases librarian expertise in online teaching, fosters relationships with instructional designers and instructor-development departments, and provides collaboration opportunities with instructors teaching online and hybrid courses.
Library and Information Science (LIS) graduate programs should also consider the inclusion of teaching pedagogy tailored to the online classroom (like the QM Rubric) in their curriculum to fully prepare future instruction librarians for providing quality distance and hybrid learning experiences in collaboration with instructors.

Conclusion
LIB 2210 was successfully reviewed and updated using the QM Rubric for Higher Education.The course learning outcomes were updated and module-level outcomes were created, connecting the course materials, learning activities, and assessments.Opportunities for learner-to-learner interaction were added, increasing student engagement with the course.After the self-review, the QM Rubric continued to be referenced as assignments were updated, a new open access textbook was selected, and the BGSU Canvas Template was applied to the course shell.The 7th edition of the QM Rubric for Higher Education is slated for release in 2023, including major revisions focused on inclusion and accessibility (Quality Matters, 2022).The authors intend to have the course officially reviewed and certified by QM once the 7 th edition of the QM Rubric is published.
The authors view the QM Rubric as a guiding document for creating quality online learning environments, recommend QM training for instruction librarians, and advocate for the use of the QM Rubric in LIS graduate programs to train new librarians.Additionally, QM training provides further opportunities for librarians to engage and connect with discipline instructors by offering curriculum support, feedback on the design and functionality of their online or hybrid courses, and integration of library resources into the LMS.

Standard 2 . 1 :
The course learning outcomes, or course/program outcomes, describe outcomes that are measurable.

Figure 1 .
Figure 1.Screenshot of module overview with learning outcomes, task list, and learning activities.