Pilot System Explores a New Dynamic for Exam Delivery at AU
Exam Item Banking Pilot
(Part of the Exam Harmonization Project)
A pilot project is exploring an innovative web-based system that would enable AU’s faculty to build exams from a bank of exam questions. The exams should be easy to create and process, and they will even be unique to each student if so desired.
The Exam Item Banking Pilot is one of the steps AU is taking as part of the Exam Harmonization project to develop an efficient online system for exam delivery. A team of AU faculty and staff is working with consultants to develop and test two components that work hand in hand: a question bank and an exam compilation system.
“As a predominantly online institution, AU is uniquely positioned to take a huge leadership role in online exam delivery,” says Akil Pessoa, CEO of MuchLearning, the Edmonton company developing the pilot system for AU.
As is the case for universities across Canada, AU currently provides most of its exams to students on paper. “This paper-based system has inefficiencies and needs upgrading,” says Mark Meunier, one of the managers of the pilot and AU’s manager of Exam Services within the Office of the Registrar.
Also, exams at AU are generally static collections of questions. Again, this is not so different from most institutions, but with advancements in technology, exams have the potential to be far more dynamic.
How the Pilot System Works
In the MuchLearning platform, exam questions are housed individually in a question bank. “We have a number of question types,” says Pessoa. “A question can be anything from an essay question to multiple choice to complex mathematics.”
All exam questions for all courses are kept in a single bank, and each question is tagged with metatags that describe what the question is about and how to use the question. A crucial part of metatagging is identifying which course learning objective(s) a question is meant to address.
“Rather than authoring an exam,” says Pessoa, “you author questions and make sure you have great questions well-tagged.”
To create an exam, faculty members search the question bank using parameters such as topic, learning objective, and difficulty level. The MuchLearning system compiles the exam based on the parameters specified. Faculty members can then review the exam and add or delete questions as they see fit.
It’s a different approach to exam development, but it has the potential to yield benefits not possible in the more traditional system of paper-based exams.
For example, the system can safeguard against cheating by randomizing questions. “We can have different combinations of questions for students writing the same final exam in a room together,” says Meunier.
“On some questions, especially math questions, we can actually randomize down to elements within the questions,” adds Pessoa.
“So your variable names will change, your coefficients will change, but it’s essentially the same question template.”
MuchLearning’s facility for math is one the reasons the Exam Harmonization working committee chose to hire the company. Of the several vendors reviewed by the committee, MuchLearning was the only one that could dynamically represent math problems, says Meunier.
“They provide an intuitive way of typing and doing math on a computer,” adds Maria Torres, an academic coordinator of mathematics at AU and one of the committee members. “We want to extend this to chemistry and physics and other disciplines.”
MuchLearning’s system is equally adept at handling exam questions for arts-related disciplines. It also has the potential to reduce the duplication of work, as it will allow faculty members to use and adapt questions that have been created by others.
“Say there’s a math question dealing with financial math,” says Pessoa. “A business professor might be putting an exam together, and as he searches the question bank, that math question might pop up. He might say, ‘Hey, this question fits exactly what I want to do, so I’m going to add a business tag to it so it’s readily available on the business side as well.’”
“Rather than an exam question being something you write once, and there is no way to access it easily, it now starts to become something living, which is really what it was in the first place, the embodiment of an idea that someone had,” Pessoa says.
What Happens Next
Throughout March, AU staff will continue populating the question bank with existing exam questions from a test group of AU courses across all faculties. When the two pilot components are ready (sometime this spring), the Exam Harmonization working committee will invite AU’s academic community to participate in system testing.
“MuchLearning is quite flexible and willing to learn with us, to really take our opinions into consideration,” says Torres. “There is a good possibility we will grow together.”
The Exam Harmonization project is one of more than 30 Open Knowledge Environment projects that have launched at AU in the past year-and-a-half thanks to funding through the Knowledge Infrastructure Program (KIP).