Oral Assessments – Use Sparingly

Opinion

​If you intend to use oral assessments (vivas), be sure to take steps to limit bias

Some time ago there was a push to remove names from exam scripts, leaving just student numbers. This was because anonymity was essential to reducing bias – including subconscious bias – when grading important works.

Bias is a real issue. People have known this for a long time.

There has been bias against female authors, and accordingly, Mary Shelley published Frankenstein anonymously, the Bronte sisters used male pseudonyms, and similarly George Eliot and George Sand are well known pseudonyms for women Mary Evans and Amantine Dupin.

Beyond this top orchestras insist that musicians audition behind a curtain and even lay down thick carpets so that footsteps don’t suggest the gender of the aspiring players.

In a globalised world where we value opportunity and inclusion, the measures used to minimize bias are now extensive. We should all work to eliminate biases due to gender, ethnicity, religion, social class, accent, age, disability, sexual orientation etc.

When we hold job interviews several steps are taken to limit bias: a broad committee is enlisted; the same questions are asked to each applicant; and the group attempts to measure candidates against a pre-agreed rubric.

When it comes to oral assessments (or vivas), we usually do have the rubrics but we need to make sure that other measures that reduce bias are also put in place.

AI has shown we need secure assessments. Two options are repeatedly bubbling up: moving back to supervised exams or introducing new oral exams. Orals have a role, but their shortcomings mean they must be done properly and even then used with caution.

Here are some of the problems with oral assessments:

1. It is hard to ask every student the same questions without revealing those questions to those yet to be examined. So, students may get questions of different difficulty which is obviously unfair.
2. Some students get an easier question first up, and others a harder question. This can flavour the flow of the rest of the oral assessment.
3. Larger classes, by necessity, tend to adopt shorter orals. This makes it harder to ensure that the full syllabus is covered. If the ‘sampling’ is too brief, a student may be unlucky and not have the opportunity to show what they have learned in some areas in which they excel. Or they may be lucky and their success may not reflect their actual mastery of the subject.
4. Having diverse assessment panels judging the oral can be both expensive and logistically complex.
5. Many students are nervous and find formal written exams stressful, but students also find high-stakes orals highly intimidating.
6. The marking of orals can be subjective. I have mentioned bias but even if there is minimal bias, say all the students are of one homogenous background, there will still be subjective responses to the ‘charisma’ of individual students. Generally, our aim should be to assess ‘learning’ and there is a risk that orals, in part, measure ‘charm’.
7. The time of day and the sequence of applicants can also have an effect. In my experience the first student seldom gets 10/10 as that result is reserved for students that everyone agrees stands out against a baseline. In good systems all the marks are reviewed at the end, but often it is difficult to remember everything.
8. It is probably necessary to record the orals to allow for appeals, remarking, and to resolve disputes. This adds additional costs and raises privacy and security issues.
9. Finally, some argue that oral examinations should not be used as full assessments but as supplements, added to verify authorship, for example checking that a student actually understands what they wrote in their thesis. This is reasonable but it is difficult to see how one can truly distinguish between a student that was capable of producing a play worthy of Shakespeare, and one who simply understands and can talk about Shakespeare. The two skills are not the same.

So why do I hear so much talk about introducing oral assessments?

Some see this as an opportunity to move forward and try something new. Many don’t want to go backwards to invigilated exams. Most importantly perhaps, if a course is online with remote learners, it is difficult to secure a written exam, so an oral may seem the best currently available answer.

Others see the stress in exams and underestimate the stress that oral vivas entail.

But mostly, I believe the appeal of oral exams is that they seem or may actually be more authentic. Life is a conversation, and in the age of AI when all knowledge is available online, the ability to communicate in real time – as in an oral exam – will remain a core human activity.

Moreover, introducing oral exams signals that we care about communication, and encourages students and staff to devote more time to enhancing communication skills.

Put simply assessment serves two purposes, it can assure learning and it can motivate learning. Introducing oral exams may motivate the acquisition of communication skills.

In summary, oral examinations have a place and they can help drive the acquisition of communication skills, but they should be used carefully and I think sparingly for summative assessment. They are potentially too unfair and too expensive to serve as major secure assessments (invigilated exams are fairer, more comprehensive, and cheaper). But they may be useful for lower stakes assessments and in motivating the acquisition of communication skills and rhetoric.

After all rhetoric is one of the most ancient subjects and Cicero would be pleased to see it championed.

Professor Merlin Crossley is Deputy Vice-Chancellor (Academic Quality) at UNSW