That student evaluations of teaching (SET’s) are unfair is widely assumed. New research suggests not.
There was a blue at UNSW a couple of years back, over management making individual course SET’s public. The National Tertiary Education Union was opposed because this could identify teachers.
The Fair Work Commission ultimately found for the university, to which the NTEU’s Damien Cahill responded, “student surveys can be useful for pedagogical development, however their well-documented limitations must be recognised – including gender and racial biases,” (Campus Morning Mail 9 March 2022).
But an analysis of 375,000 SETS at an “Australian university” by Richard O’Donovan (Monash U) found common worries about SET aren’t so.
Dr O’Donovan analysed the SET stats and related demographic data on key student types, finding:
- Students complete a SET when they have strong views on a course, which creates a risk of negative reviews: the data showed when there are strong views they are “overwhelmingly positive”
- “Revenge reviews” from angry students: don’t dominate. 20 per cent of students who fail or don’t like the lecturer complete the SET.
- Large classes compare poorly to small ones: “lower student ratings for large units probably relate to genuine concerns about their experience of teaching in such units rather than being attributable to unit size per se.”
- International students, who tend to positive are reticent, thus reducing good results: Rather, international students are “somewhat overrepresented,” “which – if anything – slightly raises average SET scores beyond where they should be”
- Racist and sexist students: the study used known gender and citizenship status of students, the gender of instructors and an “imperfect estimate” of English speaking or not backgrounds. “This is not an exact proxy for race, but it plausibly captures some aspects of what might influence students with racist attitudes.” Analyses show the highest ratings are for staff from non-English speaking backgrounds. Male students rate women academics higher than men.
- Surveys run in exam periods unfairly penalise teachers of units that have examinations: “overall time effects on average SET responses are negligible, and average sentiment about teaching remains very consistent regardless of when the responses occurred, and whether a unit had examinations or not.”
So where do staff concerns come from: Dr O’Donovan suggests studies “focus on minute differences, arguably missing the forest for the trees” without effect-size analysis, “using the full scale of available responses in order to avoid over-emphasising small or even miniscule effect”
The take-out: “SET data are not compromised by factors outside the control of educators.”
“While these results do not mitigate the negative effects of ‘incivility’ experienced by Australian academics … it is at least comforting to know that such attitudes appear to represent a tiny minority of students, and that overall, students’ feedback on teaching is not riddled with bias and bigotry”
