There is growing evidence that student opinion surveys of teaching not only fail to measure teaching effectiveness, but are significantly biased against women, members of racialized groups, and academic staff teaching in a language other than their first language. In addition, such surveys may provide opportunities for students to anonymously target instructors with discriminatory, harassing, or otherwise inappropriate comments. CAUT recommends in its policy statement on the use of student opinion surveys that, “Such student-organized opinion surveys should not be used by post-secondary institution administrations as a means of assessing teaching performance and not be included in any academic staff member’s personnel file nor be used in any career procedures and decision making.”
The Ryerson Faculty Association (RFA), now the Toronto Metropolitan University Faculty Association, achieved the elimination of the use of student opinion surveys as a measure of teaching effectiveness in its important 2018 interest arbitration award. The arbitrator found the following:
According to the evidence, which was largely uncontested, and which came in the form of expert testimony and peer reviewed publications, numerous factors, especially personal characteristics – and this is just a partial list – such as race, gender, accent, age and “attractiveness” skew SET [student evaluation of teaching] results. It is almost impossible to adjust for bias and stereotypes… SET results have demonstrable limitations that raise real issues about their use as a measure of teaching effectiveness in tenure and promotion decisions. A further complication is the practice of reducing the FCS [Faculty Course Survey] results to averages and then comparing individuals with other individuals, the Department, Faculty and University. The evidence is clear, cogent and compelling that averages establish nothing relevant or useful about teaching effectiveness.
As a result, the arbitrator ordered that the collective agreement be amended to eliminate the use of survey results as a measure of teaching effectiveness in the tenure and promotion process, and to eliminate the use of numerical averages altogether. Following the Ryerson award, other associations have negotiated articles limiting the use of student opinion surveys. At Western, for example, the association negotiated performance evaluation criteria that characterize surveys as “information about student experience” that are considered along with contextualizing data including class size and response rates and comments from the member. Arithmetic averages are not included.
While associations are likely to meet significant resistance from administrations as they attempt to negotiate the exclusion of student opinion surveys from evaluation processes, there are intermediate measures that may be more easily achievable. For example, a joint working group at Ontario Tech University issued a set of recommendations including deleting a student’s entire survey and associated scores in cases where the student has made inappropriate comments.
The Mount Allison Faculty Association (MAFA) has negotiated language acknowledging the limitations of student surveys:
B.02 The parties recognize that any results from student surveys represent only one source of information regarding student opinion and degree of satisfaction with the teaching of an employee. Information from such surveys represents only the views of students responding to the survey and does not, in and of itself, constitute an evaluation of teaching performance.
Similarly, CUPE 3902 at the University of Toronto has negotiated the creation of a working group to address equity issues related to student evaluations and has language to ensure that student evaluations are not the sole determinant of performance:
18.04 Where they are available, student evaluations, whether conducted by the Department or by a student organization or by any other means, shall not be admissible as the sole determining factor to demonstrate unsatisfactory performance in either the discipline procedure or in arbitration. Departments may make use of student evaluations as an element in the Department's method for assessing work performance.
Several associations negotiated the suspension of student evaluations in performance reviews during the COVID-19 pandemic. There is an opportunity to continue the moratorium on their use, until they can be eliminated.
Resources:
Boring, A., Ottoboni, K., and Stark, P.B.. Student evaluations of teaching (mostly) do not measure teaching effectiveness. ScienceOpen Research. 2016: Student Evaluations of Teaching (Mostly) Do Not Measure Teaching Effectiveness – ScienceOpen
CAUT. Policy Statement on Use of Student Opinion Surveys. 2016: Use of Student Opinion Surveys | CAUT
OCUFA. Student Evaluations of Teaching: https://ocufa.on.ca/student-evaluations-of-teaching/
Peterson, D.A.M., Biederman, L.A., Andersen, D., Ditonto, T.M., Roe, K.. Mitigating gender bias in student evaluations of teaching. PLOS ONE. 2019: Mitigating gender bias in student evaluations of teaching - PubMed (nih.gov)
Ryerson University v Ryerson Faculty Association. In The Matter of an Interest Arbitration, 2018: 2018 CanLII 58446 (ON LA) | Ryerson University v Ryerson Faculty Association | CanLII
UOITFA. Student Course Evaluation Working Group Final Report. 2017: Student Course Evaluation Working Group Final Report – UOIT Faculty Association