There is growing evidence that student opinion surveys of teaching not only fail to measure teaching effectiveness, but are significantly biased against women, members of racialized groups, and academic staff teaching in a language other than their first language.41 In addition, such surveys may provide opportunities for students to anonymously target instructors with discriminatory, harassing, or otherwise inappropriate comments. CAUT recommends that, “student opinion surveys should not be used in any career procedures and decision making involving academic staff.”42
The Ryerson Faculty Association (RFA) achieved the elimination of the use of student opinion surveys as a measure of teaching effectiveness in its important 2018 interest arbitration award.43 The arbitrator found the following:
According to the evidence, which was largely uncontested, and which came in the form of expert testimony and peer reviewed publications, numerous factors, especially personal characteristics – and this is just a partial list – such as race, gender, accent, age and “attractiveness” skew SET [student evaluation of teaching] results. It is almost impossible to adjust for bias and stereotypes… SET results have demonstrable limitations that raise real issues about their use as a measure of teaching effectiveness in tenure and promotion decisions. A further complication is the practice of reducing the FCS [Faculty Course Survey] results to averages and then comparing individuals with other individuals, the Department, Faculty and University. The evidence is clear, cogent and compelling that averages establish nothing relevant or useful about teaching effectiveness.
As a result, the arbitrator ordered that the collective agreement be amended to eliminate the use of survey results as a measure of teaching effectiveness in the tenure and promotion process, and to eliminate the use of numerical averages altogether. Following the Ryerson award, other associations have negotiated articles limiting the use of student opinion surveys. At Western, for example, the association negotiated performance evaluation criteria that characterize surveys as “information about student experience” that are considered along with contextualizing data including class size and response rates and comments from the member. Arithmetic averages are not included.44
While associations are likely to meet significant resistance from employers as they attempt to negotiate the exclusion of student opinion surveys from evaluation processes, there are intermediate measures that may be more easily achievable. For example, a joint working group at Ontario Tech University (formerly the University of Ontario Institute of Technology) issued a set of recommendations including deleting a student’s entire survey and associated scores in cases where the student has made inappropriate comments.45
The Mount Allison Faculty Association (MAFA) has negotiated language acknowledging the limitations of student surveys:
B.02 The parties recognize that any results from student surveys represent only one source of information regarding student opinion and degree of satisfaction with the teaching of an employee. Information from such surveys represents only the views of students responding to the survey and does not, in and of itself, constitute an evaluation of teaching performance.46
Similarly, CUPE 3902 at the University of Toronto has negotiated the creation of a working group to address equity issues related to student evaluations and has language to ensure that student evaluations are not the sole determinant of performance:
18.04 Where they are available, student evaluations, whether conducted by the Department or by a student organization or by any other means, shall not be admissible as the sole determining factor to demonstrate unsatisfactory performance in either the discipline procedure or in arbitration. Departments may make use of student evaluations as an element in the Department's method for assessing work performance.47
41 See for example Anne Boring, Kellie Ottoboni, and Philip B. Stark. “Student evaluations of teaching (mostly) do not measure teaching effectiveness.” ScienceOpen Research, January 7, 2016.
42 CAUT Policy Statement on Use of Student Opinion Surveys, approved by the CAUT Council November 2016.
43 Ryerson University v Ryerson Faculty Association, 2018 CanLII 58446 (ON LA), <http://canlii.ca/t/hsqkz>.
44 Collective Agreement between Western University and the University of Western Ontario Faculty Association. (July 1, 2018 – June 30, 2022).
45 Student Course Evaluation Working Group Final Report. http://www.uoitfa.ca/student-course-evaluation-working-group-final-report/. Retrieved February 28, 2018.
46 Collective Agreement between Mount Allison University and the Mount Allison University Faculty Association – RAS (July 1, 2016 – June 30, 2019).
47 Collective Agreement between the Governing Council of the University of Toronto and the Canadian Union of Public Employees Local 3902 – Unit 3 (September 1, 2017 – August 31, 2021).