Writer: John Lorinc

When University of Lethbridge (UL) hiring or tenure review committees onboard new members these days, they are given a somewhat unusual training document: a copy of a 2018 ruling by Ontario arbitrator William Kaplan in a case involving Ryerson University (now Toronto Metropolitan University) and its academic staff association.

At issue was the use of student opinion surveys of teaching in decisions about hiring and promotion. These so-called evaluations -- which are little more than customer satisfaction surveys, according to University of Lethbridge Faculty Association (ULFA) President Dan O'Donnell -- have been part of post-secondary life for generations and became an increasingly important part of the student experience from the 1950s onwards. O'Donnell recalls as an undergraduate in the late 1980s relying on the "Un-calendar," a compendium of student opinion surveys published at the University of Toronto, to choose courses and professors.

Because student surveys generate numerical scores, they can become an unofficial data point in decisions taken by hiring and promotion committees. However, since the 1980s, a growing body of academic literature began to demonstrate how these surveys often contained biases, particularly against women instructors and those whose first language was not English. (Canlii Connects has published a useful blog about the Kaplan ruling, which includes links to numerous studies that have identified flaws and biases in student surveys.)

Students, O'Donnell says, "are an untrained group of evaluators." He has read surveys in which a student comments on the way a woman faculty member looks, dresses or behaves. Besides these biases, O'Donnell adds, the surveys effectively ask students to pass judgments about the broader pedagogical context of the courses they take.

While collective agreements have included language cautioning against reliance on student surveys for many years, Kaplan's ruling was the first to really pull together all the evidence. After it was handed down, the decision received widespread attention within Canadian academic staff associations, including ULFA, whose executive worked to ensure that the members of hiring, promotion and tenure review committees are aware of the issues and specifically the ways that biases creep into student evaluations that then inform decisions about promotion.

Today, UL hiring committee members must not rely on surveys that are not designed to capture whether an instructor is effective or not. "It's undeniable that the student experience is the student experience," O'Donnell observes. But that experience is invariably subjective and disconnected from the formal criteria that committee members use to make their decisions. "Getting the Kaplan decision used as a training manual really drove that home. It really changed the culture."