Warm, friendly, humorous, and entertaining instructors are generally appreciated by students, but these same instructors will be quickly criticized if the course content is not well-organized and students are not learning. Students are discriminating judges of quality instruction. Instructors cannot "skate by" on entertainment alone—there must be accompanying content that is understandable and organized before students will judge you as an effective teacher. If you happen to be one of those rare teachers who combines great pedagogical skills with a fantastic personality and presentation abilities, you will indeed receive high ICES scores. For the rest of us "mortals," we can rest assured that the entertainment factor is not enough to ensure high ratings.

There has been quite a bit of controversy about student grades and student ratings of instruction. Past research has shown positive correlations, zero correlations, and even one or two negative correlations between students' expected grades and their ratings of instruction. On our campus, and many other college venues, the correlation between expected grade and assigned student rating is approximately 0.3 or below. Clearly the relationship between grades and ratings is complex, but one can assume that "easy graders" are not going to automatically receive high ICES scores.

It is absolutely true that we should attempt to measure the amount that our students have learned. This outcome measure is a great indicator of teaching effectiveness. But to think that student ratings are not connected to learning is incorrect. Meta-analysis of controlled studies in which student learning and instructor ratings were investigated show that the more students learned in a course, the higher the ratings of the instructor. The causal reasoning goes something like this: effective teaching leads to student learning which leads to positive instructor ratings. The relationship between teaching and learning is much too complex to enable this simple conclusion, but research has made links between student learning and student ratings of instructors.

Sorry, but your students' perceptions don't change much after they've left your course. Surveys of graduating seniors and alumni have shown that their ratings of professors remain rather consistent over time. While this goes against some anecdotal evidence, the research has consistently shown that alumni ratings are strongly correlated with current student ratings.

Peers are an excellent source for feedback about our teaching. In some cases, such as appropriateness and currency of course content, they are probably the best source of valid and reliable feedback. Yet studies have shown that overall peer ratings, though slightly higher, will not differ that much from student ratings. Some researchers and administrators question the reliability of peer classroom ratings when used for decision-making. Given this concern, we still find peer observations and document review to be excellent methods of formative feedback for teaching improvement. These same methodologies can also be a great supplement to student ratings for tenure and promotion decisions.

Wrong. There is substantial research that shows consistency of ratings both within a course and over time. As long as there are enough students completing forms, the inter-rater reliability of the results are rather high (reliability, not necessarily validity). Additionally, instructor's ratings are relatively stable over time for the same courses. When students are asked what they are looking for in instructors (what they are evaluating) they generally describe traits and behaviors that faculty and teaching & learning experts would find laudable: organization, preparedness, concern for students, stimulation of thinking, fair grading, availability for help, etc.

There are correlations between workload and student ratings—moderate to heavy workloads generally are correlated with HIGH student ratings of instruction! The correlations are not incredibly strong, but they are rather consistent. Demanding instructors, who also provide support for student learning, are not going to be penalized just because they require hard work from their students.

Student ratings of instruction can be utilized in many ways. Typically the results are used as one part of the tenure and promotion process. They are also used for campus and departmental teaching awards. Our office hopes that the results are used for teaching improvement too. When accompanied by an educational consultant, ICES scores (and the open-ended comments) can be very worthwhile for changing how we structure our courses and how we teach and assess our students. While there may be isolated cases where low ICES results "hurt" faculty, there are certainly many instances where high scores are also used to reward faculty. Plus, low or high ICES ratings can be used to determine what is working well and what might need to be altered in your teaching (especially when combined with open-ended student comments).

Excellent point! This is one variable that does affect student ratings. Generally, elective courses receive higher ratings than required courses. Students tend to be more motivated to take elective courses and student motivation does play a role in subsequent student evaluation of the course and the instructor. Research on this campus validates this finding. Because of this relationship, we control for this variable on our ICES reports. On the instructor summary report, we compare your ICES scores on the global items to other faculty on campus according to the elective/required nature of the course. We have three broad categories: required, mixed, and elective. These normative comparison groups are used on the instructor summary report, longitudinal profiles, and our List of Teachers Ranked as Excellent.

Global items can provide much of the useful information for summative decision.

Cohen (1981), in a meta-analysis of 41 multi-section validity studies found average correlation between student achievement and the global instructor item to be 0.47 and between student achievement and the global course item to be 0.43.

Cashin & Downey (1992), in a study of 17,183 undergraduate and graduate courses across 104 U.S. and 1 Canadian higher education institutions (2-year colleges, 4-year liberal arts colleges and research universities), found that the three global items (instructor quality, course quality, perceived learning) were highly correlated with student self-reported progress on learning objectives relevant to the course. The global instructor item had a correlation of 0.74, the global course item had a correlation of 0.77, and the global student learning item had a correlation of 0.83 with progress on learning objectives. 

Abrami, d’Appollonia & Rosenfield (2007) investigated the number of dimensions common among 225 items from 17 different student ratings forms used in multi-section validity studies. They also conducted a principal component analysis based on 6,788 inter-item correlation coefficients reported in the multi-section validity studies. They found that the first principal component explains 62.8% of the total variance and all items load heavily on this factor. This suggests an overall instructional skill factor common to student ratings on items across 17 different forms. Factor analysis revealed four factors with the first three being highly correlated.

Considering the findings from these studies, it is appropriate to use global items for summative purposes. In each course, multiple students complete ICES forms and aggregating the student ratings for a course and considering an instructor’s ratings on global items from at least 5-7 sections have high reliability.

Instructors can select items from a large list of ICES catalog items relevant to their course to get student feedback to improve teaching and courses. This makes it impractical to base summative decisions on students’ ratings on multiple items with different sets of items across ICES forms for instructors. If a department is interested in using more than the global items for summative decisions, a core set of items can be included on the ICES forms for all instructors teaching a specific subject code.

Generally, to increase the reliability of summative decisions about an instructor’s teaching effectiveness, we recommend multiple sources of evidence as mentioned above and in the Provost’s Communication # 9.

References:

Abrami, P. C., d’Apollonia, S., & Rosenfield, S. (2007). The dimensionality of student ratings of instruction: What we know and what we do not. In The scholarship of teaching and learning in higher education: An evidence-based perspective (pp. 385-456). Springer, Dordrecht.

Cashin, W. E., & Downey, R. G. (1992). Using global student rating items for summative evaluation. Journal of educational Psychology, 84(4), 563.

Cohen, P. A. (1981). Student ratings of instruction and student achievement: A meta-analysis of multisection validity studies. Review of educational Research, 51(3), 281-309.