12 Ways to Measure Teaching Effectiveness

Taken from Berk, R. A. (2005). Survey of 12 strategies to measure teaching effectiveness. International Journal of Teaching and Learning in Higher Education, 17(1), 48-62.  See “Teaching Excellence” article in this issue for additional information.

(Reminder: This series of articles about learning outcomes began in the August 2008 issue of the newsletter. Archives of our newsletters are at http://www.cte.tcu.edu/Koehler_Center_Newsletter_Archive.htm.)

Formative and Summative Assessment

The “Articles and Books” segment of last June’s newsletter talked about author/researcher Rick Stiggins’ take on assessment: formative assessment is assessment for learning (provided to help students advance toward learning outcomes while they’re in the process of learning), and summative assessment is assessment of learning (evaluation of learning that did—or did not—occur in the past).  Written comments on a student paper as a means to help the student write better and avoid similar mistakes in the future is an example of formative assessment; the final exam is a kind of summative assessment.

Both formative and summative evaluations have roles in working with student learning outcomes.  The learning outcomes for the course which you state on your syllabus are summative expectations.  “By the end of this course, students should be able to . . .,” means that the learning to have taken place by the last day of class will have included each of the outcomes listed.  You must employ some means of determining at the end of the course the degree to which the learning outcomes in the course were achieved by each student.

However, during the course you also provide feedback to students to help guide them toward achieving those end-of-term outcomes.  Though this feedback might not take the form of a graded item (e.g., you may not grade class discussions, but your answers to student comments during discussion help students better understand the material), it is still valuable formative assessment.

But do you know how valuable your formative feedback is in advancing students toward accomplishing the learning outcomes for the course?  Can your formative feedback be improved in that regard?  Is there additional formative feedback you can provide students?  What about formative feedback from students to you—can their comments to you while you’re in the teaching process be valuable in helping you better help them achieve the outcomes of the course?

Two suggestions from Stiggins’ writing can be particularly powerful for improving the effectiveness of your formative feedback.  One is to provide frequent descriptive feedback (as opposed to judgmental feedback).  Such feedback lets students know how to adjust their concepts or processes as they are learning, and this kind of feedback also sends a strong signal that making mistakes and learning from them is a natural part of college education.  Another powerful suggestion from Stiggins is to provide clear and understandable rubrics so students can self-assess their progress.  A rubric defining how an assignment will be graded is perhaps the most valuable piece of advice a student can receive during the learning, which is the entire point of the assignment.

Are there activities in your course for which students receive feedback that is not graded, that is meant only to help them learn the material better?  Office visits by students are a prime example of this kind of feedback, but students today rarely take advantage of this opportunity.  Other ways you can accomplish this kind of formative assessment, though, include your comments during threaded discussions, “practice” tests or assignments, and using clickers in class so you can respond instantly to student misunderstandings or misperceptions (more on clickers).

Regarding formative feedback from students to you, two great techniques are the muddiest point exercise (see Angelo & Cross, below) and a mid-semester student survey.  Briefly, the muddiest point technique is a way to find out via 3x5 card or other short writing method what the muddiest point is concerning material covered before the time you collect this writing.  Faculty often collect this either at the end of class in order to clear up at the start of the next class any points lots of students listed as still muddy, or they collect it during the break of a long class meeting and clear up muddy points right after the break.  The mid-semester survey is a short, informal survey you give students to collect their feedback about how the class is going for them and what things have/haven’t been helpful for their learning (sample mid-semester survey questions).  Their answers can help you adjust in time to make a difference that semester.

Your summative evaluation of how well each student did on achieving the course outcomes in aggregate is sent to the Registrar’s Office as a final grade.  This is not a learning outcome; it is your judgment of how well a student achieved the outcomes for the course.  However, your statement of how you determine the final grade (e.g., weighting and number of assignments, etc.) means you must consider which assignments related to each outcome in determining whether the student met that outcome.  Final grades will not serve student learning in terms of providing feedback to help learn the material (that time is passed), but if they are clearly tied to the level of work each student did on each assignment related to an outcome, they are extremely useful to you as you consider ways to improve the teaching and learning that will occur the next time you teach the course.   This process of using data about student achievement of learning outcomes is extremely important in accreditors’ evaluation of an institution’s effectiveness in helping students learn.

Angelo, T. A., & Cross, K. P. (1993). Classroom assessment techniques: A handbook for college teachers (2nd ed.). San Francisco: Jossey-Bass. (Available in Koehler Center Library.)

How Do You Know You’re a Good College Teacher?

Dr. Ron Berk, professor of Biostatistics and Measurement at Johns Hopkins University, surveys the landscape regarding tools and processes which can play a role in determining one’s teaching effectiveness.  In an article in the International Journal of Teaching and Learning in Higher Education (citation below), he comments on ways twelve measurement effectiveness strategies can complement each other in drawing an accurate picture of an instructor’s teaching effectiveness.

Berk, a statistician, is aware of the limitations of any single kind of measurement for judging a construct as complicated as teaching effectiveness.  He therefore argues for a unified conceptualization of teaching effectiveness which requires a triangulation of measures from various sources of data about college teaching: “By drawing on three or more different sources of evidence, the strengths of each source can compensate for the weaknesses of the other sources, thereby converging on a decision about teaching effectiveness that is more accurate than one based on any single source” (p. 49).

The purpose of Berk’s article is “to critically examine the value of these 12 sources reported in the literature on faculty evaluation and to deduce a “bottom line” recommendation for each source based on the current state of research and practice” (p. 49).

The table in the snapshot of this issue of the newsletter identifies the twelve sources Berk’s article considers.  What are his recommendations?

First, whether student ratings of teaching effectiveness is a trustworthy measure, the overwhelming evidence is yes.  Berk’s comment that there is “more research on student ratings than any other topic in higher education” (p. 50) reflects his own research in finding a “preponderance of evidence” statement about student ratings.  (He does provide a listing of citations, though, if you want to cherry-pick research about student ratings that speak to various of the limitations of such ratings).  Bottom line, student ratings drawn from well-constructed instruments provide an “excellent source of evidence for both formative and summative decisions, with the qualification that other sources also be used for the latter” (p. 50).

Next, Berk makes an important point about the process of knowing that information about one’s teaching is dependable: just as a key part of the process of confirming one’s research is peer review, so a key part of the process of confirming one’s teaching should be peer review—“teaching should be judged by the same high standards applied to other forms of scholarship: peer review” (p. 50).  Berk cautions, though, that peer review should only be a part of formative decisions and not part of summative/personnel decisions.

Then the article moves through the remaining ten teaching effectiveness measurement tools/processes.  At the end of each description (which includes information from the literature about the positives and negatives for the item being discussed), Berk supplies a piece of advice that can be helpful in deciding whether the item should be in the mix of one’s measurement of teaching effectiveness.

A special comment here about learning outcomes as a source of teaching effectiveness measurement: there are inherent issues with it that preclude its use in making summative/personnel judgments (though it is excellent to provide instructors feedback about teaching their classes so that they can make adjustments to help improve student learning).  Berk analogizes well on this point, highlighting the differences between quality control of raw material being possible for a factory worker but not for a teacher.

In summary, Berk’s advice to any institution, dean, chair, etc., trying to make decisions about how to structure a measurement approach to faculty teaching effectiveness is that the approach must include student ratings collected using a good instrument and that it include at least one other source.  In trying to decide on the mix of sources from which data can be gathered, determining the use to which you will put the judgments made about a teacher’s effectiveness should be a paramount consideration; don’t use data from a source which should not be used for a summative decision to make that kind of decision.  However, triangulating from among multiple sources strengthens the odds that one’s judgment of a teacher will be accurate.

From the individual faculty member’s perspective, Berk’s advice can be extremely useful.  His discussion of the reasons why each of the twelve kinds of teaching effectiveness measures provides insights into how to get the most complete and trustworthy big-picture answer to the question, “How do I know if I’m a good college teacher?”

Berk, R. (2005). Survey of 12 strategies to measure teaching effectiveness. International Journal of Teaching and Learning in Higher Education, 17(1), 48-62.  Available at http://www.ronberk.com/survey_12_IJTLHE8.pdf.

Most Women Will Graduate; Most Men Will Not (Leonard Sax’ Boys Adrift)

Tamar Lewin’s July 2006 New York Times article is one of many pieces reporting on the differential between male and female enrollment in college, achievement in college, and graduation from college.  She says:

Department of Education statistics show that men, whatever their race or socioeconomic group, are less likely than women to get bachelor’s degrees—and among those who do, fewer complete their degrees in four or five years.  Men also get worse grades than women. (¶7)

Dr. Leonard Sax summarizes several points made in the Lewin article and then goes on to examine much research related to the issue of male vs female educational achievement in his recent book, Boys Adrift: The Five Factors Driving the Growing Epidemic of Unmotivated Boys and Underachieving Young Men (citation below).  While TCU’s student body is a subset and not necessarily representative of the nationwide averages upon which the following statement from Sax’ book is made, the statement is nonetheless startling: “Most girls who enroll in a four-year college will eventually earn a degree.  Most boys won’t” (p. 8).

What is going on?  Have you noticed differences in general between male and female effort in your classes?  Is there actually something of substance here, and therefore something that college faculty should know about and potentially do to address this situation?

There is no dearth of discussion about this.  See the responses to this Inside Higher Ed article to get a sense of the polarizing nature of the debate when approached from the standpoint of whether the college curriculum itself is or is not contributing to the drop in male representation and achievement.  Whether a shift away from literary texts in K-12 which include stories about high-achieving male protagonists in stereotypically gendered environments (battlefield, hunting, space warfare fiction, etc.) has led to white males’ perceptions that they have become the “designated a–holes” among the higher ed student body is a topic of contentious debate.  (The indelicate description, by the way, is how one respondent to the Inside Higher Ed article described what four college men said when characterizing their perception of what they encountered as undergrads.)

You may be very surprised, though, to learn of Sax’ observations about what may be contributing to a growing number of American men afflicted with the “failure to launch” syndrome (a typical F-t-L man might be 32, living at home, unmarried, underachieving, and not particularly bothered by any of this).  Not necessarily listed in order of the degree to which each potential cause may be contributing, Sax says the following five factors play a role in “male failure to launch syndrome”:

1. Changes at School. This is multi-faceted.  One component is that some boys thrive in a competitive environment, but many schools have worked to eliminate competition. “If your son is motivated by competition, then eliminating it from his school, throwing out his toy guns, and forbidding him to write stories with violent themes won’t change him.  Those policies may disengage him from school, however.  The end result may be a boy who feels that the only place he is truly understood as he really is, is the world of video games” (p. 52).
2. Video games. Again, a complicated, multi-faceted potential contributing factor.  However, Sax is careful to separate playing video games as a cause from suffering a consequent lack of what playing games replaces, which may be more important.  He briefly covers the unequivocal connection between violent video games and violent behavior and leaves the reader to sort out why violent themes in literature—Hemingway, Steinbeck, Dostoyevsky, etc.—don’t have the same effect.
3. Medications for ADHD. More boys than girls are placed on Adderall, Ritalin, Concerta, Metadate, Dexedrine, Focalin, or Baytrana than are girls.  Researchers are beginning to study whether these drugs’ potentially damaging effects on the brain’s nucleus accumbens translates into decreased motivation: “If a boy’s nucleus accumbens is damaged, he may still feel hungry, or sexually aroused.  He just won’t feel motivated to do anything about it” (p. 90).
4. Environmental estrogens and endocrine disruptors. Chemicals like bisphenol A (plastic bottles leach this into the liquid they contain), many pesticides used on crops, steroids given to farm animals to make them bigger before slaughter, and other non-natural interventions are often bad for us.  Some of these are strongly implicated in the significantly lower sperm counts of today’s boys (compared to granddad) and significantly higher numbers of genital abnormalities and brittle bones—and some researchers postulate the concomitant lower testosterone levels may be connected to a lower drive to achieve.  Of greatest impact, though, may be these chemicals’ effect on memory and motivation and on damaging the nucleus accumbens, all of which occur to much greater degrees in boys than in girls who are exposed due to neurostructural differences between the genders.
5. Revenge of the forsaken gods.  American culture has changed in ways that send different messages to boys today than it did forty years ago.  Sax lays out numerous examples which characterize the shift; one example is the American dad’s portrayal on TV: he is no longer of the ilk in My Three Sons or Leave It to Beaver—he is the bumbling fool of The Simpsons, or he is inept and clueless (though funny) like any number of sitcom fathers today.

Sax’ thesis is evocative, and he refers to many pieces of quality research to support it (he is both M.D. and Ph.D. and has conducted a body of scholarly research himself).  You may find the book an engaging read as part of your consideration of your students’ motivational approaches toward studying.

Lewin, T. (2006, July 9). At colleges, women are leaving men in the dust. New York Times. Available at http://www.nytimes.com/2006/07/09/education/09college.html?pagewanted=1&sq=Tamar%20Lewin
%20At%20College,%20Women%20Are%20Leaving%20Men%20in%20the%20Dust&st=cse&scp=1.

Redden, E. (2009, May 22). Lost men on campus. Inside Higher Ed. Available at http://www.insidehighered.com/news/2009/05/22/men.

Sax, L. (2007). Boys adrift: The five factors driving the growing epidemic of unmotivated boys and underachieving young men. New York: Basic Books. TCU Library: LC1390 .S29 2007.

Online Assessments

When assessing your online student’s learning, consider the following:   

• Utilize all of the communication tools from the online course along with other activities you have established for your course.
• Use practice quizzes and self-assessment exercises.
• Ask students to recommend work they would like to have considered in your assessment of their learning.
• Where possible, use tasks that are authentic-- i.e. allowing students to explore, discover, discuss, and meaningfully use new learning in contexts that involve real-world problems and projects that are relevant and interesting to the learner.
• You can create exams that are "single-access exams" and exams that can be accessed many times by a single student, so that the exam becomes a self-learning tool. In designing your exams, you can also utilize them as important communication tools by putting in explanations for correct and incorrect answers in the available fields.

Engage Student Attention With the Addition of YouTube Videos in Presentations

Students appreciate the use of multimedia materials in faculty presentations and one of the simplest media to incorporate are videos. YouTube offers a variety of content that can be utilized to illustrate any number of points within a presentation. Follow these simple steps to add YouTube videos to your presentations.
Want to add a YouTube video to your classroom?  It's easy—but there is definitely a best practice associated with this process. We recommend you place the video inside a 1X1 table (1 cell = 1 column & 1 row).

1. Create a 1X1 table, using the table wizard.  Make sure your table has a border color.  Feel free to put text all around this box or nothing at all.  SAVE your changes!
2. Go to the YouTube video and find the "embed" code.  (Hint: It's in a gray box to the right of the video.)
3. Copy that embed code.
4. Come back to your course page, click into the HTML and find the table tag.  It should look like this: <TD></TD>
5. Paste the embed code in between the greater than and less than symbols like this: >embed code goes here<
6. SAVE CHANGES!

Help Desk Tip

Many Web sites, including the METNET site, use Cascading Style Sheets (CSS) to control the size and color of text on the screen. In order for you to manually resize the text in your browser, you will have to either disable style sheets or use the browser's Text Zoom feature, or both.

Internet Explorer (version 7.0 and greater)
To change text sizes in Internet Explorer (version 7.0 and greater) for Windows:

• From the Menu Bar. select View
• Scroll to Text Size and choose your preference

Safari

• From the Safari menu, select Preferences.
• Click Appearance.
• To change the standard font or fixed-width font used in pages, click the appropriate Select button.
• Change the font size using the slider bar.