From Harvard EdCast’s “The Celebrity Math Tutor” (transcript below)
Buffy Cushman-Patz: What efforts do you take to ensure that your pedagogy is consistent with what education research shows about how people learn, especially how people learn math and science?
Sal Khan: The reality is…when we’re going through the first pass of the videos there was very little effort; it really was just me doing my best shot and seeing what I would have liked to have and that my cousins and other people on YouTube seem to be benefiting from. Now we are getting pretty deep on our own analytics on our website. In terms of the broader research, I think there are people who come up with rules of thumb based on some study or another, and I’m not saying the study’s not valid, but I’m saying sometimes it’s not necessarily…you can’t come up with these rules the way all teaching has to be done like this. I think, for example, those research – you know there’s this one research study that’s been going around, kind of saying that… it first kind of hints at videos – maybe people can’t learn from videos and that if you do make a video you always have to address the misconceptions first and if you don’t address the misconceptions first, people are always going to conform whatever you say into their preexisting misconceptions. I don’t think that research is wrong; I think that is often the case. I don’t think it has to be religiously applied – that you have to, because in some areas people might not have even thought about something, they might not have misconceptions or maybe you explain once and you reemphasize that this goes against misconception A, B, C, or D. So I don’t think there’s one formula there. And I think frankly, the best way to do it is you put stuff out there and you see how people react to it; and we have exercises on our site too, so we see whether they’re able to see how they react to it anecdotally. You see, the comments they put, they’ll ask questions based on… Every time I put a YouTube video up, I look at the comments — at least the first 20, 30, 40 comments that go up — and I can normally see a theme: that look, a lot of people kind of got the wrong idea here. Or maybe some people did, and then I’ll usually make another video saying “Hey, look after the last video, I read some the comments and a lot of y’all are saying this is not what we’re talking about it’s completely different.” So that means I am attacking the misconceptions. But I think if you had a formula in place, and you do that every time, I think once again the learner will say, “This guy’s not thinking through it and he’s not teaching us his sensibilities, his thought processes. He’s just trying to meet some formula on what apparently is good video practice. “And I’ll go the other way: you can dot all the “i”s and cross all the “t”s on some research-based idea about how a video should be made, but if your voice is condescending, if you’re not thinking things through, if it’s a scripted lecture, I can guarantee you it’s not going to appeal with students. And I think the other mistake people… I’d like some research to be done with this, and it really goes against the grain against what most people assume is what even video is about is, that all the feedback that we’ve gotten is not seeing the face is, maybe, one of the most compelling things about it is hearing the voice, because the face is hugely distracting. And so long answer to a short question. I think it’s nice to look at some of the research, but I don’t think we would… and I think in general, people would be doing a disservice if they trump what one research study does and there’s a million variables there: who was the instructor, what were they teaching, what was the form factor, how did they use to produce it? You’d be doing yourself a disservice if you just take the apparent conclusions from a research study and try to blanket them onto what is really more of an art. It’s like saying that there’s a research study on what makes a nice painting and always making your painting according to that research study that would obviously be a mistake.
It’s unfortunate that “The Teacher to the World” was only able to mention one study about how students learn. A study which he then dismisses. And since he doesn’t describe any other efforts to be consistent with pedagogy, his real answer to Buffy’s question is: “I don’t.”
Let’s look at Khan’s response in more detail:
“Now we are getting pretty deep on our own analytics on our website.”
I don’t see how statistics about how many times students have watched/rewound each video or how many times students miss a question in the exercises tells us anything about how effective his videos are. I don’t see how he could use that data to refine his future videos in the same way a teacher would reflect and refine lessons from year-to-year.
“…you can’t come up with these rules the way, all teaching has to be done like this.”
He’s right. There is no one rule, no one formula, for teaching. The Physics Education Research User Guide website contains 51 different research-based teaching methods. The website can filter these methods by type, instructional setting, course level, coverage, topic, instructor effort, etc. And while 51 different methods may seem overwhelming, they all have one important characteristic in common: interactive engagement (IE).
So what is interactive engagement? Hake defines IE as methods “designed at least in part to promote conceptual understanding through interactive engagement of students in heads-on (always) and hands-on (usually) activities which yield immediate feedback through discussion with peers and/or instructors.”
A video lecture is not interactive engagement.
“…maybe you explain once and you reemphasize that this goes against misconception A, B, C, or D.”
Khan (along with most of the general public, in my opinion) has this naive notion that teaching is really just explaining. And that the way to be a better teacher is to improve your explanations. Not so! Teaching is really about creating experiences that allow students to construct meaning.
“And I think frankly, the best way to do it is you put stuff out there and you see how people react to it…”
This is flawed. People’s reactions are not indicators of effectiveness. Pre/post testing is needed to indicate effectiveness. Ah, but perhaps there is a relationship between people’s reaction and effectiveness? The research indicates otherwise. In the very research study that Khan says is valid (and then dismisses), student actually did better after watching the videos they described as confusing, and made no gains after watching the videos they described as easy to understand. Additional research indicates that when an instructor switches over to IE methods, course evaluations from students tend to be more negative than the previous year, despite gains from students going up. (Don’t worry, a few years after the switch to IE, the evaluations go back to pre-IE levels.)
“You see, the comments they put, they’ll ask questions based on… Every time I put a YouTube video up, I look at the comments — at least the first 20, 30, 40 comments that go up — and I can normally see a theme: that look, a lot of people kind of got the wrong idea here. Or maybe some people did, and then I’ll usually make another video saying “Hey, look after the last video, I read some the comments and a lot of y’all are saying this is not what we’re talking about it’s completely different.” So that means I am attacking the misconceptions.”
Again, it’s not about crafting better explanations. It’s about helping students wrestle with their conceptions and guiding them.
“But I think if you had a formula in place, and you do that every time, I think once again the learner will say, “This guy’s not thinking through it and he’s not teaching us his sensibilities, his thought processes. He’s just trying to meet some formula on what apparently is good video practice.”
Another naive notion of teaching. The goal is not for the teacher to teach the students his sensibilities and thought processes. The goal is for the teacher to have the students use their sensibilities and thought processes to reason through the concepts. Empower the student to think for themselves, rather than consuming the teacher’s ideas.
“And I’ll go the other way: you can dot all the “i”s and cross all the “t”s on some research-based idea about how a video should be made, but if your voice is condescending, if you’re not thinking things through, if it’s a scripted lecture, I can guarantee you it’s not going to appeal with students.”
Yet there are plenty of people who prefer to watch Walter Lewin’s highly-scripted performance lectures to Khan’s off-the-cuff style lectures. (Though remember that preference has nothing to do with effectiveness. In fact, Lewin’s showstopping lectures were no more effective than the mundane professors before him.)
“…and I think in general, people would be doing a disservice if they trump what one research study does and there’s a million variables there: who was the instructor, what were they teaching, what was the form factor, how did they use to produce it? You’d be doing yourself a disservice if you just take the apparent conclusions from a research study and try to blanket them onto what is really more of an art. It’s like saying that there’s a research study on what makes a nice painting and always making your painting according to that research study that would obviously be a mistake.”
Here is the most damning piece of evidence, from Hake’s famous six thousand student study:
The six thousand students in Hake’s study were not in a single class. They were in 62 different courses, from high school to university, taught by a variety of instructors with different personalities and expertise. And yet ALL the IE courses made greater gains (the slope of the graph — between 0.34 and 0.69) than the traditionally taught courses (average 0.23). It should also be noted that the green IE courses above were NOT identical and did not follow some magic teaching formula. They only had to conform to the Hake’s broad definition of IE given above. So you see, those “million variables” that Khan mentions don’t matter. METHOD trumps all those other variables.
But surely teacher expertise matters, right?
Yes and no.
NO: As seen in Hake’s study above, when comparing IE teachers to traditional teachers, expertise doesn’t matter because IE always trumps traditional.
NO: Note the small spread of the red-colored traditional classes shown above, which hover around an average gain of 0.23. Traditional methods produce very similar results no matter the level of the course or instructor.
YES: When comparing IE teachers to other IE teachers, expertise does matter. IE gains ranged from 0.34 and 0.69. As instructors get more comfortable using IE methods, gain increases. See, for example, this graph about the effectiveness of modeling instruction:
Expert modelers had higher gains than novice modelers.
But surely there is a place for lectures, right?
Yes, BUT students must be “primed” for the lecture. According to the PER User’s Guide FAQ:
It is possible for students to learn from a lecture if they are prepared to engage with it. For example, Schwartz et al. found that if students work to solve a problem on their own before hearing a lecture with the correct explanation, they learn more from the lecture. (For a short summary of this article aimed at physics instructors, see these posts – part 1 and part 2 – on the sciencegeekgirl blog.) Schwartz and Bransford argue that lectures can be effective “when students enter a learning situation with a wealth of background knowledge and a clear sense of the problems for which they seek solutions.”
For more information about how people learn, I highly recommend two great FREE online books from the National Academies Press:
- How People Learn: Brain, Mind, Experience, and School
- How Students Learn: History, Mathematics, and Science in the Classroom .
If you are a physics teacher, be sure to get these discipline specific books about how students learn physics:
- Teaching Introductory Physics (Arons)
- 5 Easy Lessons (Knight — who also wrote calc-based and algebra-based physics texts)
- Teaching Physics with the Physics Suite (Redish, FREE online)
And just in case you think I’m an armchair critic with nothing to contribute, I want you to know I’ve opened up my classroom to the whole world on my Noschese 180 blog, where I’ve been sharing a picture and a reflection from each school day. It’s not quite the Noschese Academy, but I hope you find it worth reading and commenting, as we journey through teaching together.
Action-Reaction 