By Dylan Wiliam
In a 1984 paper, Benjamin Bloom reckoned that one-on-one tutorial instruction had an impact of two standard deviations on student achievement, but this claim is meaningless unless the sensitivity to instruction of the measure of achievement is specified.
For example, standardized tests such as NAEP or TIMSS are relatively insensitive to instruction. An improvement of two standard deviations on TIMSS over a year would, in effect, represent an additional six years of learning per year, equivalent to multiplying the speed of learning by seven. On NAEP, two standard deviations would represent an additional eight years learning per year—a nine-fold increase in the speed of learning. This is simply not credible.
On the other hand, it is entirely plausible that one-on-one tutorial instruction by teachers would have a two-standard-deviation effect on material that is highly sensitive to instruction (and indeed, this explains why experimental interventions often show far bigger effects on teacher-produced tests than standardized tests—put bluntly, the teachers’ tests measured the kinds of things that teaching changes, while the standardized tests did not).
Until we understand better what makes an achievement measure more or less sensitive to instruction (Black & Wiliam, 2007), we have to be very careful with aggregating the results from studies that use different measures of student achievement.
Have a Question for Me?
I’m ready to answer it! Send me any questions you have about embedding formative assessment, college and career readiness, Common Core State Standards, or any other hot topic in education. In next month’s issue, I’ll offer my thoughts on the subjects that matter most to you.
References
Black, P., & Wiliam, D. (to appear, 2007). Large-scale assessment systems: design principles drawn from international comparisons. Measurement: Interdisciplinary Research and Perspectives, 4(3).
Bloom, B. S. (1984). The search for methods of instruction as effective as one-to-one tutoring. Educational Leadership, 41(8), 4-17.