Using Anchoring Vignettes to Improve Survey Measures of Teachers' Mathematics Instructional Practice
Description
Download Presentation SlidesHigh-quality measures of instructional practice are essential for research and evaluation of innovative instructional policies and programs, as well as for providing feedback to teachers and administrators. However, existing measures have generally proven inadequate because of cost and validity issues. To address this problem, the presenters developed vignette-based teacher survey self-report measures of ambitious mathematics instructional practices aligned with state standards for college and career readiness. Teachers read several contextualized "vignettes" describing hypothetical teachers engaged in a particular instructional practice, with vignettes varying in terms of the extent of that practice. Then, teachers rate the extent to which the hypothetical teachers in the vignettes were engaged in that practice. Lastly, teachers rate the extent to which they engaged in that practice themselves in a recently-taught lesson. Teachers' ratings of the vignettes are used to "calibrate" - or adjust - teachers' self-ratings to be more accurate. Responses from 65 teachers in grades 4-9 who responded to the survey suggested that vignettes were more reliable and valid for some instructional practices compared to others. For the instructional practices for which there was higher reliability and validity, the presenters found significant and high correlations between teachers' adjusted survey self-rating and previous observation ratings of teachers' instructional practice, including ratings provided by several widely-used observation rubrics. These results suggested that vignettes may provide an efficient, cost-effective method for gathering data on teachers' instruction. Cognitive interviews with participating teachers provided insight into possible improvements to the items.
