Indiana’s CREA State Team Examines Standard-Setting Process for Licensure Exams
In 2021, Indiana joined the Consortium for Research Based and Equitable Assessments (CREA), an initiative by the American Association of Colleges for Teacher Education to examine state-level certification assessment scores and their impact on promoting a diverse educator workforce. Our state team consisted of faculty from Indiana University’s School of Education, representatives from the Indiana Department of Education, and school district administrators from Indiana’s public schools. Together, we looked at our state-level data on entrance and content area licensure exams and reached the same conclusion many have reached for decades in Indiana and across the United States: significant pass rate gaps between white and Black teacher candidates.
Our participation in CREA however, asked us to look beyond this conclusion. As Leslie Fenwick expressed in the first CREA report, The History, Current Use, and Impact of Entrance Licensure Examinations Cut Scores on the Teacher of Color Pipeline, the adverse effects created by licensure exams for teachers of color must be considered a problem of entrenched institutional racism. To understand why, for example, white teacher candidates consistently outperformed Black and Hispanic teacher candidates by gaps as large as 52 percentage points in Indiana, we turned our attention to the construction of licensure exams. We wondered who built licensure exams, what the process of building licensure exams looked like, and who was involved in setting cut scores.
These questions led to the production of a report by Indiana University’s Center for Evaluation and Education Policy, Mis-Shaping The Teaching Force: An Analysis of Indiana’s Teacher Candidates. In this report, attention is given to the racist legacy of licensure exams and the problems associated with the psychometric method used to construct, validate, and set cut scores for licensure exams. More specifically, the report analyzed the demographic composition of the “subject matter experts” that were used to construct licensure exams. This analysis revealed the following:
- According to available data from the Indiana Department of Education, panelists that served as subject matter experts were 92% White, 2% Black, and 2% Hispanic.
- Despite the ability of Educational Testing Services (ETS) to draw from panelists across the United States, 4 of the 10 licensure exam construction panels did not include a single Black subject matter expert, and half did not include a Hispanic expert. In one panel, all subject matter experts were white.
Demographics matter because research demonstrates how racially diverse panels making high-stakes decisions improve the quality of the deliberation. For example, Sommers (2007) in his study of the deliberations of racially homogenous and racially heterogenous juries, not only found richer informational exchanges among racially heterogenous jurors, but also noted that diverse juries deliberated longer and considered a wider range of information (e.g., case facts, “missing evidence” and fewer objections when racism was mentioned).
Because the panels for both state-wide and national licensure exams are generally composed of educators, and educators in the United States and Indiana are majority white, the standard setting process creates a looped system that sustains a majority white teacher workforce in Indiana.
Our report offers several recommendations to break this self-reinforcing loop such as a commitment by test developers and state education agencies to diversify standard setting panels, creating systems that support candidates of color, more research into the standard setting process, and establishing flexibility rules for licensure exams. If national and state education policymakers truly wish to “diversify the workforce,” other solutions in addition to working harder to recruit and retain teachers of color are necessary.
Alexander Cuenca, Ph.D., is associate professor of curriculum and instruction, program coordinator of Middle/Secondary Social Studies Education, and faculty affiliate of the Center for Evaluation and Education Policy at Indiana University, Bloomington. He served as the Indiana state lead for the Consortium for Research Based and Equitable Assessments.
Tags: assessment, diversity, equity, inclusion, workforce development
