Massachusetts: One State's Approach to Setting Performance Levels on the Alternate Assessment
NCEO Synthesis Report 48
Published by the National Center on Educational Outcomes
Assessment Coordinator for Special Populations
Massachusetts Department of Education
Wiener, D. (2002). Massachusetts : One state's approach to setting performance levels on the alternate assessment(Synthesis Report 48). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/Synthesis48.html
In Massachusetts, about one percent of all students being assessed submit portfolios for the Massachusetts Comprehensive Assessment System (MCAS) Alternate Assessment. These portfolios are based on "expanded" state standards that describe academic outcomes appropriate for students with significant disabilities. Teachers collect "evidence" of their students’ performance on the standards during targeted instructional activities or structured student observations to create portfolios that contain an array of work samples, instructional data sheets, audio- and videotapes, or other evidence organized into "portfolio strands" in each content area.
MCAS Alternate Assessments are submitted to the state for scoring and designation of a performance level that gives parents and teachers information on how well these students are learning the general curriculum relative to their past performance and the performance of other students. The process used by the Massachusetts Department of Education to assign performance levels to alternate assessments is the focus of this report. This technical phase, called standard setting, reflects several steps that typically occur between scoring and reporting. However, the process reflects theoretical debates and decisions that occurred much earlier in the development process of the alternate assessment, sometimes years before the first portfolio was compiled and submitted. Several of these earlier conversations and their consequences are also described in this report since the recommendations form the philosophical basis of much that followed.
The alternate assessment in Massachusetts is one pathway to meet the state requirements for earning a "competency determination" needed to receive a regular high school diploma. Therefore, it was necessary to calibrate performance levels precisely between the alternate assessment and the general assessment, especially at the Needs Improvement level, which is the level required to earn the competency determination. Massachusetts decided to use an analytical rubric to convert raw scores to performance levels. Combinations of scores that could be obtained across the alternate assessment scoring rubrics for Level of Complexity, Demonstration of Knowledge and Skills, and Independence were discussed and reasoned perceptions were used to assign performance levels of Awareness, Emerging, Progressing, and Needs Improvement or above (Proficient, Advanced) in each portfolio strand. The reasoning behind the Massachusetts approach and the ways in which performance levels in each strand are combined to produce an overall performance level is described further in the report. This approach reflects not only the Massachusetts standards, but also its unique culture and values.
States are finding different ways to adapt their accountability systems to include all students, because the achievement of students with disabilities has typically lagged behind that of their non-disabled peers, and because recent state and federal laws require the participation of all students. Special educators are considering how, rather than whether, students with disabilities will participate in statewide assessments, while assessment policies themselves have become more flexible in accommodating the administration of those tests. Curriculum experts are placing increased emphasis on teaching students with disabilities the same content and skills being taught to their non-disabled peers, while regular and special educators are working together to adapt curriculum and instruction so diverse learners may participate more fully in academic activities.
A comparatively small number of students with the most complex and significant disabilities, though, have been more difficult to include in statewide assessments. Academic skills and subject matter have not always been a part of the curriculum for this population, and information has not systematically been collected on what these students have learned. The performance of these students is not easily determined using the same standardized paper-and-pencil tests used with the majority of students, but since participation in these assessments is now required, states have had to decide how best to include these students by giving them "alternate assessments." Alternate assessment methods and formats are determined by each state individually, though their common purpose is to improve instruction for these students and report their academic performance. By using alternate assessments with this population, schools can document what is being taught for purposes of system accountability, and demonstrate to parents and the public to what degree each of these students has learned state standards.
A majority of states have adopted individual academic portfolios as the most effective method of assessment for these "difficult-to-assess" students. Student portfolios accommodate a range of approaches to document learning, and afford teachers options for determining the ideal time, place, and method to assess their students. Portfolios provide teachers, students, and their parents with tangible evidence of student performance and feedback on their progress. While the contents of each is unique, their structure allows for evaluation and scoring using uniform criteria that can be shared with teachers beforehand.
The demands of creating and managing portfolios, and compiling this information for submission to the state, however, requires additional expertise on the part of teachers and time in which to complete this work. This fundamental change in classroom practice has required states to make a strong and continued commitment to provide professional development and technical assistance to educators who conduct alternate assessments, and to engage in an open dialogue about the efficiency, rigor, and usefulness of the process with those who are most affected by it.