Printer Friendly Version (pdf, 1.9MB)

Massachusetts: One State's Approach to Setting Performance Levels on the Alternate Assessment

Combining Scores to Yield a Performance Level

Dr. Roeber suggested that MCAS-Alt project leadership meet with regular assessment psychometricians and data analysts from the Department and from Measured Progress to review and select the most effective formula for calculating a total content area score, and to identify "cut scores" for specific performance levels based on a range of calculated score totals. During ensuing discussions, however, questions were raised about the necessity of generating a single total numerical score for each strand and content area in the alternate assessment, and whether it might cause confusion to introduce another, entirely different score scale beside the 200-280 score scale already in use for MCAS test results. Some felt this would reinforce the separateness of the alternate assessment and wondered instead whether a system could be developed that used reasoned judgment, instead of a calculation, to describe overall student performance based on different raw score combinations. After a spirited discussion, this reasoning prevailed, and the idea of calculating a total numerical portfolio score was abandoned in favor of a different approach.

Whether a mathematical equation or a reasoned approach is used to determine a student’s performance, however, some kind of scale, analytical rubric, or other consistent method must be used to convert raw scores to performance levels (Roeber, 2002). The analytical rubric developed for this purpose in Massachusetts is actually a series of grids based on a student’s score as shown in Figure 3.

Sixty-four different possible score combinations were discussed and analyzed by the group, and a performance level identified by consensus for each. Decisions were based on reasoned perceptions of what each score combination revealed about the student’s performance, and the relative position of that performance level within the hierarchy of other levels. It was easier to analyze and assign performance levels beginning with the lowest and highest levels, then working toward the middle. In the end, the group was able to define and categorize all score combinations. The model was tested using various arbitrary score combinations to check that the defined performance level made sense, given the student’s scores, and that scores were appropriately scaled relative to adjacent scores.

An analysis of several arbitrary score combinations reveals, for example, that a student who scores LC=3, DSC=2, and Ind=3 according to the MCAS-Alt scoring rubric, is a student who is working on modified (or "expanded") learning standards, who demonstrates 26-50% accuracy, and who needs assistance 51-75% of the time during standards-based activities (Massachusetts Department of Education, 2001). From this information, the student would appear to be performing above the definition of Awareness in this content area, but not yet at Progressing, in which the student would perform the skills and demonstrate the knowledge with greater independence and accuracy. Since this student is somewhere between the Awareness and Progressing performance levels, we can say with relative confidence that the student is at the Emerging level. Another student who hypothetically scored LC=3, DSC=3, Ind=4 is also working on modified standards, but performs with a sufficiently high rate of accuracy and independence to be placed in the Progressing performance level. He or she is probably ready to attempt even more challenging tasks, skills, and concepts in the coming year, since the data suggest he or she has mastered skills and content in the current portfolio. Figure 3 shows the complete analytical rubric for determining performance levels in each portfolio strand.

Figure 3. Analytical Rubric for Determining Performance Levels in Each Portfolio Strand    [D]

Calculating the Overall Performance Level

Once performance levels are determined for each of three required portfolio strands in the content area, based on the analytical rubric shown in Figure 3, these are averaged and rounded to the nearest whole number to determine the overall performance level in that subject. To calculate the average of three performance levels, consecutive numerical values are given to each performance level, as follows: Awareness = 1, Emerging = 2, Progressing = 3, Needs Improvement = 4, etc. Figure 4 shows how different combinations are averaged to yield a final performance level.

Figure 4. Performance Levels in Each Strand are Averaged to Determine an Overall Performance Level

Meeting the State’s Graduation Requirement Through MCAS Alternate Assessment

A performance level of Needs Improvement or higher is required on grade 10 MCAS assessments in English Language Arts and Mathematics in order to earn a "competency determination" (the state’s requirement to receive a regular high school diploma). As previously stated, alternate assessment is one pathway to meet that requirement. Therefore, it is necessary to calibrate performance levels precisely between the alternate assessment and the general assessment, especially at the Needs Improvement level. What does a Needs Improvement portfolio look like, and what specifically constitutes a "comparable performance" to a student who was tested and earned this score? Although portfolio scorers can accurately determine a portfolio’s completeness, accuracy, and independence of performance, an additional level of review seemed necessary in order to assure the breadth, quality, and comparability of the student’s performance to that of other students who passed the grade 10 MCAS tests in those subjects.

To accomplish this, the Department convenes a panel of math and English language arts content specialists each year to review a selection of grade 10 portfolios set aside for this purpose, and to make recommendations to the Department on whether these students have demonstrated achievement at or above Needs Improvement level based on the evidence in their portfolios. Panelists, themselves, were selected by the Department for their secondary-level teaching expertise in the content area; their experience serving on the state’s Assessment Development Committees that develop and review general assessment test items with the state’s test contractor; and their extensive familiarity with Massachusetts Curriculum Frameworks. Panelists are familiar with work typical of students who "passed" the grade 10 MCAS tests in ELA and Mathematics since they teach these students on a daily basis. Panel members were asked to examine pre-scored portfolios at Level of Complexity 4 and 5, and to verify whether they felt the contents:

• document the full range of learning standards, covering knowledge and skills tested on grade 10 MCAS tests in the content area;
• demonstrate a level of performance typical of students who perform at the Needs Improvement level on the MCAS test in that subject; and
• exemplify an even higher performance level than Needs Improvement; for example, Proficient or Advanced.

Previous  |  Next