|
What Is Scientifically-Based Research on Progress Monitoring?
Overview of research
Studies included in this overview met the following criteria. First, they relied on experimental design; that is, teachers volunteered to participate in any of the study conditions and then were randomly assigned to conditions. Second, all studies included a control group (where teachers did not use systematic progress monitoring), against which the effects of progress monitoring procedures were assessed. Third, progress monitoring procedures were implemented for at least 15 school weeks, or 4 school months. Fourth, teachers' instructional plans were analyzed to determine how planning changed as a function of progress monitoring. Fifth, students' academic achievement was measured at the beginning and end of the study on global tests to determine whether students achieved differently in the various progress monitoring conditions.
This overview is organized in three sections: (a) evidence on CBM's utility to identify students in need of additional or different forms of instruction, (b) evidence on the usefulness of CBM's graphed analysis of the overall score to help teachers improve their instructional programs and effect better student achievement, and (c) evidence on the added value of CBM's skills profiles for designing superior instructional programs that produce greater learning.
Results of these studies are described in terms of statistical significance and effect sizes. Statistical significance means that one treatment group performed so much better than another group that it is highly unlikely that the results could be attributed to chance. This speaks to the reliability of the findings: If a similar study were conducted again, we would expect to find similar results, and if a teacher were to implement the treatment, we would expect similar effects for her/his students. It is possible, however, to have a statistically significant effect, which is accurate and reliable, but is small.
To address the question about whether a treatment effect is big or small, we look at effect sizes. Effect sizes tell us how many standard deviations one treatment group performed better than another. If the mean of a test is 100 and its standard deviation is 15 (like an IQ test), then an effect size of 1 standard deviation would mean, for example, that the treatment group ended the study with a score of 100 while the control group ended with a score of 85. Generally, in educational research, an effect size of .30 is considered small, .50 is considered moderate, and .70 is considered large.
Identifying students in need of additional or different forms of instruction. Research shows that CBM can be used to prompt teacher concern about student progress and to signal the need for additional or different forms of instruction. For example, in a recent study (4), 24 second-grade teachers were randomly assigned to control or CBM progress monitoring groups. Progress monitoring teachers, with the assistance of computers, collected CBM oral reading fluency data with every student in their classes. The computer organized the CBM information into individual student graphs as well as class reports. These reports showed CBM class graphs; noted students who fell in the lowest quartile of the class; and identified students in need of comprehension instruction, fluency development, or decoding work. In addition, the report provided a rank ordering of the students in the class, sorting them into those who already had met the year-end CBM benchmark, those who were on track to meet the year-end benchmark, and those who were at risk of failing to achieve the year-end benchmark. Teachers collected CBM data for 15 weeks, with individual graphs shown at the end of every data-collection session and with class reports printed every 3 weeks. Every 3 weeks, teachers answered the questions, "Do you have children whose progress seems problematic? Which children are you concerned about?" Progress monitoring teachers expressed concern about statistically significantly more students, with effect sizes exceeding 1 standard deviation. Moreover, when asked, "Why are you concerned about ________ ?," Progress monitoring teachers described features of student performance to explain their concern; by contrast, control teachers cited reasons beyond their control (such as English Language Learner status, special education status, attention or motivation problems, or inadequate parental involvement). This pattern of results was statistically significant. Therefore, systematic progress monitoring can be used to raise teacher concern about students' reading progress and to signal the need for additional or different forms of instruction.
Usefulness of graphed analysis of thee overall CBM scores. Evidence strongly supports the utility of graphed analysis of overall CBM scores in helping teachers plan more effective programs. Studies (5) conducted over the past decade provide corroborating evidence of strong effects on students' reading, spelling, and mathematics achievement when teachers rely on CBM progress monitoring to help them plan their instruction. A study conducted in the New York City Public Schools (6) illustrates this research. Teachers participated for 18 weeks in a control group (i.e., no systematic progress monitoring) or a CBM progress monitoring group. In the progress monitoring group, teachers measured students' reading performance with CBM oral reading fluency twice weekly, scored and graphed CBM performances, and applied CBM decision rules (described in the next three paragraphs) to those graphs to plan their students' reading programs. Children whose teachers employed CBM progress monitoring to develop reading programs achieved statistically significantly better than students in the control group on measures tapping a variety of reading skills, including a fluency test as well as the decoding and comprehension subtests of the Stanford Diagnostic Reading Test. Effect sizes were large, ranging between .94 and 1.18 standard deviations. So, teachers used CBM's graphed analysis to effect greater reading achievement in terms of fluency, decoding, and comprehension.
CBM progress monitoring, using the graphed analysis, relies on decision rules that help teachers set ambitious student goals and help them determine when instructional adjustments are needed to prompt better student growth. The student's initial CBM scores are graphed. The teacher uses normative information about expected rates of CBM growth to set a goal for the end of the school year. A diagonal line is drawn from the initial scores to the goal level/date. This diagonal line represents the desired rate of improvement for that student. As the instructional program is implemented, weekly CBM data are collected and graphed. A line of best fit is drawn through the student's graphed scores to estimate the child's actual weekly rate of improvement, or CBM slope. The steepness of the goal line is compared to the steepness of the student's actual rate of improvement. If the steepness of the student's actual rate of improvement is greater, then the CBM decision is to raise the goal. If the steepness of the goal line is greater, then the CBM decision is to adjust the instructional program to stimulate greater learning.
Fuchs, Fuchs, and Hamlett (7) explored the contribution of the goal-raising CBM decision rule. Teachers were assigned randomly to and participated in one of three treatments for 15 weeks in mathematics: no CBM, CBM without a goal-raising rule, and CBM with a goal-raising rule. The goal-raising rule required teachers to increase goals whenever the student's actual rate of growth (represented by the slope through the actual, graphed scores) was greater than the growth rate anticipated by the teacher (reflected in the goal line). Teachers in the CBM goal-raising condition raised goals statistically significantly more frequently (for 15 of 30 students) than teachers in the nongoal-raising conditions (for 1 of 30 students). Moreover, concurrent with teachers' goal raising was statistically significantly differential student achievement on pre/post standardized achievement tests: The effect size comparing the pre/post change of the two CBM conditions (i.e., with and without the goal-raising rule) was .50 standard deviation. Consequently, using CBM to monitor the appropriateness of instructional goals and to adjust goals upward whenever possible is one means by which CBM can be used to assist teachers in their instructional planning.
A second way in which CBM can be used to enhance instructional decisions is to assess the adequacy of student progress and determine whether, and if so when, instructional adjustments are necessary. When actual growth rate is less than expected growth rate, the teacher modifies the instructional program to promote stronger learning. Fuchs, Fuchs, and Hamlett (8) estimated the contribution of this CBM decision-making strategy with 29 teachers who implemented CBM for 15 school weeks with 53 students. Teachers in a "CBM-measurement only" group measured students' reading growth as required but did not use the assessment information to structure students' reading programs. Teachers in the CBM-"change the program" group measured student performance and used CBM to determine when to introduce program adjustments to enhance student learning. Results indicated that, although teachers in both groups monitored student progress, important differences were associated with the use of the "change the program" decision rule. As shown on the Stanford Achievement Test-Reading Comprehension subtest, students in the "change the program" group achieved statistically significantly better than a no-CBM control group (effect size=.72), whereas the "measurement only" CBM group did not (effect size=.36). Moreover, the slopes of the two CBM treatment groups were significantly different, favoring the achievement of the "change the program" group (effect size=.86). As suggested by these findings and results of other studies (9), collecting CBM data, in and of itself, exerts only a small effect on student learning. To enhance student outcomes in substantial ways, teachers need to use the CBM data to build effective programs for difficult-to-teach students.
Added value of skills profiles. To obtain rich descriptions of student performance, alternative ways of summarizing and describing student performance are necessary. Because CBM assesses performance on the year's curriculum at each testing, rich descriptions of strengths and weaknesses in the curriculum can be generated, and studies show how these skills profiles enhance teacher planning and student learning. In a series of investigations in reading (10), math (11), and spelling (12), teachers were assigned randomly to one of three conditions: no CBM, CBM with goal-raising and change-the-program decision rules, and CBM with goal-raising and change-the-program decision rules plus CBM skills profiles. In all three studies, teachers in the skills profile group generated instructional plans that were statistically significantly more varied and more responsive to individuals' learning needs. Moreover, they effected statistically significantly better student learning as measured on change between pre- and posttest performance on global measures of achievement. Effect sizes associated with the CBM diagnostic profile groups ranged from .65 to 1.23 standard deviations. This series of studies demonstrates how structured, well-organized CBM information about students' strengths and difficulties in the curriculum can help teachers build better programs and effect greater learning.
Summary
As demonstrated via the randomized field trials described above, teachers can use systematic progress monitoring in reading, mathematics, and spelling to identify students in need of additional or different forms of instruction, to design stronger instructional programs, and to effect better achievement outcomes for their students.
 Previous | Next 
|
Information About PDF