|
Progress Monitoring in an Inclusive Standards-based Assessment and Accountability System
Limited Use of Data for Effective Provision of Instructional Strategies, Interventions, and Supports
Gersten, Keating, and Irvin (1995) argue that assessment is only valid if it results in improved learning; in many systems, assessment data from district, school, classroom, and individual levels are not used effectively to improve outcomes. Systems level school improvement processes are essential to ensuring that all children have access to rich and challenging content on a daily basis. School improvement processes must look at classroom and schoolwide formative and summative data to determine research-based practices, including UDL, that can improve outcomes for the entire student population. Progress monitoring methods can give us data to determine whether some groups of children or individual children need additional interventions and supports. Challenges remain in using the data at all levels. School improvement teams, intervention specialists, those involved in pre-referral processes, and IEP teams must have access to and an understanding of research-based practices to effectively use all available data to make empirically based decisions about instructional programming.
Many states and districts are working to ensure that school improvement processes required by state and federal law consider the achievement of all students, and resulting school improvement plans provide universal access to challenging and effective curriculum and instruction. States and districts are also working to improve educators’ skills in designing interventions and managing pre-referral processes. Ultimately, if students are identified as requiring special education services and supports, then training and support on decision-making processes and best practices need to be in place for IEP team members. Learning how to balance competing learning needs while maintaining a longitudinal view of student learning is essential (McLaughlin, 2000). Thus, partnerships with experts and organizational resources on pre-referral, intervention, and IEP team processes are necessary. These need to be designed in a way that constantly emphasizes increased data, thought, and attention to longitudinal implications of instructional decisions. This will define the necessary changes in instruction, policy, and accountability for teachers and schools.
Challenge Implications: High Expectations
These three interrelated contextual challenges lead to an essential question that needs to be articulated and answered. That is, how do we build research-based understanding and stakeholder consensus on high student expectations that are essential to the success of all children? The expectation in No Child Left Behind is that all children will achieve at state defined proficiency levels. Yet, "by the end of kindergarten, the distribution of mathematical development in typical public school classrooms spans more than four grade levels, with many children, especially those from high-poverty backgrounds, demonstrating large deficits in their mathematical thinking and skills" (Fuchs, Fuchs, Yazdian, & Powell, 2002, p. 569). This is consistent with literature on early reading, which suggests that children from poor or minority backgrounds have a higher risk of poorer reading outcomes than white, middle class children (National Research Council, 1998). Simply put, children come to school as kindergarteners with extremely varied preparation for learning, regardless of any risk for disability determination.
The challenge of implementing a comprehensive progress monitoring improvement process to ensure that all children learn to high levels regardless of how prepared they are when they come to school is considerable but must be met. Addressing this basic question of expectations is central to the challenge, and is the foundation on which all efforts must rest. In the next section, we describe what such a process would look like, and identify multiple progress monitoring techniques that can be used.
A Comprehensive Progress Monitoring Improvement Process
Figure 1 shows a comprehensive progress monitoring improvement process to ensure that each child achieves to state defined proficiency levels. The essence of progress monitoring is that assessment data inform educators when students are not progressing as they should so that the educators can take action to improve student progress, whether they are performing above or below the grade-level targets. These actions might include changing instructional practices, adjusting curricula, and adding necessary services, supports, and interventions that will ensure that each child makes the progress expected for all children. This requires multiple methods that carefully measure progress on the development of basic skills in reading, math, and writing, as well as the challenging content in rich and varied content areas. These progress measures can provide data for whole school and district improvement, classroom improvement, and improvements for subgroups of children including those with disabilities, English language learners, and individual children.
Figure 1. Comprehensive Progress Monitoring Improvement Processes
In the past, some approaches to progress monitoring looked to expectations that were set individually through the IEP process, or to local or national normative data to gauge what the expectations for students should be, and thus what benchmarks were set throughout the year (Carnine & Granzin, 2001; Deno, Fuchs, Marston, & Shin, 2001). Placing these approaches within the context of standards set for the grade level, as required in current standards-based systems, is essential. Deno et al. (2001) addressed the implications of a normative approach to setting expectations, which assumes that the current typically observed growth rates for students with disabilities are reasonable and what should be expected. Following this line of reasoning, students with learning disabilities will learn at a slower rate than other students. The researchers speculate that this kind of reasoning reflects the "well-accepted fact that special education, as typically practiced in this country, fails to regularly incorporate demonstrably effective methods" (p. 515).
Under current federal requirements, for each classroom and for each child other than those included in the small exception for alternate achievement standards, the current achievement levels must be compared with yearend standards-based grade-level expectations. The learning slope represents the targeted rate of progress, and the student’s actual rate of progress should be monitored daily, weekly, monthly, or quarterly to determine whether yearend mastery is on target. If not, the teacher intervenes on instruction in hopes of accelerating learning. Then progress monitoring continues, and the efficacy of the intervention is determined, adjusted, and so on. Aggressive interventions can be implemented and monitored, with special attention paid to students who continue to be resistant to intervention treatment (Vaughn, Linan-Thompson, & Hickman, 2003).
We discuss the benefits and uses of progress monitoring methods and formative data sources in four general categories: (1) Curriculum-Based Measurement; (2) Classroom assessments (system or teacher-developed); (3) Adaptive assessments; and (4) Large-scale assessments used during the year to monitor growth of individual students and groups of students.
Benefits and Uses of Multiple Progress Monitoring Methods
A comprehensive progress monitoring improvement process includes multiple assessment methods that have different strengths and meet different needs. As these methods are implemented, we find that they tend to be blended in practice. For example, curriculum-based measures are used to target focused skill development which in turn helps inform the design of teacher-developed classroom assessments across content areas; classroom assessment strategies are informed by large-scale assessment data both at grade level and at instructional level (if it is different); reports of student progress to parents tend to reflect data from all measures in use. This is the ideal use of multiple measures for ensuring that all students succeed.
Curriculum-Based Measurement (CBM)
Experimental research on the use of progress monitoring to enhance student performance has focused primarily on one form of progress monitoring: Curriculum-Based Measurement (CBM). Academic areas frequently assessed using CBM include reading, mathematics, written expression, and spelling. Criteria that describe the measures used as part of CBM include valid and reliable indicators of generalized performance, short duration to facilitate frequent administration, a focus on direct and repeated measures of student performance, multiple forms that are inexpensive to create and produce, and sensitivity to changes in student achievement over time (Fuchs, Fuchs, Hamlett, Walz, & Germann, 1993). Especially important for ongoing classroom use is the demonstrated technical adequacy of the approach for accurate, easy, and repeatable measures (Crawford, Stieber, & Tindal, 2000; Crawford, Tindal, & Stieber, 2001; Helwig, Anderson, & Tindal, 2002; Helwig, et al., 2000; Ketterlin-Geller, McCoy, Twyman, & Tindal, 2003). Shinn and Bamonto (1998) estimated that over 150 articles have been published since 1988 on Curriculum-Based Measurement. CBMs have been used primarily in basic skill areas and in elementary school grades, but development work is taking place to expand their use to other content and to other age and grade levels (Espin, Busch, Shin, & Kruschwitz, 2001; Espin & Foegen, 1996; Foegen, 2000; Foegen & Deno, 2001; Gansle, Noell, & VanDerHeyden, 2002).
There are a number of strengths of CBM that other approaches view as challenges. For example, CBM has the advantage of simple hand scoring. Other classroom assessments often involve more time consuming, sometimes subjective scoring procedures. CBM has viable ways to conduct error analyses to identify specific targets for intervention efforts. Other approaches have not developed similar focused intervention tools. Graphing of student progress (baseline to target and trend line) has been used effectively in CBM (Calhoon & Fuchs, 2003; Deno, 1992; Fuchs, Fuchs, Hamlett, Thompson, Roberts, Kubek, & Stecker, 1994; Pemberton, 2003). Graphing provides a pictorial representation of the student’s progress so that teachers, parents, and students themselves can see exactly what student progress looks like. This can eliminate the false perceptions of where the child is academically and could potentially motivate students to continue working to reach the target. Although many forms of classroom assessment provide visual representations of student work compared to exemplars, these other approaches do not lend themselves to graphing. Other approaches to progress monitoring need to meet the challenge of providing similar high quality data tools and strategies.
Those who have been conducting the strong program of research on curriculum-based measures (e.g., see Deno et al., 2001) recognize that there are many issues yet to be addressed. These issues may include the best method for setting long-range goals, the frequency with which assessment should occur, the need to revise measures according to student performance level, and ways to help teachers use the data for instructional decision making. The clear commitment by researchers to continued rigorous debate, recognized by the editors of the 2002 special issue School Psychology Review: Special Topic Development in Academic Assessment and Intervention, is an outstanding resource to efforts to ensure that all children succeed (Daly & McCurdy, 2002).
Classroom Assessments (System or Teacher Developed)
Stiggins (2001) noted that the current "dismal state of classroom assessment" has "kept classroom assessment from even approximating its immense potential as a school improvement tool" (p. 5). Yet, progress monitoring in the context of federal legislation may be the catalyst needed to bring about a transformation in classroom assessment practices. Stiggins reinforces this, concluding that "we have an excellent foundation from which to develop strong classroom assessments" (p. 12). The same message is proclaimed by many other leaders in the field: that classroom assessment is important, that it typically is not done well now, but that teachers can be taught to do it well. Marzano (2003) bases his work on 35 years of research that shows progress monitoring (labeled goals and feedback) as one of five school-level factors and one of nine teacher-level factors that can improve student learning. Black (2003) and Shepard (2000) elaborate on effective feedback from classroom assessments. Wiggins and McTigue (1998) show teachers how to use standards-based classroom assessment as the starting point and then how to design backward from the skills and knowledge to be assessed to plan for instruction.
The importance of classroom assessment at this time is underscored by the recent formation of the National Research Council’s Committee on Assessment in Support of Instruction and Learning (see Bridging the Gap Between Large-Scale and Classroom Assessment, Petit, 2003). Projecting success on large-scale assessments by monitoring progress on classroom assessments will lead educators to improve the quality of classroom assessments and the alignment between classroom assessments and large-scale tests. Classroom assessment can take many forms. That is a feature that fits particularly well with an emphasis of the National Research Council’s Committee on the Foundations of Assessment (Pellegrino, Chudowsky, & Glaser, 2001, p. 293) on the "need for comprehensive systems of assessment consisting of multiple measures, including those that rely on the professional judgments of teachers." Several states continue to work toward a combination of summative state large-scale assessments and formative local classroom assessments as the best method of accountability. The kinds of observations to be made and the process for interpreting these observations vary according to the nature of what is to be measured.
A less teacher-dependent form of classroom assessment relies on assessments provided by publishers for use with books and other instructional materials. Publisher-supplied tests are valuable for progress monitoring only to the extent that the published materials and accompanying assessments are aligned with the target standards. Yet the potential is great. Pelegrino et al. (2001) cite developments in statistical modeling methods (e.g., Bayesian inference networks, or Bayes nets) that allow for more complex reasoning about complex student competencies – far beyond basic skills, and tailored to specific content domains. They speculate that by building these complex assessment approaches into intelligent tutoring systems or other instructional materials, teachers will have increased ability to understand student progress at multiple levels.
Adaptive Assessments
Adaptive assessments are tests that determine the items to which a student responds based on the student’s performance levels. Although this approach can be implemented without computers, it works best with computers. On computers it is called Computerized Adaptive Testing (CAT) (Thompson, Thurlow, Quenemoen, & Lehr, 2002). CAT improves testing efficiency and precision because instant item scoring lets the computer exclude tasks that are too easy or too hard for a student and focuses only on reasonably challenging tasks. Depending on their design and implementation, CATs can be administered several times each year. CAT systems typically have an Item Response Theory (IRT) foundation that provides a built-in scale suitable for reporting progress. One of the challenges for CAT is that unless the range of the assessment has been constrained, results may say very little about a student’s progress on grade-level standards (Olson, 2002). This is a serious constraint in the context of NCLB accountability requirements, and also could result in inadvertent lowering of expectations. If a core assessment is given that is controlled to remain at grade level only, and then additional items added to provide more precise diagnostic information, this serious limitation can be addressed.
Large-scale Assessments
The final category of progress monitoring approaches is different from the others described here. It involves the use of large-scale tests that are usually administered one to three times per year to show growth over time, and thus to monitor student progress. This is one goal of many states’ current efforts to vertically scale their annual tests and to maintain longitudinal student data. This effort involves implementing vertical scaling, which involves creating a single scale on which the test for each grade can be placed. There is considerable debate on whether the goal of vertical scaling is achievable or desirable. For example, there are fundamental questions about whether grade-level content lends itself to vertical explication, or if there are content "chunks" that are not coherently linked from grade to grade (e.g., panel presentations by Orr, Chin-Chance, Rabinowitz, & Vukminovic, 2003). Beyond this debate among measurement and curriculum experts, the extent to which this approach would meet the desired characteristics of frequent checks on student performance is questionable given the infrequent and summative nature of most of these assessments, and the time required for administration. Yet many states and districts continue to explore how a large-scale test can be used for multiple purposes, including tracking progress and growth of individual students as well as for diagnostic purposes. Whether that is a reasonable expectation is debatable, but there may be a place for use of large-scale assessment data in progress monitoring as one of multiple measures to be considered.
Broader Progress Monitoring Issues
Successful implementation of progress monitoring is not just a matter of picking an approach or a combination of approaches. Regardless of methods used, progress monitoring approaches in a standards-based assessment and accountability system must include defined strategies for scoring, analyzing, reporting, and tracking data, and defined strategies for creating meaning from the data gathered across all sources to develop effective improvement plans.
Strategies for Scoring, Analyzing, and Tracking Data
Solid strategies for local implementation of scoring and analysis of procedures depend on the nature of the assessment (e.g., hand scoring, computerized scoring). Identifying procedures for quick turn-around of scores is desired to provide feedback for instruction. It is important for districts and schools to keep longitudinal data for each student. At the classroom level, this could be relatively simple using paper-and-pencil or gradebook-like computer programs. But to allow the data to be rolled up or down to and from the school, district, or even state level, and to allow for year-to-year progress monitoring, more sophisticated computer programs and a system for uniquely identifying each student is needed. Using a relational database that would link progress in one area to progress in other areas and to other student or school information would allow educational administrators and researchers to try to understand differential student progress within the bounds of student privacy limitations. Web sites such as the American Association of School Administrators (http://www.aasa.org/cas/resources_and_tools.htm) list a range of such technology resources.
Deriving Meaning From the Data to Develop Effective Improvement Plans
The primary purpose of data, as shown in Figure 1, is to identify classrooms, subgroups of students within classrooms, and individual students who are not showing expected progress in the content areas. The benefit of progress monitoring relies on effective use of this information by teachers to identify areas of needed support, more instruction, or changes in instruction. It also supports effective communication with students and parents to help particular students or the entire classroom make greater progress. Teachers need help learning how to do this. Leadership, information and staff development are needed for currently employed teachers; changes in teacher education are needed to help teacher candidates (Stiggins, 2001).
The difficulty of understanding the meaning of data from multiple measures of progress monitoring for individual students and for groups of students and then choosing the best research-based interventions for a particular situation cannot be overemphasized. Numerous educators, researchers, and policymakers are grappling with this reality. Principals, teachers, parents, and students need the data on which instructional planning is based to be meaningful, understandable, and useful in making decisions on instructional interventions, services, and supports. Hand charting sets of data for entire classrooms has also been found to be effective for helping teachers and school level administrators get a feel for changes that may be needed at that level (Sargent, 2001). More complex graphics and statistical reports are needed to help administrators and researchers understand group progress from the classroom level on up to the state level (Pellegrino et al., 2001). In the long term, preservice training will play a role to ensure that teachers and school psychologists are equipped with the necessary knowledge and skills to use these tools and strategies when they start their careers.
District level administrators also need help in using data, as suggested by the quote from a superintendent in the forward to Using Data to Improve Schools: What’s Working: "We spend a lot of time on testing but not much time on what to do with the results" (AASA, 2002, p. iii). Administrators who use progress information effectively examine progress data to determine whether some teachers or schools are making more progress than others in particular curricular areas or with particular groups of students. Special attention is given to any areas where students are not on course to show proficiency on state standards. These data are combined with other information to, for example, make data-guided decisions about changes in organizational structure and staffing, allocating resources for additional help, and staff development. Staff and schools that seem to be particularly strong in some areas or with some students can be used as resources to help others in those areas. Similarly, resources might be committed to revising curricula or in other ways strengthening areas that seem to be a weakness for the entire system.
>Previous | Next  |
Information About PDF