
Designing from the Ground Floor: Alternate Assessment on Alternate Achievement Standards
Balance of Representation
Balance of Representation [D]

The balance of representation criterion is used to indicate the extent to which items are evenly distributed across the content standards and the objectives under the content standards. In our measurement content standard with five objectives, we would expect items would be evenly distributed across the five objectives. In practice educational agencies may place greater emphasis on specific objectives and content standards. In this case the assumption of an even distribution would be replaced with the expected proportion, or emphasis, as specified by the educational agency. The formula used to compute the balance of representation index is the following
)/2,
where O is the total number of objectives hit (i.e., item has been judged to be aligned) for the content standard, I k is the number of items hit corresponding to objective k, and H is the total number of items hit for the content standard. The balance index can range from 0 (indicating unbalanced representation) to 1.0 (indicating balance representation) with values from .6 to .7 considered a weak acceptable balance and values .7 or greater considered acceptable.
Depth of Knowledge
Depth of Knowledge
 Consistency between the cognitive demands of the standards and cognitive demands of assessments
 Recall (Level 1), Skill or Concept (Level 2), Strategic Thinking (Level 3) and Extended Thinking (Level 4)
 We had a level 0 for prerequisite skills

Depth of Knowledge
 Consistency between standards and assessments indicates alignment of what is elicited from the students is demanding cognitively as what is stated in the standard.

Depthofknowledge (DOK) examines the consistency between the cognitive demands of the standards and cognitive demands of assessments. Important aspects of learning go beyond academic topics and include students' organization of knowledge, problem representations, use of strategies, and selfmonitoring skills (Glaser, Linn, & Bohrnstedt, 1997). Completely aligned standards and assessments requires an assessment system designed to measure in some way the full range of cognitive complexity within each specified content standard. Webb identified four levels for assessing the DOK of content standards and assessment items. DOK levels are Recall (Level 1), Skill or Concept (Level 2), Strategic Thinking (Level 3) and Extended Thinking (Level 4). Of course to accurately evaluate the DOK level, each level needs to be behaviorally defined and examples given of types of student behaviors. A more descriptive example of DOK can be found in Table 1. This example was taken from an alignment study we conducted for assessments given to students with significant cognitive disabilities.
To examine the DOK, all item on the assessment and all academic content standards are rated for DOK. We expect assessments to have items that are below the expected DOK, but there should be items at or above the expected DOK. According to Webb, an acceptable level for the DOK is 50% or more of the assessment items are at or above the content standard DOK level. A weakly met criterion for DOK level would be between 40% and 50%.
Table 1
Mathematics Description of DepthofKnowledge Levels
Level 1 
Level 2 
Level 3 
Level 4 
Requires students to recall or observe facts, definitions, terms. Involves simple onestep procedures. Involves computing simple algorithms (e.g., sum, quotient). 
Requires students to make decisions on how to approach a problem. Requires students to compare, classify, organize, estimate or order data. Typically involves twostep procedures.
Level 2 items include word problems with simple onestep solutions, graphing and regrouping. 
Requires reasoning, planning, or use of evidence to solve problem or algorithm. May involve activity with more than one possible answer. Requires conjecture or restructuring of problems. Involves drawing conclusions from observations, citing evidence and developing logical arguments for concepts. Uses concepts to solve nonroutine problems.
Level 3 items include openended word problems where the operation is not given. 
Requires complex reasoning, planning, developing and thinking. Typically requires extended time to complete problem, but time spent not on repetitive tasks. Requires students to make several connections and apply one approach among many to solve the problem. Involves complex restructuring of data, establishing and evaluating criteria to solve problems.
Level 4 items are projectbased, involve explanation & justification. 
Surveys of Enacted Curriculum
Surveys of Enacted Curriculum
 Alignment of standards, assessments, instruction, and more
 Use a two dimensional common content matrix (Content by Cognitive Demand)

The second alignment method we will discuss is the Surveys of Enacted Curriculum (SEC). The SEC alignment approach analyzes standards, assessments, and instruction using a common content matrix, which consists of two dimensions for categorizing subject content, which include content topics and cognitive demands (Porter & Smithson, 2001). Using this approach, content matrixes for standards, assessments, and instruction are created and the relationships between these matrices are examined. In addition to alignment statistics that can be calculated from the twodimensional matrix, content maps and graphs can be produced to visually illustrate differences and similarities between standards, assessments, and instruction.
Example of Matrix [D]

For illustrative purposes we will present all data using only three content areas and three categories for cognitive demand, producing a three by three matrix. In practice there are usually five or more content areas and six or more categories for cognitive demand.
To analyze assessments and standards, a panel of content experts conducts a content analysis and codes the assessment and/or standards by topic and cognitive demand. Results from the panel are then placed in a topic by cognitive demand matrix, with values in the cells representing the proportion of the overall content description. Each cell is the proportion of assessment items coded in each content topic by the cognitive demand. The cell with .30 indicates that 30% of the items were coded into the content topic area of operations and cognitive demand category of skill. The remaining cells indicate the proportion of assessment items that align to specific content topic and level of cognitive demand. The same procedure would be used to code academic content standards.
Content Mapping [D]

Porter (2002) reported that practitioners prefer visual representations of the matrix. The information in the matrix could be visually represented using content maps or surface area. The darker shades represent higher proportion or percentages of items. Most of the assessment items are in the content topic area of operations and at the cognitive demand categories of skills and application. The figure on the slide was produced using a simple spreadsheet program, but Porter and Smithson use software that produce much more sophisticated and detailed content maps.
Alignment Index [D]

Indices of alignment are created by a cell by cell comparison between matrices. The formula for calculated the alignment index is
,
where X is the matrix of assessment cell proportion and Y is matrix of standards cell proportions. Table 3 demonstrates the calculation of the alignment index. In this example the alignment between standards and an assessment is being calculated. After subtracting the corresponding cells of the standards and assessment matrix and taking the absolute value, the elements in the new matrix are summed across all the cells. In this example, the sum of the elements in the absolute difference between the two matrices is .6. This value is divided by 2 and then subtracted from 1, which results in an alignment index of .7.
Alignment = 1((.0+.2+.0+.0+.1+.1+.1+.0+.1)/2)=.7
Previous  Next 