U.S. Department of Education: Promoting Educational Excellence for all Americans - Link to ED.gov Home Page
OSEP Ideas tha Work-U.S. Office of Special Education Programs
Ideas that work logo
  Home  Contact us
Technical Assistance Products: Assessment
Instructional Practices
 Information About PDF

 Printer Friendly Version (pdf, 4.1M)

Designing from the Ground Floor: Alternate Assessment on Alternate Achievement Standards

Balance of Representation

Balance of Representation [D]

Flow diagram of balance of representation


The balance of representation criterion is used to indicate the extent to which items are evenly distributed across the content standards and the objectives under the content standards. In our measurement content standard with five objectives, we would expect items would be evenly distributed across the five objectives. In practice educational agencies may place greater emphasis on specific objectives and content standards. In this case the assumption of an even distribution would be replaced with the expected proportion, or emphasis, as specified by the educational agency. The formula used to compute the balance of representation index is the following

Equation - balance equals one minus the sum across objectives times the absolute value of one divided by the total number of objectives hit minus the number of items hit corresponding to objective k divided by the number of items hit for the content standard, all divided by two.)/2,

where O is the total number of objectives hit (i.e., item has been judged to be aligned) for the content standard, I k is the number of items hit corresponding to objective k, and H is the total number of items hit for the content standard. The balance index can range from 0 (indicating unbalanced representation) to 1.0 (indicating balance representation) with values from .6 to .7 considered a weak acceptable balance and values .7 or greater considered acceptable.

Depth of Knowledge

Depth of Knowledge

  • Consistency between the cognitive demands of the standards and cognitive demands of assessments
  • Recall (Level 1), Skill or Concept (Level 2), Strategic Thinking (Level 3) and Extended Thinking (Level 4)
  • We had a level 0 for prerequisite skills

Depth of Knowledge

  • Consistency between standards and assessments indicates alignment of what is elicited from the students is demanding cognitively as what is stated in the standard.
    • (Webb, 1997, p. 5)

Depth-of-knowledge (DOK) examines the consistency between the cognitive demands of the standards and cognitive demands of assessments. Important aspects of learning go beyond academic topics and include students' organization of knowledge, problem representations, use of strategies, and self-monitoring skills (Glaser, Linn, & Bohrnstedt, 1997). Completely aligned standards and assessments requires an assessment system designed to measure in some way the full range of cognitive complexity within each specified content standard. Webb identified four levels for assessing the DOK of content standards and assessment items. DOK levels are Recall (Level 1), Skill or Concept (Level 2), Strategic Thinking (Level 3) and Extended Thinking (Level 4). Of course to accurately evaluate the DOK level, each level needs to be behaviorally defined and examples given of types of student behaviors. A more descriptive example of DOK can be found in Table 1. This example was taken from an alignment study we conducted for assessments given to students with significant cognitive disabilities.

To examine the DOK, all item on the assessment and all academic content standards are rated for DOK. We expect assessments to have items that are below the expected DOK, but there should be items at or above the expected DOK. According to Webb, an acceptable level for the DOK is 50% or more of the assessment items are at or above the content standard DOK level. A weakly met criterion for DOK level would be between 40% and 50%.

Table 1

Mathematics Description of Depth-of-Knowledge Levels

Level 1

Level 2

Level 3

Level 4

Requires students to recall or observe facts, definitions, terms. Involves simple one-step procedures. Involves computing simple algorithms (e.g., sum, quotient). Requires students to make decisions on how to approach a problem. Requires students to compare, classify, organize, estimate or order data. Typically involves two-step procedures.

Level 2 items include word problems with simple one-step solutions, graphing and regrouping.
Requires reasoning, planning, or use of evidence to solve problem or algorithm. May involve activity with more than one possible answer. Requires conjecture or restructuring of problems. Involves drawing conclusions from observations, citing evidence and developing logical arguments for concepts. Uses concepts to solve non-routine problems.

Level 3 items include open-ended word problems where the operation is not given.
Requires complex reasoning, planning, developing and thinking. Typically requires extended time to complete problem, but time spent not on repetitive tasks. Requires students to make several connections and apply one approach among many to solve the problem. Involves complex restructuring of data, establishing and evaluating criteria to solve problems.

Level 4 items are project-based, involve explanation & justification.

Surveys of Enacted Curriculum

Surveys of Enacted Curriculum

  • Alignment of standards, assessments, instruction, and more
  • Use a two dimensional common content matrix (Content by Cognitive Demand)

The second alignment method we will discuss is the Surveys of Enacted Curriculum (SEC). The SEC alignment approach analyzes standards, assessments, and instruction using a common content matrix, which consists of two dimensions for categorizing subject content, which include content topics and cognitive demands (Porter & Smithson, 2001). Using this approach, content matrixes for standards, assessments, and instruction are created and the relationships between these matrices are examined. In addition to alignment statistics that can be calculated from the two-dimensional matrix, content maps and graphs can be produced to visually illustrate differences and similarities between standards, assessments, and instruction.

Example of Matrix [D]

example of a two-dimensional matrix

For illustrative purposes we will present all data using only three content areas and three categories for cognitive demand, producing a three by three matrix. In practice there are usually five or more content areas and six or more categories for cognitive demand.

To analyze assessments and standards, a panel of content experts conducts a content analysis and codes the assessment and/or standards by topic and cognitive demand. Results from the panel are then placed in a topic by cognitive demand matrix, with values in the cells representing the proportion of the overall content description. Each cell is the proportion of assessment items coded in each content topic by the cognitive demand. The cell with .30 indicates that 30% of the items were coded into the content topic area of operations and cognitive demand category of skill. The remaining cells indicate the proportion of assessment items that align to specific content topic and level of cognitive demand. The same procedure would be used to code academic content standards.

Content Mapping [D]

graphic of a content map

Porter (2002) reported that practitioners prefer visual representations of the matrix. The information in the matrix could be visually represented using content maps or surface area. The darker shades represent higher proportion or percentages of items. Most of the assessment items are in the content topic area of operations and at the cognitive demand categories of skills and application. The figure on the slide was produced using a simple spreadsheet program, but Porter and Smithson use software that produce much more sophisticated and detailed content maps.

Alignment Index [D]

graphic of an alignment index

Indices of alignment are created by a cell by cell comparison between matrices. The formula for calculated the alignment index is

Equation:  alignment equals one minus the absolute value of the matrix of assessment cell proportion minus the matrix of standards cell proportion, summed across all the cells divided by two.,

where X is the matrix of assessment cell proportion and Y is matrix of standards cell proportions. Table 3 demonstrates the calculation of the alignment index. In this example the alignment between standards and an assessment is being calculated. After subtracting the corresponding cells of the standards and assessment matrix and taking the absolute value, the elements in the new matrix are summed across all the cells. In this example, the sum of the elements in the absolute difference between the two matrices is .6. This value is divided by 2 and then subtracted from 1, which results in an alignment index of .7.

Alignment = 1-((.0+.2+.0+.0+.1+.1+.1+.0+.1)/2)=.7


 Previous  |  Next