What is a NAEP Framework?

It is the blueprint for an assessment.  In general, NAEP frameworks have been documents drawn up by teams of teachers, school officials, and members of the general public to guide item development and scoring of items in each subject.

NAEP frameworks generally have a ten-year lifespan.  The 1992 Reading framework was developed in what the National Assessment Governing Board (NAGB) calls a “national consensus effort” managed by the Council of Chief State School Officers (CCSSO).

The NAEP Mathematics framework, which has been revised several times since 1990 to align with National Council of Teachers of Mathematics (NCTM) standards, is due to be replaced in 2005.

The 2007 NAEP Reading framework is currently being developed by a national committee of educators supervised by American Institutes for Research (AIR). 

How Do NAEP Frameworks Generate Assessment Results?

NAEP reports results for states and the nation as scaled scores, percentiles, and in relation to Achievement Levels.  NAEP also reports subscale scores corresponding to Reading contexts and Mathematics strands described in their respective frameworks:

Reading Contexts         

Mathematics Strands  

How Do NAEP Frameworks Compare with Maine ’s Learning Results?

NAEP Reading subscales are based upon items sorted by passage type using the contexts specified by the NAEP Reading framework; MEA categories of items are structured differently because they are based upon Learning Results standards

In addition, NAEP Reading and Mathematics surveys use sets of sub-categories that are not reflected in the scaled scores.  The NAEP Reading framework also specifies four stances; NAEP Mathematics, two domains:      

Reading Stances      

Mathematics Domains

These sub-categories are used to classify and distribute the items in an assessment for a given year.  There appears to be a correspondence between NAEP Mathematics strands and the proposed MEA Mathematics clusters; the distribution of categories of items in both assessments is roughly similar if counted at the strand/cluster level.

The distributions of multiple choice and constructed response items in NAEP and MEA assessments differ in that NAEP uses more constructed response and MEA uses more multiple-choice items overall, but this effect is offset by assigning different values to the different item types in MEA scoring.

Since the NAEP and MEA Reading blueprints differ so greatly, direct comparison of results cannot really be made except in respect to NAEP Achievement Levels, which represent the application of standards to the NAEP scaled scores.

NAEP Achievement Levels

In order to interpret the scaled scores in terms of student proficiency in each subject, the National Assessment Governing Board (NAGB) has developed Achievement Levels of Basic, Proficient, and Advanced in mathematics, reading, history, geography, science, writing, and civics since 1990.  These are more easily compared with performance levels in Maine ’s Learning Results.

While MEA scores are directly connected to standards set by Maine ’s Learning Results, NAEP scores are based upon the comparative performance of students across the nation.  Maine students participating in an assessment make up a representative cross-section of the student population of the state.  In 2000, fourth graders in Maine produced an average NAEP mathematics scaled score of 231 (compared with 226 nationwide).  On the NAEP achievement scale for Mathematics, 214 is the lower cut-off point for Basic and 249 is the lower cut-off for Proficient.  Only seven other states had average scores that were higher than Maine ’s, and those were only 1 to 3 points higher.  Statistically speaking, Maine ’s 4^th graders performed on average as well as or better than their peers across the nation, but the majority of students sampled in the state and across the nation ranked in the less-than-proficient range.

The percentage of Maine 4^th graders at or above the Proficient level in Mathematics on the 2000 NAEP assessment was 25%, compared to 23% of the students meeting or exceeding standards on the MEA.  Clearly, both MEA and NAEP standards are rigorous.

Using NAEP Item Maps

The NAEP Achievement Levels have another implication for state assessments as a result of NAEP’s development of item maps.  The Item Response Theory (IRT) statistical model that NAEP uses to develop scaled scores can be used to predict student performance on specific items. Each year NAEP releases 25% of the items used in that year’s assessment.  IRT modeling can place these items on the achievement scale to show how the ability to answer a specific type of question correctly would relate to the overall achievement of students participating in the assessment.

For example, a 4^th grade student performing at the Basic level overall on the Mathematics assessment would probably be able to answer the following question correctly:

“Determine how much change a person will get back from a purchase.”

But a student also answering the following question correctly has a high probability of achieving the Proficient level in 4^th grade Mathematics:

“Apply the concept of symmetry to visualize the result of folding a marked strip of paper.”

This suggests that specific MEA and NAEP items might be compared in content to make connections between the two assessments.  Released NAEP items can be found online.  Log on to the National Center for Educational Statistics (NCES) web site (www.nces.ed.gov).  Click on the arrow at the corner of the “Visit Popular NCES Sites” window, and select “Data Search Tools.”  Click on the link “National Assessment of Educational Progress (NAEP) Questions Tool” about halfway down the page. 

Comparing Scores

NAEP and MEA scoring scales are not as similar as they may seem.  MEA scores are statistically computed from individual students’ raw scores, while NAEP scores are derived from abstract statistical values called thetas.

Both assessments use IRT statistics, but each uses a different method and uses its results for different purposes.  MEA uses IRT for equating, which is a way to ensure that students taking different tests in different years in the same subject at the same level receive comparable scores.  MEA does not use IRT to create the scores themselves; NAEP does—because the Nation’s Report Card is building a projection of student performance at the state and national level.

MEA results have implications for instructional practices at the school level, while NAEP results do not. NAEP does not report scores at the school level because of the sampling methods it uses.  Each student participating in a NAEP assessment sees only a small portion of the items for that subject and level. 

Schools and students are selected by NCES across the state to provide complete coverage of the assessment items for a subject and level by a student population representative of the entire state. The same kind of sampling of students and items is done to generate a prediction of student performance for the nation as a whole.  It is unlikely that the group of students selected for a single school in Maine will see all of the items in an assessment.

Both are valuable information resources for Maine schools in complying with NCLB requirements.  The federal government takes full financial responsibility for the NAEP assessment.  The state’s contribution to the Nation’s Report Card is the cooperation of some of its schools for a few hours each year or two.

NAEP State Coordinator for Maine : john.kennedy@maine.gov

All current NAEP frameworks may be viewed on- line at the NCES website.