2002 Reading Results Released

OVERVIEW

The National Assessment of Educational Progress (NAEP) is part of a comprehensive assessment plan for Maine.  It is important to understand how NAEP is different from the state and local assessments now being developed in Maine.  In particular, small changes in NAEP numbers between years are not necessarily significant because these results are taken from a statistical model of student performance and not the actual scores of individual students across the state and nation.  For further discussion of the differences between Maine’s assessments and NAEP, see NAEP Newsletter #2, available by e-mail or post from the NAEP State Coordinator for Maine (john.kennedy@maine.gov ).

MAINE RESULTS

In general, Maine continues to lead the nation in reading scores.  In 4^th grade Reading, only Massachusetts and Connecticut had average scaled scores that were higher and only by a few points.  In 8^th grade Reading only the Department of Defense schools had higher scores on average.

The gender gap in 8^th grade reading became smaller both in Maine and in the nation.  In both jurisdictions, the 8^th grade gap has consistently been larger than the 4^th grade gap and females have scored higher than males.  In 2002, the 8^th grade males’ scores improved and the 8^th grade females’ average scaled scores experienced a small but significant decline.

The 8^th grade gender gap in reading was larger in Vermont and Connecticut and smaller in Delaware and Rhode Island .   

Maine is in the process of implementing intensive reading assistance programs in the lower grades, and the Department of Education (DOE) will be following the 8^th grade females’ state and national reading scores with particular interest as these initiatives go forward.  Scores for 2003 reading should give us additional information about this phenomenon, and it should be noted that we have only 1998 and 2002 8^th grade reading scores to compare for NAEP.  4^th grade NAEP reading scores are available back to 1992 for Maine and remain stable.

On student questionnaires, 57% of the 4^th grade girls assessed responded that they read for fun on their own almost every day compared to 38% of the boys.  34% of the boys responded that they hardly ever talk with friends about what they read compared to 22% of the girls.

Exclusion rates in Maine, which are in the mid-range compared to the rest of the nation, did not appear to have a significant effect upon scores; some of the other states excluded twice as many students, and some excluded half as many—but overall there did not seem to be a correlation between number of students with disabilities exempted from the assessment and average scale scores or national ranking of states.

NAEP developed a new very experimental statistical method to predict what average state scores would have been if no student had been excused from taking tests.  These alternative scores will be reported by NAEP with cautions about their use starting this year.  In Maine ’s case, there was no significant difference between the regular and experimental scores.

The gap between average scaled scores for students with disabilities and those without was smaller than the national average.  NAEP data suggest that Maine has a higher percentage of students with disabilities than the national average.  This data was provided by the schools and has remained stable since 1998.

No ethnic minority population in Maine exists in large enough numbers for NAEP to predict subgroup performance.  In the past, scores for 4^th grade Hispanic students have been generated on the basis of student-reported information.  This resulted in unrealistically high projections of this population in Maine .  Therefore the scores reported for this sub-group cannot be considered reliable.  In the future, NAEP will be reporting scores in relation to ethnicity only on the basis of school-reported and census information.

There was no significant difference in the performance of students in rural and urban areas in Maine , and Maine continues to have smallest gap in the scores of poor and non-poor students in the nation.

Student reported data suggest that there was a significant increase in the percentage of 4^th graders whose parents graduated from college.  Also, 89% of the 4^th graders and 93% of the 8^th graders surveyed indicated that there was a computer in their homes.

4^th grade teacher-reported information suggested that 98% of the teachers of assessed students integrated reading instruction into other subjects.  The teachers’ questionnaires also indicated a trend in the use of instructional materials that ran counter to the national trend.  In Maine , 63% of the teachers responding indicated that they used trade books for the core of instruction compared to 16% nationwide.  Nationally, 67% of the respondents reported using a mix of trade books and basal readers compared to 33% in Maine .

These data support the perception that Maine has never fully rejected the basal approach but generally expects students to learn reading by reading real books.  Also, reading instruction in Maine makes use of close observations of individual students and miscue analysis to work with individuals and small groups in reading remediation while the basal approach depends largely upon large group, standardized instruction.

93% of Maine ’s 4^th grade teachers also reported having release time for professional development.  87% indicated that their schools provided additional professional development related to standards or assessments to a moderate or large extent.  85% responded a moderate to large influence of professional development on their language arts instruction.

NAEP PUBLISHES LAST STATE REPORTS

With the release of 2002 reading results, NAEP has issued its last state reports; in the future, NAEP State Coordinators will each write reports tailored to the concerns of their individual states.

The NAEP 2002 Reading Report Card for Maine may be obtained by writing to J. H. Kennedy, 223 State House Station, Augusta, Maine 04333 or calling 207-624-6636.  It is also available online at NAEP’s website (nces.ed.gov/nationsreportcard).

NAEP DATA NOW AVAILABLE TO THE PUBLIC ONLINE

Teachers, parents, administrators, and the general public can access all of NAEP’s published data in all of the subjects assessed by logging on to the National Center for Educational Statistics (NCES) web site (www.nces.ed.gov).  Click on the arrow at the corner of the “Visit Popular NCES Sites” window, and select “Data Search Tools.”  Click on the link “National Assessment of Educational Progress (NAEP) Data Tools” about halfway down the page.  Results for Maine , other states, or the nation can be accessed through the Data Tool. 

CAUTIONS ABOUT DRAWING CONCLUSIONS FROM NAEP DATA

Depending upon a number of statistical factors, a difference of a few points may not mean that the performance of the group assessed (state, region, or nation) has changed over time.  Users of the NAEP Data Tool can perform their own tests of Statistical Significance by selecting this option from the User Options menu choices.

While Maine ’s NAEP scores are among the highest in the nation, more students fall in the NAEP Achievement Level of At or Above Basic than Proficient.  This predicts that most of the 4^th and 8^th graders in the state have the basic skills to progress over years to the level of Proficient.  NAEP goals are long-term and Maine is on track to meet them. 

Since Maine Educational Assessment (MEA) standards are also long-term and set quite high, percentages in the MEA category of Meets or Exceeds Standards can be close to those for NAEP’s Proficient.  However, it is important to remember that these are two different ways at looking at student performance.  NAEP and MEA scores are not directly comparable because they are measuring the same student populations but in very different ways for different purposes.  See NAEP Newsletter #1 for additional discussion of this issue.

MAINE ’S READING 2002 DATA

COMMISSIONER’S PRESS CONFERENCE

Commissioner Gendron made the following statement to the press in Portland on June 19.  An  excerpt from the Q & A sheet distributed at the Commissioner’s press conference  follows.

“Though we are very pleased with the overall results, our initial focus needs to be on the gender gap.  It is my intention to form a Commissioner’s Task Force on Gender Achievement Differences to clearly define the extent of problem, identify current best practices, and analyze research findings on both causes and solutions.  My hope is to have results reported from the Task Force and an action plan developed by the end of this calendar year.”  Susan A. Gendron, Portland , June 19

Q.  Why does NAEP include only some students in some schools in Maine that it selects for its assessments?  Why don't they just select whole schools on a rotating basis?

A.  It would be prohibitively expensive to test every student in the nation or in the state to produce national and state results, so NAEP uses a method of question sampling and student sampling to produce a statistical model predicting the performance of the entire population of students at selected grade levels.  The sampling of students involves finding a specific distribution of demographic features in the population selected as the sample.  In a state with a relatively small population spread out over a large area, it is very difficult to find a completely representative sample within a few schools or districts.

Q.   How are Maine students in the sampled schools selected for participation in NAEP assessments?

A.   NCES collects information from the states about the makeup of their student populations, as well as information from other sources (such as the U.S. Census), and builds statistical models of the nation’s and state’s schools.  They then make selections according to rules for obtaining a statistically reliable sample of a relatively small group of students to represent the entire population.  In 2003, 5,500 schools and 140,000 students were selected at the 4^th grade and 4,700 schools and 115,000 students were selected at the 8^th grade nation-wide.

Q.  Why do Maine schools have to give both Maine Educational Assessment (MEA) and NAEP tests in the same subjects?

A.  Both types of tests are mandated by the No Child Left Behind (NCLB) Act and serve different purposes.  MEA focuses upon issues within the state and can give information about performance at the student and school level.  NAEP assessments allow states to compare their performance to each other and to the nation overall.  In Maine , both types of assessment are combined with a strong local assessment initiative to provide the educational system and the public a complete picture of the progress our schools are making in meeting standards.

Q.  Why was the NAEP Reading Assessment administered to Maine's students two years in a row?

A.   NCLB required NAEP to move from a four-year cycle to a two-year cycle of testing in reading and mathematics.  In order to bring the two subjects into the same testing year, the reading assessment was moved up three years, and the mathematics assessment was moved up one year from the previous NAEP schedule.  In the future, reading and mathematics will be assessed by NAEP in odd-numbered years.

Q.  How can Maine 's students be said to have done well on the NAEP assessment if only 35% of the 4^th graders and 38% of the 8^th graders are identified by NAEP as being at or above Proficient?

A.  72% of Maine's 4^th and 82% of Maine's 8^th graders are identified as at or above Basic.  This means that the vast majority of these students possess the basic skills necessary to work towards the eventual national goal of proficiency.  NAEP defines Proficiency as demonstrating "competency over challenging subject matter" (see page 3 of the 2002 Report Card for Maine), which is a goal that the nation is intended to achieve over the next decade.

NOTE: 4^th and 8^th graders are placed on the same scoring scale.  One sign of progress is the percentage of students that have moved out of the Below Basic category between 4^th and 8^th grade.

Q.  Why does a difference of a few points in NAEP average scaled scores sometimes mean something and sometimes not?

A.  NAEP data is produced by a method somewhat like a survey of public opinion and therefore has a sampling error based upon the size of the population surveyed and the degree to which it is representative of the greater population that it represents.  In order for the result of a NAEP assessment to be valid, the data must pass a statistical test that determines its significance.  In reality, NAEP scaled scores represent ranges of scores when the sampling error is factored in, so 225 and 227 may actually represent the same performance of a specific population.