Unharmonic Convergence
April 29, 2010RTTT Update
April 30, 2010Fact-Checking Linda Darling-Hammond
Bob Braun at the Star-Ledger writes of renowned education scholar Linda Darling-Hammond’s lecture in New Brunswick this week in which she lauds New Jersey’s success in closing the achievement gap among White, Black, and Hispanic students. “She listed measures of success in New Jersey — higher graduation rates, higher test scores, higher national rankings. Darling-Hammond drew gasps of appreciation by noting that, on one national exam, the average scores of black and Latino students in New Jersey were as high as the average scores of all students in her home state, California.”
Let’s put aside graduation rates for the moment (though just for the moment) and look more closely at the data that Darling-Hammond cites. There’s only one national test that NJ and California students take: the National Assessment of Educational Progress, fondly known as the NAEP. And while it’s true that average scores in California for all 4th and 8th graders (the two age groups tested by NAEP) are comparable to average scores for Black and Latino students in NJ, there’s one piece of data missing from Dr. Darling-Hammond’s analysis: 53% of California’s students are eligible for free and reduced lunch, the metric for establishing economic disadvantage. In contrast, 31% of NJ’s 4th graders are eligible for free and reduced lunch and 26% of 8th graders are. So the differences in state averages directly correlates with degree of affluence. On average we’re richer here than in California. Our kids score better on tests.
Let’s dig a little deeper. The 4th grade reading results for California show that white 4th graders average 227 points. Black Californians average 27 points lower (200) and Hispanic 4th graders average 31 points lower (196). 4th graders in NJ have a slightly smaller gap. White students in NJ average 237 points. Black and Hispanic NJ 4th graders average 213 (24 points lower). The 8th grade reading test results demonstrate more similar profiles. In California, 8th grade white students average 269, while black 8th graders average a 26 point achievement gap and Hispanic students average a 28 point gap (243 and 241 respectively). In New Jersey white 8th graders average 281 points. Black students average 250 points, a 31 point gap, and Hispanic students average 256, a 25 point gap.
Not much difference. While we can be justifiably proud of our test score averages, we might as well be proud of affluence. Here’s the bottom line from the NAEP commentary:
In 2009, the score gap between students in New Jersey at the 75th percentile and students at the 25th percentile was 41 points. This performance gap was not significantly different from that of 2003 (43 points).
Dr. Darling-Hammond’s lecture was sponsored by the Education Law Center, not a disinterested party. ELC is currently fighting Gov. Christie’s school aid cuts and arguing that the State is obliged to fund poor students at the same rate as rich students, the backbone of the Abbott decisions, now codified in the School Funding Reform Act. Bob Braun writes in the article, “[h]er purpose was to describe how, at least until recently, New Jersey’s dramatic, if expensive, policy of ensuring financial parity in the public schools had actually begun to pay off.” That may well be, but her Exhibit A –the NAEP scores – don’t do much to advance the argument that our system of urban education for impoverished students in NJ has borne much fruit, or at least done much for the achievement gap.
2 Comments
There are major problems with simplistic comparisons of state to state NAEP results.
For one thing, the proportion of students classified as English Language Learners (ELL) is dramatically different in these two states.
In the 2009 NAEP fourth grade reading samples:
Percent ELL identified in California – 30% of raw sample selected to test, but only 1% of the raw sample was excluded as ELL
Percent ELL identified in New Jersey – 4% of raw sample, and 3% of raw sample was excluded as ELL.
In other words, almost all in California took NAEP regardless of ELL status, while 75% of the ELL in New Jersey got excluded.
In California, fourth grade ELL scored 184 while non-ELL scored 220.
In New Jersey, fourth grade ELL DID NOT EVEN GET SCORES BECAUSE TOO FEW WERE LEFT IN THE TESTED SAMPLE TO ALLOW VALID RESULTS! (not shouting, but really emphasizing this huge issue).
In grade 8 the numbers are:
Percent ELL identified in California – 20% of raw sample selected to test, but only 1% of the raw sample was excluded as ELL
Percent ELL identified in New Jersey – 2% of raw sample, and 2% of raw sample was excluded as ELL.
So, within rounding error, NO ELL eighth grade students in New Jersey got to sit for the NAEP.
In California, eighth grade ELL scored 215 while non-ELL scored 261.
In New Jersey, eighth grade ELL DID NOT EVEN GET SCORES BECAUSE TOO FEW WERE LEFT IN THE TESTED SAMPLE TO ALLOW VALID RESULTS! (not shouting, but really emphasizing this huge issue).
Since Ms. Darling-Hammond comes from California, it is hard to understand how she missed this huge demographic disparity.
There are two other areas I'll just briefly touch on:
First — uneven exclusion of learning disabled students.
California only excluded 3% of its entire fourth grade raw sample NAEP wanted to test due to severe learning disabilities (SD).
New Jersey excluded 7% of its raw sample.
In eighth grade, California excluded just 2% of its raw sample due to SD while New Jersey excluded 5%.
Because SD kids score very low on NAEP, it doesn't take a lot of them to start moving scores around. In the current case, the differences between New Jersey and California are notable and probably impact scores by at least several NAEP scale score points to California's disadvantage.
Second — Poverty rates vary dramatically in the two states. That also makes simplistic score comparisons dubious.
Basically, given the HUGE demographic differences and the very different exclusion rates for SD, along with the poverty issue Jill discussed in her post, there is no way that simplistic comparisons of scores between California and New Jersey can provide a fair comparison.
For a short, but important, general primer on these and other problems with NAEP interpretation, log onto this address:
http://www.freedomkentucky.org/index.php?title=The_National_Assessment_of_Educational_Progress
Richard Innes
Education Analyst
Bluegrass Institute for Public Policy Solutions (www.bipps.org)
It sounds like we're saying the same thing: Dr. Darling-Hammond's direct comparison of California and NJ NAEP scores is not terribly meaningful: not only are more CA students more impoverished, but NJ manages to exclude the majority of ELL's from test results. In addition, as you also correctly point out, NJ excludes many more children with disabilities from testing, which artificially raises our scores.