The dustup over charter schools reached the big time a few weeks back when it landed on the front page of The New York Times under the headline “Charter Schools Trail in Results, U.S. Data Reveals.” The story, which centered on a study by the American Federation of Teachers (AFT), led this way: “The first national comparison of test scores among children in charter schools and regular public schools shows charter school students often doing worse than comparable students in regular public schools.”
Reaction was swift. As Chester Finn Jr., a leading right-wing education-policy expert, described it on his blog, “Charter supporters rushed to the barricades after last week's AFT-coordinated blast in The New York Times. On Aug. 25, 31 policy types and number crunchers ran a full-page ad in the Times rebutting some of the claims.”
The statement concludes that the AFT “study does not meet professional research standards. As a result, it tells us nothing about whether charter schools are succeeding.”
What makes the criticism from the right especially interesting is that in their own attacks on public education, many of these guardians of professional research standards have repeatedly violated the principles they now proclaim. But that's a historical tale for another time. What is really telling right now is the conservatives' reaction to a new study on charter-school performance by Harvard professor Caroline Hoxby, perhaps the leading conservative, pro-voucher/charter/free-market-education economist.
The New York Post opened fire with its own story. “Charters Get High Grades In Report” leads with, “Kids in charter schools do better on reading and math tests than traditional public-school students, a new national study shows” -- almost parallel to the lead on the much-vilified New York Times story. This leaves conservatives with a big dilemma, because Hoxby's analysis fails to meet the right's own criteria for good research on charter schools (more on that below). So what will right-wingers say about Hoxby's study and the New York Post's coverage of it?
Early indications are that double standards will abound. A blogger who's a self-defined centrist claimed that Hoxby's study is “much more sophisticated than the recent AFT report.” Sophisticated? Hoxby herself notes, “No complicated statistics are used … [just] the calculation of standard errors.”
By the way, that's the same level of statistical analysis as the AFT report, compiling data so that one can examine how one dimension (charter/noncharter) compares with another (test scores) and judging whether charter schools have higher or lower scores (using standard errors, or in polling terms, the margin of error, to determine whether a difference is statistically significant or not). This is the very type of analysis that many government agencies present, such as when they report poverty or unemployment rates. Calling Hoxby's research “sophisticated” may say more about the blogger's ability to judge research than it does about the research itself. It also suggests that the centrist blogger's characterization of the AFT report as a “hatchet job” should be served up with a ton of salt.
Granted, Hoxby's research does cover a larger sample of schools than the data used by the AFT. Hoxby's data, however, are weaker than the AFT's in two important ways. First, her assessment of school outcomes is based on the share of students who are proficient at reading or math but not the average test score of the students. That's like knowing the poverty rate but not the average income of a community -- useful but incomplete. The AFT analyzed both test scores and proficiency levels. Second, Hoxby controlled for student characteristics less ably than the AFT did (more below).
Hoxby was one of the signers of The New York Times ad criticizing the AFT's study. So how did she do at meeting her own standards? Not so well. Let's look at what the ad says and how Hoxby did on three key criteria that she and her conservative colleagues set out.
First, data quality is a big issue in the ad. It says: “But since only limited family background information is currently available for the 2002 NAEP [National Assessment of Educational Process], the study (AFT) does not provide reliable information on the effectiveness of any particular school.”
Hoxby includes no information on family background of the charter-school students, or of the equivalent students in public schools. She claims to be making an “apples-to-apples” comparison when she compares charter school test proficiency levels to those of the nearest public school. However, that is true only if we accept her entirely unsubstantiated assertion that the populations of these schools are the same as that of the charter schools. Maybe, but probably not. Hoxby didn't have to speculate on this -- the data were available to her. It makes me wonder why she didn't use them.
In fact, there's every reason to think that charters have different types of students than nearby schools. Finn's critique of the AFT study acknowledges as much. “In fact,” he wrote, “the variability among charter schools surpasses that of regular public and private schools. That's one reason they're hard to study -- because having a ‘charter' may be less important than the school's core mission, which might be dropout recovery, or the arts, or bilingualism, or giving new options to disabled children.”
Second, the ad criticizes the AFT for using only test scores from one point in time. “But without better background information,” the ad reads, “accurately measuring school effectiveness requires information on student performance from at least two points in time.”
The ads signers are arguing, quite reasonably, that it's important to look at how test scores change over time. This helps keep us avoid ascribing poor performance evaluations to schools whose students are poorer or challenged, because they tend to have started kindergarten with much less developed cognitive skills (i.e., lower scores). Consider this quote from The New York Times: "'[Charter schools are] doing so much to help kids that are so much farther behind, and who typically weren't even continuing in school,' said Jeanne Allen, president of the Center for Education Reform, in Washington, which represents charter schools and sponsored the ad. She said the [AFT] results reflect only ‘a point in time,' and said nothing about the progress of students in charter schools."
And Hoxby? She totally misses the mark on this criterion because she has educational-proficiency measures for only one year and, what's more, she does not include detailed student background information.
In contrast, the AFT study compares charter and public school students by race/ethnicity, by eligibility for free/reduced-price lunch (reflecting low-income), and by location of school (showing whether the school is located in a central part of a city, on an urban fringe, or in a rural area). The AFT analysis was not able to control for many factors at the same time, nor did it examine test scores over time. I would agree that the results are not the last word, though they are informative.
Finally, the ad faults the AFT's report for including limited background information. “The study,” the ad reads, “does not take into account such key characteristics known to affect their performance as parental education, household income, and the quality of learning resources in the home.” As you can guess, Hoxby's research has no controls for these characteristics.
So, the conservatives are facing a challenging test. Will they stick to the principles they laid out in the Times ad and criticize one of their own? Or will ideology trump principle? (Drumroll offstage.)
Not to keep us in suspense, Finn has already flunked this test by praising Hoxby's new study, saying, “She's done a far better study [than the AFT], and it yields far different results.”
I would question whether Hoxby's study is better from a research standpoint than the AFT's, but the fact is that the conservative scholars set out standards for professional standards that neither the AFT's nor Hoxby's study meets. When it comes down to applying these standards, the conservatives have a decidedly double standard, painfully evident in Finn's justification of his praise for Hoxby:
“As Hoxby acknowledges, it's not a perfect study; like the NAEP-based analysis, it's a snapshot of student performance at a single point in time and does not show ‘value added.' Nor is it based on randomized assignment of children to charter and district schools. But it's promising enough, as the author remarks, not to blow the whistle on charter schools, as the AFT and their allies would do, but to ‘make us patient enough to wait for the results of multi-year studies based on random lotteries among charter school applicants.'”
In other words, “Never mind all those highfalutin principles, I like the results.”
Lawrence Mishel is the president of the Economic Policy Institute (EPI). He writes a monthly column on economic issues for the Prospect's online edition.
