W ith George W. Bush's assumption of the presidency, a campaign to provide vouchers for private schooling may gain new life. The idea of public funding of private schools is not new, nor does it belong exclusively to conservative free market reformers.
In the 1960s and early 1970s, academics on the left, such as Christopher Jencks of Harvard University's Kennedy School of Government, argued that vast differences between the quality of public schooling for inner-city blacks and suburban whites could not be resolved within the structure of a residentially segregated public-education system. Jencks argued for a policy concept introduced by economist Milton Friedman more than a decade earlier. Friedman proposed to offer public funds to families that could only be used for education, but in any educational institution, public or private. Such "vouchers" would serve to give families increased choice about the kind of education their children received. Friedman saw vouchers as a way to break the "monopoly" of the public sector over education, increase consumer choice, and, hence, promote economic well-being. Jencks saw vouchers as a way of improving educational opportunities for a historically discriminated-against group within American society. Both shared a distrust of the state--Friedman of the bureau-centric state interfering with "democratic" markets, Jencks of the class-centric and race-centric state reproducing inequality through public education.
But conditions may have changed in the last 40 years. While there is still a glaring gap between the achievement of black students and that of white students, it has been considerably narrowed. In the last decade, however, the progress seems to have stopped, and it is unclear what the causes of the continued disparity might be. In his latest book (co-edited with Meredith Phillips), The Black-White Test Score Gap, Jencks seems to argue that the biggest cause of the persistent gap is differences in family characteristics over which schools, public or private, have very little control [see also Christopher Jencks and Meredith Phillips, "America's Next Achievement Test: Closing the Black-White Test Score Gap", TAP, September-October 1998]. But he does suggest that some school improvements, like smaller classes or better-prepared teachers, might make a difference.
With so much attention focused on the problem, voucher advocates have been attempting to prove that private schools supported by public funds actually can do a better job than public schools of educating the children most at risk of school failure--whether because vouchers are a route to smaller classes and better teachers, or because private schools are superior in other respects. Over the last few years, the leading proponent of the idea that private schools are demonstrably more effective at educating low-income African-American students has been Harvard Professor Paul Peterson.
In the most recent salvo in this dispute, Peterson and his colleagues (David Campbell, also of Harvard; William Howell of the University of Wisconsin, Madison; and Patrick Wolf of Georgetown University) announced last August that their voucher experiments in New York City, Washington, D.C., and Dayton, Ohio, showed that at least some pupils--African Americans--achieve better in private than in public schools. The finding was widely hailed by voucher supporters across the political spectrum as showing that private schools could solve a problem public schools apparently could not: that of raising the lagging achievement of low-income inner-city black children.
"This hard evidence is not what teacher unionists want to hear," William Safire noted in The New York Times. "The Harvard study shows Bush is on the right side of this. He should embrace the successful voucher students and joyfully join the controversy."
But soon after the results were presented, another member of the Peterson team, David Myers, contractor for the New York City part of the research, openly challenged Peterson's interpretation, arguing that the New York results do not show voucher students--even African Americans--with a statistically significant advantage. Earlier voucher studies in Milwaukee and Cleveland seemed to support this more skeptical view.
Who is right? How sanguine should we be that black, inner-city pupils would gain by switching to private schools?
The short answer is that the three-city study is not nearly as reliable as its authors claim. As a basis for educational policy, it should be interpreted cautiously. A more structured private-school environment with smaller classes and higher-achieving peers could possibly help African Americans make greater gains than if they stayed in public schools. It is also possible that improvements to public schools would yield comparable improvements. But that said, the Peterson results may misrepresent gains that typical low-income African-American students can make by switching to private schools. Using statistical techniques not easily understood by the media or the public, the studies' methodology is laced with potential biases. In the context of an intense ideological push for privatizing education, the question to ask is not whether this latest report overestimates private school effects, but by how much.
In four cities--Dayton, New York, Washington, and Charlotte, North Carolina (where data was released more recently)--the Peterson team built evaluations into the voucher plans themselves. Evaluating these evaluations is not easy, because the researchers have not publicly released their data.
Earlier, in Milwaukee and Cleveland, the Peterson studies were constructed after the fact, in response to research originally carried out by scholars not politically committed to vouchers. The Peterson estimates in those studies have a different character. For one, the data were available to others and thus are subject to re-analysis. More important, "experimental" controls were nonexistent.
The Milwaukee Voucher Experiment
The longest-running voucher initiative in the United States is Milwaukee's. It began in 1990-91 on the initiative of Polly Williams, an African-American Wisconsin legislator. The $2,500 vouchers were awarded by lottery to very-low-income families, almost all of them African Americans, to be used only in secular private schools. Five private schools agreed to take 1,500 voucher students.
The legislature commissioned John Witte, a professor at the University of Wisconsin, Madison, to study the students who received vouchers and compare their achievement with similar students in public schools. Witte found high levels of satisfaction among families who received vouchers, but no significant differences in math and reading performance between pupils in private schools and socioeconomically similar students who remained in public schools.
The initial voucher amount was set at roughly one-half of the annual per-pupil spending in Milwaukee's public schools. This changed quickly, with private schools demanding and getting a higher voucher payment, until it was at least as high as (or even higher than) public costs. Of the five private schools in the initial experiment, one subsequently closed. And of the initial voucher takers, more than a third left the private schools by the end of the fourth year.
In 1996 Peterson obtained the Milwaukee data and published his own study, using a "quasi-experimental" design that compared achievement of students who got vouchers in the lottery with those who did not. Results showed the pupils attending private schools making gains in math but not reading. At the same time, Peterson claimed that Witte had misspecified his model by comparing private-school pupils with those who remained in public schools but had not applied for vouchers.
Witte countered that students who applied but did not get vouchers included students rejected by the private schools; they displayed falling test scores, unlike most public-school students of similar background.
A third party, Princeton economist Cecilia Rouse, then took the same data, reworked them, and found that students in private schools made small gains in math but none in reading. A second Rouse paper found that in Milwaukee gains for low-income public-school students in smaller classes were higher than gains of voucher students in private schools.
In 1997 the Wisconsin legislature expanded the voucher program to 15,000 low-income students and included religious schools. The Wisconsin Supreme Court upheld this legislation. By the school year 2001-02, about 10,000 children will use vouchers at more than 100 mostly religious private schools. No one knows whether voucher students are performing better because, unlike public students, they are not required to take state-mandated tests.
The Cleveland Voucher Program
Cleveland's program began in 1995 with a $2,500 voucher. Private schools have since unsuccessfully tried to increase it. Almost all voucher students attend religious schools. On December 11, the Sixth Circuit Court of Appeals upheld a lower court ruling that these vouchers gave unconstitutional aid to religious schools. The U.S. Supreme Court will decide the program's future.
The Peterson group evaluated Cleveland vouchers in 1997. The study found significant test score gains. In 1997 University of Indiana researchers led by Kim Metcalf undertook a separate study and found no significant gains. When the Peterson group re-analyzed Metcalf's data, it found only barely significant gains for private-school students.
The Peterson group's new round of research concerns efforts by well-financed public-school opponents to fund "scholarships" (vouchers) for low-income children to attend private schools. The programs establish lotteries for parents who apply, give applicants a baseline test, award scholarships to applicants at random, and later test children who did and did not receive them. Some families who get vouchers do not actually send their children to private school, because they either cannot come up with the extra tuition or cannot find convenient private schools to accept their children.
Results for Dayton, New York, and Washington show no significant test score gains for Hispanic and white voucher recipients. Gains for African Americans are found to be statistically significant in New York and Washington, D.C., and marginally significant in reading (but not math) in Dayton. Reported gains are largest in Washington.
If the same methodology is used, the Charlotte gains are found to be about 6 percentile points in reading and about 6 percentile points in math. These are not broken down by ethnic group, but 80 percent of the sample is African American.
Closer Scrutiny
Comparing students who are already in public or in private schools has a major disadvantage: private-school students may come from more motivated families and may have survived selection processes. Solving this requires an "experiment": randomly assigning students to private and to public schools. But a truly blind trial, as in medical experiments in which control subjects are given a placebo and do not know whether they are receiving the treatment, is not possible, since families know whether they get a voucher. This makes education experiments subject to the so-called Hawthorne effect, whereby the participants' knowledge that they are involved in a program designed to produce a positive impact can cause them to try harder. The motivation of families whose voucher applications were rejected (that is, the control group) can also be affected.
The voucher experiments in various cities call for families to apply for a voucher, give baseline tests to all applicants, and then randomly select some to get vouchers to attend private schools. But the students in these experiments are not necessarily representative of low-income urban students. Families applying for vouchers whose children attend public schools are more motivated to switch their children and more dissatisfied with public schools than are average low-income parents, most of whom do not apply. Not receiving a voucher for parents already dissatisfied with their child's schooling could have an adverse "disappointment" effect on the child's performance.
The differential gains recorded in these experiments may therefore be partly attributable to lower gains by discouraged voucher rejectees rather than to greater gains by recipients. For a better comparison, voucher experiments would also need to draw a random sample of pupils from urban public schools whose low-income parents do not apply for vouchers and give the students the initial and follow-up tests. These pupils would come from families who are probably more satisfied with their current situation.
Another problem is self-selection of who gets the follow-up evaluation. Voucher researchers measure academic gains by convincing families to bring children in on a weekend to take follow-up math and reading tests. As in medical trials, high participation rates may require inducements. For those families who received vouchers, the New York inducement was that children would have to take the test to continue getting a voucher. Researchers used only moral suasion in other cities. For those who received but did not use the voucher and for those in the control group (who did not get a voucher), the inducement was typically $20 plus eligibility for a voucher in the future. Participation rates varied: The highest rates occurred in New York (about 66 percent in the second-year follow-up), while D.C. and Dayton both experienced about 50 percent. The participation rate in Charlotte was particularly low--40 percent. All these are considerably lower than in medical trials.
The Peterson group deals with participation problems by estimating the probability that a student with a certain initial test score and set of family characteristics and attitudes would participate in each follow-up test; then the actual actual scores are weighted according to this probability. Thus, students who came back to take follow-up tests but had characteristics that made them unlikely follow-up participants were counted more heavily, so test scores would be more "representative" of the original group of students.
The researchers could not do much more than this to correct for no-shows. But the procedure is hardly free of potential bias. It assumes that follow-up test scores for the many who didn't take the tests would be the same as scores for those who did show up and had similar initial scores and parent characteristics. But we really don't know how follow-up scores of no-shows might be related to their not showing up. The large nonparticipation rates could easily have affected the relative gains of voucher recipients and nonrecipients.
Yet another problem is bias in who accepts the voucher offer. The vouchers, which range in value from $1,200 to $1,700, depending on the city, are not large enough to cover tuition at most of the available private schools. Many families that received vouchers were unable to use them. In New York, 62 percent of families whose children started out in public school used the scholarships for two years; in Dayton and Washington, 53 percent used the voucher in the first year, with an unreported drop-off in the second year.
Voucher takers in each city, as would be expected, have higher income than nontakers. Critics have argued that this biases results. But the researchers have made a valid attempt to deal with the problem by comparing the controls with all students who were offered the voucher, not only those who actually used it.
In New York, the only students who made significant improvement were African Americans who switched to private schools when they were entering the fifth grade and whose gains were large enough to produce a significant average gain for the entire New York sample of African Americans.
Results for African-American students in Dayton also have a strange inconsistency. Certain cohorts--those who entered second, fourth, and sixth grades in the first year of the experiment--had large gains and those in the other grades did not. The D.C. results are more consistent.
With gains so variable by cohort, it is fair to ask, as did David Myers, whether one can claim that students in private schools do better than those in public schools. Shouldn't we, instead, wonder what conditions produced such large gains for some cohorts but not for others?
A final problem is erratic results. Big differences between first- and second-year gains in Washington, D.C., may relate to which students failed to show up for testing in the second year. Students might have failed to participate in the second-year testing either because of negative first-year experiences in private schools or because of disappointment with the first-year testing result. For such students, the probability is higher that they would do badly again than that they would do well. If they leave the sample, that alone could drive up the second-year result.
What Have We Learned?
Voucher evaluators have made a concerted effort to eliminate "selection bias" by offering vouchers to families with low incomes by lottery. In comparing pupils who receive vouchers to those who do not, bias from differential motivation and socioeconomic background is allegedly eliminated.
But this strategy does not speak to other issues. The sample of low-income, urban parents seeking vouchers does not represent the average low-income urban parent with children in public school. Parents who file for vouchers are, for one thing, more dissatisfied than other parents. To argue that they are more satisfied with their child's new, private school than parents who never applied for a voucher does not measure whether private schools are better than public, even if the comparison only concerns parental satisfaction.
The new studies also suffer from uncorrected potential bias, including "disappointment effects" of families that do not receive vouchers, low participation rates in follow-up tests, concentration of gains in small, particular cohorts that the researchers do not attempt to explain, and possible nonrandom declines in participation between the first and second years of testing.
The Peterson group's model has yet another problem. Low-income urban pupils who attend private schools may do better because private schools are able to select their students, so the influence of peers on student achievement is more positive than in a public school. The ability to select students is not a feature of private education that voucher advocates care to stress. And peer effects can run out quickly as private education expands in inner cities.
The Peterson group could deal with this problem if they tested a random sample of students already in the private schools attended by voucher recipients, identified them by school, and estimated the peer effect (the average test score of nonvoucher students in each school) on the scores of voucher recipients attending the school. It may not be easy to convince the private schools to allow such testing, simply because they would then be subjecting themselves to evaluation. But without such information, it is difficult to understand the source of private-school advantage--if such advantage even exists. ¤
