Tag Archives: Eric A. Hanushek

What , if anything, do education tests mean?

27 Nov

Moi received a review copy from Princeton University Press of Howard Wainer’s Uneducated Guesses. The publication date was September 14, 2011. In the preface Wainer states the goal of the book, “It deals with education in general and the use of tests and test scores in support of educational goals in particular.” Wainer tries to avoid not only the policy, but the ethical analysis of the analysis of the improper use of tests and test results by tightly defining the objective of the book at page four. The policy implications of using tests and test results to not only decide the direction of education, but to decide what happens to the participants in education are huge. Moi wonders if Wainer was really trying to avoid the unavoidable?

For moi, the real meat of the book comes in chapter 4. Wainer says:

In chapter 3 we learned that the PSAT, the shorter and easier version of the SAT, can be used effectively as one part of the selection decision for scholarships. In this chapter we expand on this discussion to illustrate that the PSAT also provides evidence that can help us allocate scarce educational resources…. [Emphasis Added]

Wainer examines the connection by analyzing and comparing test results from three high school districts. Those schools are Garfield High School in L.A., the site of the movie “Stand and Deliver.” La Canada High School in an upscale L.A. Suburb and Detroit, a very poor inner city school district. The really scary policy implication of Wainer’s very thorough analysis is found at page 44, “Limited resources mean that choices must be made.” Table 4-4 illustrates that real life choices are being made by districts like Detroit. What is really scary is that these choices affect the lives of real human beings. Of course, Wainer is simply the messenger and can’t be faulted for his analysis. According to Wainer, it is very tricky to use test results in predicting school performance and his discussion at page 53 summarizes his conclusions.

Perhaps the most chilling part of Wainer’s book is chapter 8 which deals with how testing and test results can adversely impact the career of a teacher when so-called “experts” incorrectly analyze test data. It should be required reading for those who want to evaluate teacher performance based upon test results.

Overall, Uneducated Guesses is a good, solid, and surprisingly readable book about test design, test results, and the use of test results. The truly scary part of the book describes how the uninformed, unknowing, and possibly venal can use what they perceive to be the correct interpretation to make policy judgments which result in horrific societal consequences.

Wainer makes statistics as readable as possible, because really folks, it is still statistics.

Here is the full citation for the book:

Uneducated Guesses: Using Evidence to Uncover Misguided Education Policies

Howard Wainer

Cloth: $24.95 ISBN: 9780691149288


Wainer’s book will come in handy when reading Eric A. Hanushek’s analysis of a National Research Council report.

Joy Resmovits writes about Eric A. Hanushek’s analysis of a National Research Council report in the Huffington Post article, Stanford Economist Rebuts Much-Cited Report That Debunks Test-Based Education:

When the National Research Council published the results of a decade-long study on the effects of standardized testing on student learning this summer, critics who have long opposed the use of exams as a teaching incentive rejoiced.

But Eric Hanushek, a Stanford University economist who is influential in education research, now says the “told you so” knee-jerk reaction was unwarranted: In an article released Monday by Harvard University’s journal Education Next, Hanushek argues that the report misrepresents its own findings, unjustifiably amplifying the perspective of those who don’t believe in testing. His article has even caused some authors of the NRC report to express concerns with its conclusions….

According to Hanushek’s analysis, the panel’s thorough examination of multiple studies is not evident in its conclusions.

“Instead of weighing the full evidence before it in the neutral manner expected of an NRC committee, the panel selectively uses available evidence and then twists it into bizarre, one might say biased, conclusions,” Hanushek wrote.

The anti-testing bias, he says, comes from the fact that “nobody in the schools wants people looking over their shoulders.”

Hanushek, an economist, claims that the .08 standard deviation increase in student learning is not as insignificant as the report makes it sound. According to his calculations, the benefits of such gains outweigh the costs: that amount of learning, he claims, translates to a value of $14 trillion. He notes that if testing is expanded at the expense of $100 per student, the rate of return on that investment is 9,189 percent. Hanushek criticized the report for not giving enough attention to the benefits NCLB provided disadvantaged students.

The report, Hanushek said, hid that evidence.

“They had that in their report, but it’s buried behind a line of discussion that’s led everybody who’s ever read it to conclude that test-based accountability is a bad idea,” he said. Hanushek reacted strongly, he said, because of the “complacency of many policymakers” who say education should be improved but that there are no effective options.



Grinding the Antitesting Ax: More bias than evidence behind NRC panel’s conclusions
Eric A. Hanushek,Education Next, WINTER 2012 / VOL. 12, NO. 2

Incentives and Test-Based Accountability in Education: A report from the National Research Council Checked by Eric A. Hanushek


One of the reasons why Hanushek’s critique is so important, aside from the implications that testing has under No Child Left Behind, is the push to use student test results in teacher evaluation. Valerie Strauss has an article in the Washington Post about a study which questions the use of student testing in the teacher evaluation process and the article includes links to the full report. In Study Blast Popular Teacher Evaluation Method Strauss reports:

Student standardized test scores are not reliable indicators of how effective any teacher is in the classroom, not even with the addition of new “value-added” methods, according to a study released today. It calls on policymakers and educators to stop using test scores as a central factor in holding teachers accountable.

Value-added modeling” is indeed all the rage in teacher evaluation: The Obama administration supports it, and the Los Angles Times used it to grade more than 6,000 California teachers in a controversial project. States are changing laws in order to make standardized tests an important part of teacher evaluation.

Unfortunately, this rush is being done without evidence that it works well. The study, by the Economic Policy Institute, a nonpartisan nonprofit think tank based in Washington, concludes that heavy reliance on VAM methods should not dominate high-stakes decisions about teacher evaluation and pay.

Here is the report link

Sarah Garland of the Hechinger Report has written the article, Should value-added teacher ratings be adjusted for poverty?

In Washington, D.C., one of the first places in the country to use value-added teacher ratings to fire teachers, teacher-union president Nathan Saunders likes to point to the following statistic as proof that the ratings are flawed: Ward 8, one of the poorest areas of the city, has only 5 percent of the teachers defined as effective under the new evaluation system known as IMPACT, but more than a quarter of the ineffective ones. Ward 3, encompassing some of the city’s more affluent neighborhoods, has nearly a quarter of the best teachers, but only 8 percent of the worst.

The discrepancy highlights an ongoing debate about the value-added test scores that an increasing number of states—soon to include Florida—are using to evaluate teachers. Are the best, most experienced D.C. teachers concentrated in the wealthiest schools, while the worst are concentrated in the poorest schools? Or does the statistical model ignore the possibility that it’s more difficult to teach a roomfull of impoverished children?

Saunders thinks it’s harder for teachers in high-poverty schools. “The fact that kids show up to school hungry and distracted and they have no eyeglasses and can’t see the board, it doesn’t even acknowledge that,” he said.


The question is what do test results mean and more importantly, how are test scores to be used? Wainer’s book attempts to analyze these questions.


Should value-added teacher ratings be adjusted for poverty?

Sarah Garland, The Hechinger Report, November 22, 2011


Every population of kids is different and they arrive at school at various points on the ready to learn continuum. Schools and teachers must be accountable, but there should be various measures of judging teacher effectiveness for a particular population of children. Perhaps, more time and effort should be spent in developing a strong principal corps and giving principals the training and assistance in evaluation and mentoring techniques. There should be evaluation measures which look at where children are on the learning continuum and design a program to address that child’s needs.

Dr. Wilda says this about that ©