Moi wrote in What, if anything, do education tests mean?
Moi received a review copy from Princeton University Press of Howard Wainer’s Uneducated Guesses. The publication date was September 14, 2011. In the preface Wainer states the goal of the book, “It deals with education in general and the use of tests and test scores in support of educational goals in particular.” Wainer tries to avoid not only the policy, but the ethical analysis of the analysis of the improper use of tests and test results by tightly defining the objective of the book at page four. The policy implications of using tests and test results to not only decide the direction of education, but to decide what happens to the participants in education are huge. Moi wonders if Wainer was really trying to avoid the unavoidable?
For moi, the real meat of the book comes in chapter 4. Wainer says:
In chapter 3 we learned that the PSAT, the shorter and easier version of the SAT, can be used effectively as one part of the selection decision for scholarships. In this chapter we expand on this discussion to illustrate that the PSAT also provides evidence that can help us allocate scarce educational resources…. [Emphasis Added]
Wainer examines the connection by analyzing and comparing test results from three high school districts. Those schools are Garfield High School in L.A., the site of the movie “Stand and Deliver.” La Canada High School in an upscale L.A. Suburb and Detroit, a very poor inner city school district. The really scary policy implication of Wainer’s very thorough analysis is found at page 44, “Limited resources mean that choices must be made.” Table 4-4 illustrates that real life choices are being made by districts like Detroit. What is really scary is that these choices affect the lives of real human beings. Of course, Wainer is simply the messenger and can’t be faulted for his analysis. According to Wainer, it is very tricky to use test results in predicting school performance and his discussion at page 53 summarizes his conclusions.
Perhaps the most chilling part of Wainer’s book is chapter 8 which deals with how testing and test results can adversely impact the career of a teacher when so-called “experts” incorrectly analyze test data. It should be required reading for those who want to evaluate teacher performance based upon test results.
Overall, Uneducated Guesses is a good, solid, and surprisingly readable book about test design, test results, and the use of test results. The truly scary part of the book describes how the uninformed, unknowing, and possibly venal can use what they perceive to be the correct interpretation to make policy judgments which result in horrific societal consequences.
Wainer makes statistics as readable as possible, because really folks, it is still statistics.
Here is the full citation for the book:
Uneducated Guesses: Using Evidence to Uncover Misguided Education Policies
Howard Wainer
Cloth: $24.95 ISBN: 9780691149288
200pp.
https://drwilda.com/2011/11/27/what-if-anything-do-education-tests-mean/
Many do not know about the National Assessment of Educational Progress (NAEP). Here is a description of the test:
NAEP Overview
Here are some FAQs:
The National Assessment of Educational Progress (NAEP) is a program with many components—from developing subject-area questions, to selecting schools to participate, to reporting the results. Given its complexity, NAEP receives a variety of questions from visitors to the website; these special pages have been developed to provide answers to some of the most common questions.
- If you have additional questions about school selection and NAEP sample design, see FAQs on sampling issues.
- Parents whose children are selected for NAEP will find answers to their questions in the Information for Parents section, also available in Spanish.
- There are also FAQs selected for educators, and FAQs for students.
- The website for The Nation’s Report Card has FAQs related to the very latest release, such as numbers of students participating, inclusion rates, changes in student demographics, and more.
- Several special studies are supported with FAQs:
If you can’t find the answer to your question on any of our FAQ pages, please click Contact NAEP on the left.
What is NAEP?
NAEP, or the National Assessment of Educational Progress, produces the Nation’s Report Card, to inform the public about the academic achievement of elementary and secondary students in the United States. Sponsored by the department of Education, NAEP assessments have been conducted periodically in reading, mathematics, science, writing, U.S. history, civics, geography, and other subjects, beginning in 1969. NAEP collects and reports academic achievement at the national level, and for certain assessments, at the state and district levels. The results are widely reported by the national and local media, and are an integral part of our nation’s evaluation of the condition and progress of education.
For more general information about NAEP, read the NAEP Overview.
For technical information about NAEP, consult the NAEP Technical Documentation.What is the difference between state NAEP and national NAEP?
The NAEP sample in each state is designed to be representative of the students in that state. At the state level, results are currently reported for public school students only and are broken down by several demographic groupings of students. When NAEP is conducted at the state level (i.e., in mathematics, reading, science, and writing), results are also reported for the nation. The national NAEP sample is then composed of all the state samples of public school students, as well as a national sample of nonpublic school students. If there are states that do not participate, a certain number of schools and students are selected to complete the national-level sample.
For assessments conducted at the national level only, samples are designed to be representative of the nation as a whole. Data are reported for public and nonpublic school students as well as for several major demographic groups of students.
Read technical information about the differences in the sample selection for state and national assessments in NAEP Assessment Sample Design.
What are the goals of the NAEP program?
NAEP has two major goals: to compare student achievement in states and other jurisdictions and to track changes in achievement of fourth-, eighth-, and twelfth-graders over time in mathematics, reading, writing, science, and other content domains. To meet these dual goals, NAEP selects nationally representative samples of students who participate in either the main NAEP assessments or the long-term trend NAEP assessments.
For technical aspects of reporting student achievement, see Analysis and Scaling for NAEP.
Is participation in NAEP voluntary?
Federal law specifies that NAEP is voluntary for every student, school, school district, and state. However, federal law also requires all states that receive Title I funds to participate in NAEP reading and mathematics assessments at fourth and eighth grades. Similarly, school districts that receive Title I funds and are selected for the NAEP sample are also required to participate in NAEP reading and mathematics assessments at fourth and eighth grades. All other NAEP assessments are voluntary. Learn more about NAEP and why participation is important.
Are the data confidential?
Federal law dictates complete privacy for all test takers and their families. Under the National Assessment of Educational Progress Authorization Act (Public Law 107-279 III, section 303), the Commissioner of the National Center for Education Statistics (NCES) is charged with ensuring that NAEP tests do not question test-takers about personal or family beliefs or make information about their personal identity publicly available.
After publishing NAEP reports, NCES makes data available to researchers but withholds students’ names and other identifying information. The names of all participating students are not allowed to leave the schools after NAEP assessments are administered. Because it might be possible to deduce from data the identities of some NAEP schools, researchers must promise, under penalty of fines and jail terms, to keep these identities confidential.
For technical details, read about Questionnaires and Tracking Forms and Non-Cognitive Items in Student Booklets.
Who are the students assessed by NAEP?
The national results are based on a representative sample of students in public schools, private schools, Bureau of Indian Education schools, and Department of Defense schools. Private schools include Catholic, Conservative Christian, Lutheran, and other private schools. The state results are based on public school students only. The main NAEP assessment is usually administered at grades 4 and 8 (at the state level) plus grade 12 at the national level. The long-term trend assessments report national results (in mathematics and reading only) for age samples 9, 13, and 17 in public and nonpublic schools.
For technical details, read about the NAEP Assessment Sample Design.
Who evaluates NAEP?
Because NAEP findings have an impact on the public’s understanding of student academic achievement, precautions are taken to ensure the reliability of these findings. In its current legislation, as in previous legislative mandates, Congress has called for an ongoing evaluation of the assessment as a whole. In response to these legislative mandates, the National Center for Education Statistics (NCES) has established various panels of technical experts to study NAEP, and panels are formed periodically by NCES or external organizations, such as the National Academy of Sciences, to conduct evaluations. The Buros Center for Testing, in collaboration with the University of Massachusetts/Center for Educational Assessment and the University of Georgia, more recently conducted an external evaluation of NAEP.
For technical aspects of reporting student achievement, see Analysis and Scaling for NAEP.
How do I know what publications are available from NAEP and how do I get them?
The NAEP Publications page is accessible via the Publications link at the top of every screen.
Printed copies of NAEP publications can be ordered by contacting:
http://edpubs.ed.gov
Phone: (877) 4-ED-PUBS (433-7827)
TDD/TTY: (877) 576-7734
Mail: Ed Pubs, U.S. Department of Education, P.O. Box 22207, Alexandria, VA 22304
Para español, llame al (877) 433-7827
It is important to understand what the NEAP is because there are attempts to use the test as a predictive tool.
Sarah D. Sparks reports in the Education Week article, Can NAEP Predict College Readiness?
College Indicators
For college, at least, there are signs NAEP performance may be linked to how well a student will do in initial coursework. Researchers from WestEd, a San Francisco-based research group working under contract to the governing board, found that the 12th grade reading and math tests cover content very similar to that of the SAT.
Moreover, a 2009 study of more than 15,000 12th graders who took both the national assessment and the SAT showed that performing at the proficient level on the math NAEP was associated
with an 80 percent chance of earning 500 points out of a possible 800 on the math portion of the SAT, and that the proficient level in reading was associated with a 50-50 chance of scoring 500 on the SAT verbal test.
The SAT has internally pegged a score of 500 to earning at least a B-minus in freshman-level college courses.
NAEP 12th grade content less closely mirrored that used in the ACT, the nation’s other major college-entrance exam; in particular, some arithmetic and applied-math items on the ACT would be covered in more depth on the 8th grade than the 12th grade NAEP in math. NAEP has not been able to compare its performance levels to those in the ACT, though Ms. Orr said the board plans to do so during the 2013 studies, which will also include more state-specific analyses.
Individual states’ data are likely to be critical, North Carolina’s Mr. Fabrizio said, because course requirements vary widely from state to state and even between college systems within the same state.
Hazy Work Picture
The connection between NAEP and preparation for careers that don’t require a four-year college degree is much more tenuous.
The governing board found less overlap between NAEP 12th grade content and that covered on the career-related WorkKeys test, also by ACT Inc. Last spring, panels of professional trainers in five careers—computer-support specialists, automotive master technicians, licensed practical nurses, pharmacy staff, and heating, ventilation, and air conditioning technicians—could not agree
They did agree, however, that most of the content on the test wouldn’t say much about students’ potential in those fields.
For example, “there are hardly any test items in the pool at 12th grade that are applied, based on some use of mathematics rather than theoretical stuff,” said Jeremy Kilpatrick, a co-author of the study
The assessment governing board will try to bring more clarity around job skills next year, with an analysis that compares the skills and knowledge covered in job-training programs in the five career areas with the math and reading content in the 12th grade NAEP tests.
Still, Ms. Orr was less hopeful about whether NAEP will be useful for gauging career readiness. http://www.edweek.org/ew/articles/2012/09/12/03nagb.h32.html?tkn=WYYFs6Fvb3qWQ7tlD%2B4kB4B80di2bmJy6Rje&intc=es
Moi wrote about testing in More are questioning the value of one-size-fits-all testing:
The goal of education is of course, the educate students. Purdue University has a concise synopsis of Bloom’s Taxonomy which one attempt at describing education objectives:
Bloom’s (1956) Taxonomy of Educational Objectives is the most renowned description of the levels of cognitive performance. The levels of the Taxonomy and examples of activities at each level are given in Table 3.3. The levels of this taxonomy are considered to be hierarchical. That is, learners must master lower level objectives first before they can build on them to reach higher level objectives. http://education.calumet.purdue.edu/vockell/edPsybook/Edpsy3/edpsy3_bloom.htm
See, Bloom’s Taxonomy http://en.wikipedia.org/wiki/Bloom%27s_Taxonomy More and more people are asking if testing really advances the goals of education or directs testing’s objectives, which may or may not be the same as the goals of education. https://drwilda.com/2012/02/20/more-are-questioning-the-value-of-one-size-fits-all-testing/
Related:
What , if anything, do education tests mean? https://drwilda.wordpress.com/2011/11/27/what-if-anything-do-education-tests-mean/
Complete College America report: The failure of remediation https://drwilda.wordpress.com/2012/06/21/complete-college-america-report-the-failure-of-remediation/
What the ACT college readiness assessment means https://drwilda.com/2012/08/25/what-the-act-college-readiness-assessment-means/
Dr. Wilda says this about that ©
drwilda 
3 Responses to “The importance of the National Assessment of Educational Progress”