Manhattan Institute study: Evidence that ‘value-added modeling’ may be effective

8 Sep

In New Jersey may eliminate teacher tenure, moi discussed teacher tenure:

A good basic description of teacher tenure as found at teacher tenure. James gives the following definition:

WHAT IS TENURE?

Tenure is a form of job security for teachers who have successfully completed a probationary period. Its primary purpose is to protect competent teachers from arbitrary nonrenewal of contract for reasons unrelated to the educational process — personal beliefs, personality conflicts with administrators or school board members, and the like.

WHAT PROTECTION DOES TENURE OFFER THE PUBLIC SCHOOL TEACHER?

The type and amount of protection vary from state to state and — depending on agreements with teachers’ unions — may even vary from school district to school district. In general, a tenured teacher is entitled to due process when he or she is threatened with dismissal or nonrenewal of contract for cause: that is, for failure to maintain some clearly defined standard that serves an educational purpose.  

Time has a good summary of the history of teacher tenure at A Brief History of Tenure One of the best concise defenses of K-12 teacher tenure is from Cleolaf’s blog at Why K12 Teachers Need Tenure Cleolaf points toward insufficient teacher assessment and evaluation as a prime cause of problems with teacher tenure. Research confirms that good principals are key to high performing schools. Good principals are also the key in Cleolaf’s view to making a tenure system work.

Another view of teacher tenure is found at Teacher Tenure: A Life Sentence for Kids This paper begins with the following case:

In 1986, after school administrators in the El Cajon School District in California spent years documenting the more than 400 reasons for why high school English teacher Juliet Ellory was an unfit teacher, the district finally succeeded in firing her. It cost the district more than $300,000 and eight years of preparing and litigating the case.

According to the overwhelming evidence against her, Ms. Ellory “hardly ever lectured, gave baffling assignments, belittled students and ignored repeated efforts by the high school principal to get her to improve.”1 Ellory’s tenure status had protected her from automatic dismissal. Though stories such as this one do not depict the average K-12 teacher, they are sufficiently widespread to provoke criticism and concern about the state of our public schools, as well as skepticism regarding the actual benefits of teacher tenure.

A key component of reforming teacher tenure is an improved evaluation system for teachers, which focuses on improving traits that produce student achievement.

Teacher Evaluation

The Center has produced a report, which focuses on teacher evaluation.Teacher Evaluation  Proper evaluation seems to be key to both addressing many problems teacher tenure was developed to protect from faulty evaluation of a teacher and to improve the quality of those in the teaching profession. Evaluation is just one component, however. New teachers need a proper induction into the profession and mentors to help them hone their skills and methods of teaching. If problems emerge, teachers need proper training and coaching to progress. http://drwilda.com/2012/01/16/new-jersey-may-eliminate-teacher-tenure/  Marcus A.Winters, a senior fellow at the Manhattan Institute has written the Civic Report, Transforming Tenure: Using Value-Added Modeling to Identify Ineffective Teachers.

Here is the executive summary for Transforming Tenure: Using Value-Added Modeling to Identify Ineffective Teachers:

Executive Summary

Public school teachers in the United States are famously difficult to dismiss. The reason is simple: after three years on the job, most receive tenure—after a brief and subjective evaluation process (typically, a classroom visit or two by an administrator or another teacher) in which few receive negative ratings. Once tenured, teachers are armored against efforts to remove them, and most do not face any serious reevaluation to ensure that their skills stay up to standard. With this traditional approach, tenured teachers sometimes lose their positions for insubordination, criminal conduct, gross neglect, or other reasons—but almost never for simply being bad at the job.

This state of affairs protects teachers (both good and bad) quite well but is clearly harmful to students. The effects of a poor teacher, research has shown, haunt pupils for years afterward. Being assigned to such a teacher reduces the amount that a student learns in school and is associated with lower earnings in adulthood (in part because having an inadequate teacher makes a child more likely to have an early pregnancy and less likely to go to college). An education system that protects bad teachers does a grave disservice to the children in its care.

In recent years, some school districts have experimented with changes in tenure rules. They seek the power to remove ineffective teachers and, in some jurisdictions, to reevaluate teachers throughout their careers.

A keystone of this reform movement is the replacement of subjective evaluation with quantifiable measures of each teacher’s effectiveness. The quantitative method is known as value-added modeling (VAM), a statistical analysis of student scores that seeks to identify how much an individual teacher contributes to a pupil’s progress over the years. The use of VAM in teacher evaluations is growing, but the method remains extremely controversial. Critics often claim that it does not and cannot measure actual teacher quality.

This paper addresses that claim. Part I analyzes data from Florida public schools to show that a VAM score in a teacher’s third year is a good predictor of that teacher’s success in his or her fifth year. Having established that VAM is a useful predictive tool, Part II of the paper addresses the most effective ways that VAM can be used in tenure reform.

VAM is not a perfect measure of teacher quality because, like any statistical test, it is subject to random measurement errors. So it should not be regarded as the “magic bullet” solution to the problem of evaluating teacher performance. However, the method is reliable enough to be part of a sensible policy of tenure reform—one that replaces “automatic” tenure with rigorous evaluation of new candidates and periodic reexamination of those who have already received tenure.

About the Author

Marcus A. Winters is a senior fellow at the Manhattan Institute and an assistant professor at the University of Colorado Colorado Springs. He conducts research and writes extensively on education policy, including topics such as school choice, high school graduation rates, accountability, and special education. Winters has performed several studies on a variety of education policy issues including high-stakes testing, performance-pay for teachers, and the effects of vouchers on the public school system. His research has been published in the journals Educational Evaluation and Policy Analysis, Education Finance and Policy, Economics of Education Review, Teachers College Record, and Education Next. His op-ed articles have appeared in numerous newspapers and magazines, including The Wall Street Journal, The Washington Post, USA Today, the New York Post, the New York Daily News, the Weekly Standard, and National Affairs. He is often quoted in the media on education issues. Winters received a B.A. in political science from Ohio University in 2002, and a Ph.D. in economics from the University of Arkansas in 2008.

Citation:

Civic Report

No. 70 August 2012

Transforming Tenure: Using Value-Added Modeling to Identify Ineffective Teachers

Marcus Winters, Senior Fellow, Manhattan Institute for Policy Research

http://www.manhattan-institute.org/html/cr_70.htm

TAP defines ‘value-added evaluation.”

According to TAP ‘value-added evaluation’ is:

Understanding Value-Added Analysis of Student Achievement

What is Value-Added Analysis?
Value-added analysis is a statistical technique that uses student achievement data over time to measure the learning gains students make. This methodology offers a way to estimate the impact schools and teachers have on student learning isolated from other contributing factors such as family characteristics and socioeconomic background. In other words, value-added analysis provides a way to measure the effect a school or teacher has on student academic performance over the course of a school year or another period of time.

Academic Attainment v. Academic Growth
Academic attainment is the level of achievement a student reaches at a point in time (e.g., on the state standardized test given at the end of any given school year). Usually referred to by a specific numerical score or standard of achievement (e.g., basic, proficient, advanced, etc.), academic attainment levels are what are typically used to rate school and/or teacher performance.

In contrast academic growth is the amount of academic gain or progress a student makes over a period of time (e.g., on the state test given over several grades). Value-added analysis is a methodology to measure academic growth and attribute it to the impact the school or teacher has had on student learning.

Benefits of Using Value-Added Analysis
Value-added analysis provides a more useful indicator of school and teacher performance than looking at student attainment levels, which is commonly used in public education today, for several reasons.

First, value-added analysis provides a more accurate way to measure student academic progress. Value-added analysis tracks the same student over time and compares his/her test scores over several years. In contrast, systems like Adequate Yearly Progress (AYP) look at the fourth grade math scores for one year, for example, and compare them to the fourth grade math scores from the previous year. This yields an inaccurate comparison because the groups of students may be significantly different from year to year (Braun, 2005).

Second, value-added analysis provides a measure for how much of an impact the school and individual teachers have on student achievement. Looking at the attainment level of a school or classroom on a state test provides little information about the impact the school or teacher has had on the final score as compared with other influences on student achievement like family background and socioeconomic status. By judging only one score, it is difficult to identify how much of that score was influenced by factors outside of the school as compared to other factors that can be controlled within the school (e.g., the contributions of the teacher and school).

Third, when student achievement is tied to accountability systems, value-added analysis provides a fairer method to measure school and teacher impact on student achievement because it takes into account where a student started the school year academically and how much that student grew. Judging a school’s or teacher’s performance by looking at student academic attainment levels is unfair because some students may enter a teacher’s classroom already at high levels of achievement — or conversely, several grade levels behind their peers. Without considering the academic growth teachers and schools are able to make with their students, some teachers and schools may inaccurately be attributed with making a significant impact while others may be unfairly penalized.

How TAP Schools Use Value-Added Data
School districts that are implementing TAP district-wide often use value-added data to identify schools, grades and content areas that have or have not increased student achievement. These data help district officials plan how to target professional development so that it is most effective for teachers and schools. Districts can also use these data to identify effective teachers and administrators who can be utilized as mentors for others at schools that have not made significant academic gains.

At the school level, TAP leadership teams utilize value-added data to address the instructional needs of teachers both at the individual and group levels. By analyzing teacher value-added scores and comparing them to a teacher’s evaluation scores (based on observations of classroom instruction), leadership teams are able to identify “best practices” that are having a positive impact on student achievement. Leadership teams can then share these best practices with other teachers during weekly cluster group meetings (professional learning communities) to promote effective instruction. Leadership team members also use comparative data to conference with teachers on a one-on-one basis and inform the development of teachers’ individual professional growth plans to reach instructional goals.

At the classroom level, teachers analyze the value-added data from their own students by subgroups (such as high, medium and low performing students) to identify trends in their own instruction. The data may reveal that their instruction is targeted more to a specific subgroup and, as a result, teachers make adjustments in their instruction. This data analysis process allows teachers to meet the needs of all students more effectively and support the individual academic growth of their students regardless of their ability level. http://www.tapsystem.org/policyresearch/policyresearch.taf?page=valueadded

Brookings Institute wrote about ‘value-added models’ in a 2010 report.

Steven Glazerman, Dan Goldhaber, Susanna Loeb , Stephen Raudenbush, and Douglas Staiger wrote in the article, Evaluating Teachers: The Important Role of Value-Added:

The evaluation of teachers based on the contribution they make to the learning of their students, value-added, is an increasingly popular but controversial education reform policy. We highlight and try to clarify four areas of confusion about value-added. The first is between value-added information and the uses to which it can be put. One can, for example, be in favor of an evaluation system that includes value-added information without endorsing the release to the public of value-added data on individual teachers. The second is between the consequences for teachers vs. those for students of classifying and misclassifying teachers as effective or ineffective — the interests of students are not always perfectly congruent with those of teachers. The third is between the reliability of value-added measures of teacher performance and the standards for evaluations in other fields — value-added scores for individual teachers turn out to be about as reliable as performance assessments used elsewhere for high stakes decisions. The fourth is between the reliability of teacher evaluation systems that include value-added vs. those that do not — ignoring value-added typically lowers the reliability of personnel decisions about teachers. We conclude that value-added data has an important role to play in teacher evaluation systems, but that there is much to be learned about how best to use value-added information in human resource decisions.                                                                                  http://www.brookings.edu/research/reports/2010/11/17-evaluating-teachers

The key point is the use of ‘value-added modeling’ may or may not be useful.

Resources:

Leading mathematician debunks ‘value-added’ http://www.washingtonpost.com/blogs/answer-sheet/post/leading-mathematician-debunks-value-added/2011/05/08/AFb999UG_blog.html

Evaluating Value-Added Models for Teacher Accountability http://www.rand.org/pubs/monographs/2004/RAND_MG158.pdf

The National Institute for Excellence in Teaching recommends the following:

Comparisons Among Various Educational Assessment Value-Added Models
William L. Sanders, Presented at “The Power of Two” National Value-Added Conference, October 16, 2006
This paper discusses the differences between value-added models as well as their advantages and disadvantages.

FORUM: “Accountability Gains: Are we measuring achievement gains accurately enough?”
Education Next (2002) No. 2
This forum includes four articles by Dale Ballou, Anita A. Summer, Jay P. Greene and Donald R. McAdams discussing the pros and cons of value-added measurement of student achievement.

Research Brief: The Promise and Peril of Using Value-Added Modeling to Measure Teacher Effectiveness
RAND Corporation (2004)

This brief summarizes the findings of a longer research report, “Evaluating Value-Added Models for Teacher Accountability,” by Daniel F. McCaffrey, Daniel M. Koretz, J.R. Lockwood and Laura S. Hamilton (2004), which compares several value-added models and discusses the strengths and weaknesses of using such methodology for both diagnostic and accountability purposes.

Roundtable Discussion on Value-Added Analysis of Student Achievement: A Summary of Findings
The Working Group on Teacher Quality (2007)

Summarized from a roundtable discussion held in October 2007 among policymakers, researchers and practitioners, this document presents major themes, findings and lessons learned in value-added analysis of student achievement. The purpose of the discussion was to create a broader understanding of how value-added analysis of student achievement can be used as an indicator of teacher effectiveness and the implications this has for policy and practice.

Using Student Progress to Evaluate Teachers: A Primer on Value-Added Models
Henry I. Braun (September 2005)
This policy perspective provides reader-friendly information on the more technical issues associated with value-added modeling.

Value-Added Modeling: The Challenge of Measuring Educational Outcomes
Barbara Elizabeth Stewart (2006)
This article provides a summary of the history, definition, strengths and weaknesses of value-added modeling.

Websites

Battelle for Kids
Battelle for Kids is a non-profit organization based in Ohio, whose activities include helping districts and schools use value-added data to improve their instruction.

Houston Independent School District’s Accelerating Student Progress, Increasing Results & Expectations (ASPIRE) Program
This website provides information on the value-added model used in Houston Independent School District’s ASPIRE program. The site includes a guide to value-added for parents and families.

Related:

Study: Teacher merit pay works in some situations http://drwilda.com/2012/07/27/study-teacher-merit-pay-works-in-some-situations/

Report: Measuring teacher effectiveness                          http://drwilda.com/2012/06/13/report-measuring-teacher-effectiveness/

Dr. Wilda says this about that ©

 

About these ads

One Response to “Manhattan Institute study: Evidence that ‘value-added modeling’ may be effective”

Trackbacks/Pingbacks

  1. National Education Policy Brief: Designing teacher evaluations « drwilda - September 25, 2012

    [...] Manhattan Institute study: Evidence that ‘value-added modeling’ may be effective                                                         http://drwilda.com/2012/09/08/manhattan-institute-study-evidence-that-value-added-modeling-may-be-ef… [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 1,021 other followers

%d bloggers like this: