Tag Archives: Education Research

University of York study: Trials testing new educational methods in schools ‘often fail to produce useful evidence’

21 Mar

More and more, individuals with gravitas are opining about the American education system for reasons ranging from national security to economic competitiveness. Regarding  the Condoleezza Rice and Joel Klein report about American Education, moi wrote:

The Council on Foreign Relations issued the report, U.S. Education Reform and National Security. The chairs for the report are Joel I. Klein, News Corporation and Condoleezza Rice, Stanford University. Moi opined about the state of education in U.S. education failure: Running out of excuses https://drwilda.wordpress.com/2011/12/13/u-s-education-failure-running-out-of-excuses/ Education tends to be populated by idealists and dreamers who are true believers and who think of what is possible. Otherwise, why would one look at children in second grade and think one of those children could win the Nobel Prize or be president? Maybe, that is why education as a discipline is so prone to fads and the constant quest for the “Holy Grail” or the next, next magic bullet. There is no one answer, there is what works for a particular population of kids. https://drwilda.wordpress.com/2012/03/19/condoleezza-rice-and-joel-klein-report-about-american-education/

Citation:

U.S. Education Reform and National Security
Publisher Council on Foreign Relations Press
Release Date March 2012
Price $15.00
108 pages
ISBN 978-0-87609-520-1
Task Force Report No. 68

Related:

Joy Resmovits of Huffington Post,Schools Report: Failing To Prepare Students Hurts National Security, Prosperity http://www.huffingtonpost.com/2012/03/19/schools-report-condoleezza-rice-joel-klein_n_1365144.html

Moi often says education is a partnership between the student, the teacher(s) and parent(s). All parties in the partnership must share the load. The student has to arrive at school ready to learn. The parent has to set boundaries, encourage, and provide support. Teachers must be knowledgeable in their subject area and proficient in transmitting that knowledge to students. All must participate and fulfill their role in the education process.

Science Daily reported in Trials testing new educational methods in schools ‘often fail to produce useful evidence’:

Educational trials aimed at boosting academic achievement in schools are often uninformative, new research suggests.
The new study, published in the journal Educational Researcher, found that 40% of large-scale randomised controlled trials (RCTs) in the UK and the US failed to produce any evidence as to whether an educational intervention helped to boost academic attainment or not.
The researchers evaluated 141 trials involving more than one million students, which tested schemes ranging from whether providing free school breakfasts raises grades in Maths and English, to whether playing chess improves numeracy skills.
The trials, which were carried out by the charitable organisation the Education Endowment Foundation (EEF) in the UK and the National Center for Education Evaluation and Regional Assistance (NCEE) in the US, are expensive — with costs often exceeding £500,000.
The authors of the study argue that more research is urgently needed to understand why RCTs in education are so often uninformative.
Lead author of the research, Dr Hugues Lortie-Forgues, from the Department of Education at the University of York, UK, said: “Just like in medicine, trials of educational interventions are an important way to allow policy makers and teachers to make informed decisions about how to improve education. However, many of these trials are currently not fulfilling their main aim of demonstrating which interventions are effective and which are not….”
In recent years there have been a growing number of RCTs conducted in education. For example, in the UK, the Education Endowment Foundation (EEF) has commissioned more than 191 trials since 2012.
The researchers cite possible reasons why current trials may be ineffective, including:
• The interventions being tested may not be suitable for trial in the first place.
• Interventions may not be being correctly implemented during trials — for example due to inadequate training of teachers in the methods being tested.
• The trials themselves may be poorly designed
The authors suggest a series of changes that could make the trials more informative, including higher-standards when considering which new initiatives are trialled…. https://www.sciencedaily.com/releases/2019/03/190315110913.htm

Citation:

Trials testing new educational methods in schools ‘often fail to produce useful evidence’
Educational trials aimed at boosting academic achievement in schools are often uninformative
Date: March 15, 2019
Source: University of York
Summary:
The new study found that 40% of large-scale randomized controlled trials (RCTs) in the UK and the US failed to produce any evidence as to whether an educational intervention helped to boost academic attainment or not.

Journal Reference:
Hugues Lortie-Forgues, Matthew Inglis. Rigorous Large-Scale Educational RCTs Are Often Uninformative: Should We Be Concerned? Educational Researcher, 2019; 0013189X1983285 DOI: 10.3102/0013189X19832850

Here is the press release from the University of York:

Trials testing new educational methods in schools often fail to produce useful evidence, say researchers
Posted on 15 March 2019
Educational trials aimed at boosting academic achievement in schools are often uninformative, new research suggests.
The new study, published in the journal Educational Researcher, found that 40% of large-scale randomised controlled trials (RCTs) in the UK and the US failed to produce any evidence as to whether an educational intervention helped to boost academic attainment or not.
The researchers evaluated 141 trials involving more than one million students, which tested schemes ranging from whether providing free school breakfasts raises grades in Maths and English, to whether playing chess improves numeracy skills.
The trials, which were carried out by the charitable organisation the Education Endowment Foundation (EEF) in the UK and the National Center for Education Evaluation and Regional Assistance (NCEE) in the US, are expensive – with costs often exceeding £500,000.
Informed decisions
The authors of the study argue that more research is urgently needed to understand why RCTs in education are so often uninformative.
Lead author of the research, Dr Hugues Lortie-Forgues, from the Department of Education at the University of York, UK, said: “Just like in medicine, trials of educational interventions are an important way to allow policy makers and teachers to make informed decisions about how to improve education. However, many of these trials are currently not fulfilling their main aim of demonstrating which interventions are effective and which are not.”
“Further research to investigate the reasons for this should be a priority. These organisations are trying to achieve something positive and reform is urgently needed to help them to do so.”
Higher standards
In recent years there have been a growing number of RCTs conducted in education. For example, in the UK, the Education Endowment Foundation (EEF) has commissioned more than 191 trials since 2012.
The researchers cite possible reasons why current trials may be ineffective, including:
The interventions being tested may not be suitable for trial in the first place.
Interventions may not be being correctly implemented during trials – for example due to inadequate training of teachers in the methods being tested.
The trials themselves may be poorly designed
The authors suggest a series of changes that could make the trials more informative, including higher-standards when considering which new initiatives are trialled. https://www.york.ac.uk/news-and-events/news/2019/research/trials-testing-new-educational-methods/

In Paul E. Peterson will piss you off, you might want to listen, moi said:
Moi has been saying for decades that the optimum situation for raising children is a two-parent family for a variety of reasons. This two-parent family is an economic unit with the prospect of two incomes and a division of labor for the chores necessary to maintain the family structure. Parents also need a degree of maturity to raise children, after all, you and your child should not be raising each other. Moi said this in Hard truths: The failure of the family:
This is a problem which never should have been swept under the carpet and if the chattering classes, politicians, and elite can’t see the magnitude of this problem, they are not just brain dead, they are flat-liners. There must be a new women’s movement, this time it doesn’t involve the “me first” philosophy of the social “progressives” or the elite who in order to validate their own particular life choices espouse philosophies that are dangerous or even poisonous to those who have fewer economic resources. This movement must urge women of color to be responsible for their reproductive choices. They cannot have children without having the resources both financial and having a committed partner. For all the talk of genocide involving the response and aftermath of Katrina, the real genocide is self-inflicted. https://drwilda.wordpress.com/2011/11/06/hard-truths-the-failure-of-the-family/ It is interesting that the ruling elites do not want to touch the issue of unwed births with a ten thousand foot pole. After all, that would violate some one’s right to _____. Let moi fill in the blank, the right to be stupid, probably live in poverty, and not be able to give your child the advantages that a more prepared parent can give a child because to tell you to your face that you are an idiot for not using birth control is not P.C.
https://drwilda.wordpress.com/2012/03/08/paul-e-peterson-will-piss-you-off-you-might-want-to-listen/

Where information leads to Hope. ©

Dr. Wilda says this about that ©

Blogs by Dr. Wilda:

COMMENTS FROM AN OLD FART©
http://drwildaoldfart.wordpress.com/

Dr. Wilda Reviews ©
http://drwildareviews.wordpress.com/

Dr. Wilda ©                                                                                                                                                                           https://drwilda.com/

The importance of appropriate grading

11 Jan

Education is a partnership between the student, parent(s) or guardian(s), teacher(s), and school. Standards are a benchmark, but students and families need to prepare for and support student education success. Teachers must be prepared and supported in meeting the standards adopted by the schools. Schools must be learning environments which support and mentor teachers and keep children safe. Otherwise, standards are simply a nice goal.

A report, Standing On the Shoulders of Giants by Mark S. Tucker examines high performing education systems. Among the recommendations are:

What follows is a new agenda for recasting the structure of the preceding section, derived from the experience of the countries that have consistently outperformed the United States. It was constructed simply by taking the subsection headings and reframing the language of the preceding sections in the form of an action agenda. To be clear, this is not an agenda for the United States; it is an agenda for individual states:

Benchmark the Education Systems of the Top-Performing Countries

  • Make sure you know what the leaders are trying to achieve, the extent to which they achieve it and how they do on common measures

  • Compare your state to the best performers, with particular attention to countries that share your goals

  • Conduct careful research on the policies and practices of the best performing nations to understand how they get the results they get

  • Benchmark often, because the best never stand still

Design for Quality

  • Get your goals clear, and get public and professional consensus on them

  • Create world-class instructional systems and gateways

  • Define a limited number of gateways — not more than the end of basic education, end of lower secondary and end of upper secondary (matched up to college entrance and work-ready requirements)

  • Create standards for each gateway, making sure they are properly nested and are world class

  • Create logically ordered curriculum frameworks (topics for each year or each subject) for the basic education sequence

  • Create curriculum (broad guidelines, not lesson plans) for each school level leading up to the gateway exams (the level of detail at which this is done should be inversely related to the quality of your teachers)

  • Create exams for each gateway, based on standards and curricula

  • Train teachers to teach those curricula well to students from many different backgrounds….

Appropriate use of grading and testing are methods to determine whether the education system is meeting stated goals.

University of Michigan Center For Research On Learning and Teaching suggests in Best Practices for Designing and Grading Exams, adapted from M.E. Piontek (2008) :

The most obvious function of assessment methods (such as exams, quizzes, papers, and presentations) is to enable instructors to make judgments about the quality of student learning  (i.e., assign grades). However, the method of assessment also can have a direct impact on the quality of student learning. Students assume that the focus of exams and assignments reflects the educational goals most valued by an instructor, and they direct their learning and studying accordingly  (McKeachie  & Svinicki, 2006).  General grading systems can have an impact as well.  For example, a strict bell curve (i.e., norm-reference grading) has the potential to dampen motivation and cooperation in a classroom, while a system that strictly rewards proficiency (i.e., criterion-referenced grading) could be perceived as contributing to grade inflation. Given the importance of assessment for both faculty and student interactions about learning, how can instructors develop exams that provide useful and relevant data about their students’ learning and also direct students to spend their time on the important aspects of a course or course unit? How do grading practices further influence this process?

Guidelines for Designing Valid and Reliable Exams

Ideally, effective exams have four characteristics. They are:

  • Valid, (providing useful information about the concepts they were designed to test),
  • Reliable (allowing consistent measurement and discriminating between different levels of performance),
  • Recognizable  (instruction has prepared students for the assessment), and
  • Realistic (concerning time and effort required to complete the assignment)  (Svinicki, 1999). 

Most importantly, exams and assignments should focus on the most important content and behaviors emphasized during the course (or particular section of the course). What are the primary ideas, issues, and skills you hope students learn during a particular course/unit/module? These are the learning outcomes you wish to measure. For example, if your learning outcome involves memorization, then you should assess for memorization or classification; if you hope students will develop problem-solving capacities, your exams should focus on assessing students’ application and analysis skills.  As a general rule, assessments that focus too heavily on details (e.g., isolated facts, figures, etc.) “will probably lead to better student retention of the footnotes at the cost of the main points” (Halpern & Hakel, 2003, p. 40). As noted in Table 1, each type of exam item may be better suited to measuring some learning outcomes than others, and each has its advantages and disadvantages in terms of ease of design, implementation, and scoring.

Table 1: Advantages and Disadvantages of Commonly Used Types of Achievement Test Items

Type of Item Advantages Disadvantages
True-False Many items can be administered in a relatively short time. Moderately easy to write; easily scored. Limited primarily to testing knowledge of information.  Easy to guess correctly on many items, even if material has not been mastered.
Multiple-Choice Can be used to assess broad range of content in a brief period. Skillfully written items can measure higher order cognitive skills. Can be scored quickly. Difficult and time consuming to write good items. Possible to assess higher order cognitive skills, but most items assess only knowledge.  Some correct answers can be guesses.
Matching Items can be written quickly. A broad range of content can be assessed. Scoring can be done efficiently. Higher order cognitive skills are difficult to assess.
Short Answer or Completion Many can be administered in a brief amount of time. Relatively efficient to score. Moderately easy to write. Difficult to identify defensible criteria for correct answers. Limited to questions that can be answered or completed in very few words.
Essay Can be used to measure higher order cognitive skills. Relatively easy to write questions. Difficult for respondent to get correct answer by guessing. Time consuming to administer and score. Difficult to identify reliable criteria for scoring. Only a limited range of content can be sampled during any one testing period.

Adapted from Table 10.1 of Worthen, et al., 1993, p. 261.

http://www.crlt.umich.edu/gsis/P8_0.php

Choosing the appropriate measurement is important for accurate evaluation.

Education Research reports in the article, Teachers so focused on fairness issues they overlook best practices in grading:

The 77 teachers who participated in the study completed questionnaires asking them to rate their awareness of 4 grading principles and to provide information about their own grading practices.  Among the results:

  • 29% reported considerable awareness of recommended grading principles
  • 40 % reported some degree of awareness of grading principles 
  • 17% of teachers said they had only very little awareness
  • 13% said they had no awareness of the grading principles.

When asked how much they used these principles in their grading practices, 

  • 23% of teachers agreed that they followed the principles
  • 43% somewhat agreed that they followed   the principles
  • 10% said the principles did not apply to them
  • 13% somewhat disagreed that they fol- lowed the principles
  • 3% disagreed that they followed the prin ciples and 10% felt they did not apply to them.

In a standards-based system,  it’s important for teachers to stay focused on assessing a student’s achievement against standards, the study says.  However, teachers persist in taking into account a hodgepodge of other factors such as student effort  or whether the student hands in work on time, according to the study on the grading beliefs and practices of 10th-grade math teachers in Ontario….

The 4 principles that formed the framework for the study were:

  1. Grades should be referenced to the curriculum objectives or learning expectations (criterion referenced)
  2. A grade should be an accurate representation of achievement and non-achievement factors should be reported separately 
  3. Results from multiple assessments should be combined carefully with weighting that reflects learning expectations 
  4. Information about grading should be clearly communicated so that grades are justified and their meaning is understood by students, parents, and other teachers. 

The study was part of a larger nationally funded study on teachers’ grading practices in the provinces of Saskatchewan and Ontario. One goal of the study was to determine how teachers calculated students’ final report card grades in 2 educational systems with differing assessment policies. http://www.ernweb.com/public/Grading-best-practices-fairness-achievement-standards-based.cfm

Appropriate grading practices is an important component of ensuring that an education system is using best practices.

Dr. Wilda says this about that ©