Tag Archives: National Research Council

‘Public Access ‘to research

9 Jun

A huge issue in research is open access or public access. Peter Suber proposes one definition in Open Access Overview:

Many OA initiatives focus on publicly-funded research.

  • The argument for public access to publicly funded research is strong, and a growing number of countries require OA to publicly-funded research.

  • The campaign for OA to publicly-funded research usually recognizes exceptions for (1) classified, military research, (2) research resulting in patentable discoveries, and (3) research that authors publish in some royalty-producing form, such as books. Recognizing these exceptions is at least pragmatic, and helps avoid needless battles while working for OA to the largest, easiest subset of publicly-funded research.

  • The lowest of the low-hanging fruit is research that is both royalty-free and publicly-funded. The policy of the US National Institutes of Health (NIH) is a good example.

  • But the OA movement is not limited to publicly-funded research, and seeks OA to research that is unfunded or funded by private foundations (like the Wellcome Trust or Howard Hughes Medical Institute).

http://legacy.earlham.edu/~peters/fos/overview.htm

A consortium of groups is looking at open access for research in the U.S.

Jennifer Howard writes in the Chronicle of Higher Education article, Universities and Libraries Envision a ‘Federated System’ for Public Access to Research about public access to research:

This week, the Association of American Universities, the Association of Public and Land-Grant Universities, and the Association of Research Libraries are offering a plan they call the Shared Access Research Ecosystem, or Share.

Share would expand on systems that universities and libraries have long been building to support the sharing and preservation of research. The groups behind Share have been circulating a document, dated June 7, that lays out the basics behind the idea.

Academic institutions have invested heavily in “the infrastructure, tools, and services necessary to provide effective and efficient access to their research and scholarship,” the document says. “Share envisions that universities will collaborate with the federal government and others to host cross-institutional digital repositories of public-access research publications.”

Under Share, each university or research institute that gets federal research money would designate an existing digital repository “as the site where its articles will be deposited for public access and long-term preservation,” meeting the requirements of the Obama administration’s policy. Many universities already have digital repositories up and running. Those that do not could piggyback on the repositories of other institutions. A smaller institution could designate one of its state’s public universities as its deposit site, for instance.

The document also emphasizes elements that would be essential to make Share a viable way to comply with the new public-access policy. For example, principal investigators would need identifiers such as Orcid numbers to track their research activity, and every publication would need to have copyright-license terms embedded in its metadata so that repository systems would know how to handle it.

With those protocols in place, Share would be “a federated system of university repositories,” John C. Vaughn, the Association of American Universities’ executive vice president, said in an interview. “Potentially there’s a way to connect the whole corpus of U.S. higher-education institutions that receive federal research funding.”http://chronicle.com/blogs/wiredcampus/universities-and-libraries-envision-a-federated-system-for-public-access-to-research/44147?cid=pm&utm_source=pm&utm_medium=en

Here is the statement of the National Research Council, NAS Forum:

Statement on Expanded Public Access to Publications

May 14, 2013

National Research Council, NAS Forum

The Office of Science and Technology Policy (OSTP) Memorandum, “Increasing Access to the Results of Federally Funded Scientific Research,” provides new opportunities for a productive partnership between research universities and Federal research funding agencies. Enhanced access to the results of federally funded research accelerates the pace of scientific discovery, promotes innovation, and enriches education.

Research institutions are mission-driven, and common to their individual missions is a shared commitment to create and build upon new knowledge, make accessible the results of their research, and preserve information for future generations. The member universities of the Association of American Universities and the Association of Public and Land-grant Universities conduct nearly two-thirds of federally funded research, and their members generate a significant percentage of the peer-reviewed scholarly publications covered by the OSTP memorandum. Research libraries, with others in their institutions, supply much of the infrastructure in support of this research and in many cases also provide access to the final peer-reviewed scholarly publications and digital data produced by institutional researchers. Research institutions also have a long history of and experience with collaboration.

It will be important that the policies federal agencies develop minimize the cost and complexity of compliance with grant requirements for both principal investigators and their institutions. University budgets are stretched thin with few discretionary resources. Federal research funding is under considerable pressure, and there is understandable concern within the university research community that implementation of a broad national public access policy should proceed, but in a manner that respectfully balances the rewards that will come from making research results more widely available with any sacrifices to current research progress the effort may entail.

The OSTP memorandum wisely provides agencies with latitude in developing and maintaining the public access repositories called for, stipulating that such repositories could be maintained by the agency funding the research, another federal agency, or through a public/private partnership with external parties. We believe that the most important issue at this time, however, is not who builds and operates the repository but the functionality of that public access resource.

Functional Properties of Public Access Repositories

Our three university and library associations believe that the following functionality considerations will be key to achieving the public access goals of the OSTP policy directive:

1. Copyright or IP rights should not be assigned to final peer-reviewed scholarly publications in an exclusive manner that would prohibit preservation, discovery, sharing, and machine-based services such as text mining. Similarly, federal policies can stimulate the development of new tools and services (human and machine), and licensing arrangements should ensure that no one single entity or group secures exclusive rights.

2. In order for the broad goals of the OSTP plan to be achieved, agency compliance requirements should be transparent, and deposit requirements should be easy for the researcher – or institution or publisher depositing on behalf of the researcher — to accomplish. In addition, agencies should develop effective grant tracking tools in advance of public access policy implementation that will enable universities to better manage compliance with agency regulations. To the extent possible, agency requirements should be comparable across agencies to minimize the burden on universities of mandated compliance requirements.

3. Final peer-reviewed scholarly publications should be linked openly to their source data to the extent possible to allow for reuse and replication of results, and such links should be established in a generalizable, sustainable manner.

4. Open standards are necessary to ensure interoperability in repository system design for search and discovery, and the metadata describing publications should be based on open standards to ensure that the public can read, download, and perform text mining on the publications.

5. Agencies should require the use of persistent, unique identifiers for grants, publications, data, and authors to foster reuse of content and development of new services by individuals and machines.

6. A variety of forms of metrics and identifiers should be supported to provide information on access, use, and impact of final peer-reviewed scholarly publications.

7. Final peer-reviewed scholarly publications from publically funded research should be accessible to persons with disabilities consistent with Section 508 of the Rehabilitation Act of 1973.

8. Bulk downloads of the corpus of scholarly publications for research purposes should be allowed under specified terms and conditions developed by agencies in consultation their external constituencies.

The Role of University Repositories

The experience of many research university institutional repositories demonstrates that they can operate effectively with the functional properties noted above. Members of the higher education community are initiating a study of the feasibility of federating existing institutional repositories and other campus-based infrastructure into a virtual repository which, in cooperation with federal funding agency repositories and others, could serve as a distributed system in support of the goals of the OSTP Public Access Policy. Such an approach would build on extensive, existing cyberinfrastructure already in place. This feasibility study will build on earlier work already conducted, in particular, a recent NSF report, A Feasibility Study for a National Science Foundation Open-Access Publication Repository. Should this current study demonstrate that such a federated system could be integrated into and extend current capabilities and capacities in an efficient and effective manner, we suggest that OSTP, federal agencies and other pertinent partners carefully consider joining with research universities in its creation and use as one component of the requirements of the OSTP article repository. The feasibility study will have to address a realistic timeline for implementation and the sustainability of the institutional repository model, including stability and adequacy of the business/financing plan. The study also will need to address the development of common standards and capacities to support comparable functionality across institutional repositories and the incorporation of universities and organizations not included in the AAU/APLU/ARL memberships.

Harvard Law has Notes on the Federal Research Public Access Act:

This page is part of the Harvard Open Access Project (HOAP).

 

The bill itself

  • The Federal Research Public Access Act (FRPAA) requires “free online public access” to a very large swath of publicly-funded research in the United States. It strengthens the open access (OA) mandate at the NIH by reducing the maximum embargo period from 12 months to six months, and extends the strengthened policy to all the major agencies of the federal government.
  • It doesn’t merely reduce the maximum embargo to six months, it requires OA “as soon as practicable” after publication (Section 4.b.4), but no later than six months after publication.
  • It asks agencies to come up with their own policies within the general guidelines laid down in the bill. It’s not a one-size-fits-all solution and agencies are free to differ on the details. They will have one year from the bill’s passage to develop their policies (4.a).
  • The FRPAA guidelines require agencies to mandate “free online public access” for agency-funded research. The guidelines do not define “free online public access” (4.b.4). Nor do they stipulate the timing of deposits, only the timing of OA. For researchers employed and not merely funded by the federal government, FRPAA allows no embargo at all (4.c.2).
  • Like the NIH policy, FRPAA applies to the authors’ peer-reviewed manuscripts (4.b.2), not to the published editions of their articles. Like the NIH policy, it allows consenting publishers to replace the peer-reviewed manuscripts with the published editions (4.b.3). It does not apply to classified research or royalty-producing work such as books (4.d.3). It also exempts patentable discoveries, but only “to the extent necessary to protect a…patent” (4.d.3).
  • Unlike the NIH policy, FRPAA doesn’t specify the OA repository in which authors must deposit their manuscripts, the way the NIH specifies PubMed Central. FRPAA leaves this decision up to the individual agencies. They could host their own repositories or make use of existing repositories, including the institutional repositories of their researchers. FRPAA only requires that the repositories meet certain conditions of OA, interoperability, and long-term preservation (4.b.6).
  • FRPAA and the NIH policy differ slightly in how they secure permission for the mandated OA. The NIH requires grantees to retain the non-exclusive right to authorize OA through PubMed Central. If a given publisher is not willing to allow OA on the NIH’s terms, then grantees must look for another publisher. FRPAA requires agencies to “make effective use of any law or guidance relating to the creation and reservation of a Government license that provides for the reproduction, publication, release, or other uses of a final manuscript for Federal purposes” (4.c.3). The FRPAA approach gives agencies more flexibility. Agencies may use the battle-tested NIH method if they wish. They may use a federal-purpose license such as that codified in 2 CFR 215.36(a) (January 2005) if they wish. Or they may make use of “any [other] law or guidance” that would be “effective” in steering clear of infringement.
  • FRPAA does not amend copyright or patent law (4.e).
  • FRPAA applies to all unclassified research funded in whole or part (4.b.1) by agencies whose budgets for extramural research are $100 million/year or more (4.a). This includes the Department of Agriculture, Department of Commerce, Department of Defense, Department of Education, Department of Energy, Department of Health and Human Services, Department of Homeland Security, Department of Transportation, Environmental Protection Agency, National Aeronautics and Space Administration, and the National Science Foundation.
  • FRPAA mandates OA for more research literature than any other policy ever adopted or ever proposed.

Contents

Every issue has both pros and cons.

The pros and cons of open access detailed in the Nature article,Open Access by Kate Warlock:

Supporters of Open Access to scientific literature often portray it as the definitive and inevitable model for scientific publishing, but it is far from being the last word on new modes of access. In reality, stakeholders in scientific publishing are in the midst of adjusting to the revolutionary new possibilities offered by the Web and the online journal article for scholarly communication.

In this Nature forum, a range of stakeholders in scientific publishing have made their cases at length, and often persuasively. Agreement in the industry on the best route forward remains distant, however, and the level of emotion behind the debate has served in some cases to obfuscate discussion. This article aims to provide an independent assessment of the key arguments, and to flag up areas where questions remain unanswered.

Proponents of a move to open access argue that this will benefit science and society in general. A report published last April by the UK Wellcome Trust1 assumes that “the benefits of research are derived principally from access to research results”, and therefore that “society as a whole is made worse off if access to scientific research results is restricted”.

Part of the remit of not-for-profit organisations such as the Wellcome Trust which fund research may be the full dissemination of results. But even where research is publicly-funded, taxes are generally not paid so that taxpayers can access research results, but rather so that society can benefit from the results of that research; in the form of new medical treatments, for example. Publishers claim that 90% of potential readers can access 90% of all available content through national or research libraries, and while this may not be as easy as accessing an article online directly it is certainly possible.

Funding for scientific research also comes from a variety of sources – in some countries such as Australia and New Zealand around 80% of R&D funding comes from the public purse, while in Japan and Switzerland only about 10% is government-funded1. It is therefore not necessarily the case that taxpayers fund most scientific research.

Another criticism of open access is that payment for publication could create conflicts of interest and have a negative impact on the perceived neutrality of peer review, as there would be a financial incentive for journals to publish more articles. The importance of the role of peer review does not diminish under an Open Access model, and structures need to be in place to ensure that peer reviewers are not unduly influenced by the needs of their publishers.

In some ways though this argument can apply as much to the current subscription-based system as publishers often justify price increases on the grounds of an increase in the number of journal articles published. This suggests that there are financial advantages for both Open Access and subscription-based publishers in publishing more articles. http://www.nature.com/nature/focus/accessdebate/34.html

Research can be impacted if a system to share knowledge is not devised.

Resources:

Overview: The NIH Public Access Policy

http://publicaccess.nih.gov/

Expanding Public Access to the Results of Federally Funded Research                                                                           http://www.whitehouse.gov/blog/2013/02/22/expanding-public-access-results-federally-funded-research

Where information leads to Hope. ©  Dr. Wilda.com

Dr. Wilda says this about that ©

Blogs by Dr. Wilda:

COMMENTS FROM AN OLD FART©                           http://drwildaoldfart.wordpress.com/

Dr. Wilda Reviews ©                                                http://drwildareviews.wordpress.com/

Dr. Wilda ©                                                                                      https://drwilda.com/

What , if anything, do education tests mean?

27 Nov

Moi received a review copy from Princeton University Press of Howard Wainer’s Uneducated Guesses. The publication date was September 14, 2011. In the preface Wainer states the goal of the book, “It deals with education in general and the use of tests and test scores in support of educational goals in particular.” Wainer tries to avoid not only the policy, but the ethical analysis of the analysis of the improper use of tests and test results by tightly defining the objective of the book at page four. The policy implications of using tests and test results to not only decide the direction of education, but to decide what happens to the participants in education are huge. Moi wonders if Wainer was really trying to avoid the unavoidable?

For moi, the real meat of the book comes in chapter 4. Wainer says:

In chapter 3 we learned that the PSAT, the shorter and easier version of the SAT, can be used effectively as one part of the selection decision for scholarships. In this chapter we expand on this discussion to illustrate that the PSAT also provides evidence that can help us allocate scarce educational resources…. [Emphasis Added]

Wainer examines the connection by analyzing and comparing test results from three high school districts. Those schools are Garfield High School in L.A., the site of the movie “Stand and Deliver.” La Canada High School in an upscale L.A. Suburb and Detroit, a very poor inner city school district. The really scary policy implication of Wainer’s very thorough analysis is found at page 44, “Limited resources mean that choices must be made.” Table 4-4 illustrates that real life choices are being made by districts like Detroit. What is really scary is that these choices affect the lives of real human beings. Of course, Wainer is simply the messenger and can’t be faulted for his analysis. According to Wainer, it is very tricky to use test results in predicting school performance and his discussion at page 53 summarizes his conclusions.

Perhaps the most chilling part of Wainer’s book is chapter 8 which deals with how testing and test results can adversely impact the career of a teacher when so-called “experts” incorrectly analyze test data. It should be required reading for those who want to evaluate teacher performance based upon test results.

Overall, Uneducated Guesses is a good, solid, and surprisingly readable book about test design, test results, and the use of test results. The truly scary part of the book describes how the uninformed, unknowing, and possibly venal can use what they perceive to be the correct interpretation to make policy judgments which result in horrific societal consequences.

Wainer makes statistics as readable as possible, because really folks, it is still statistics.

Here is the full citation for the book:

Uneducated Guesses: Using Evidence to Uncover Misguided Education Policies

Howard Wainer

Cloth: $24.95 ISBN: 9780691149288

200pp.

Wainer’s book will come in handy when reading Eric A. Hanushek’s analysis of a National Research Council report.

Joy Resmovits writes about Eric A. Hanushek’s analysis of a National Research Council report in the Huffington Post article, Stanford Economist Rebuts Much-Cited Report That Debunks Test-Based Education:

When the National Research Council published the results of a decade-long study on the effects of standardized testing on student learning this summer, critics who have long opposed the use of exams as a teaching incentive rejoiced.

But Eric Hanushek, a Stanford University economist who is influential in education research, now says the “told you so” knee-jerk reaction was unwarranted: In an article released Monday by Harvard University’s journal Education Next, Hanushek argues that the report misrepresents its own findings, unjustifiably amplifying the perspective of those who don’t believe in testing. His article has even caused some authors of the NRC report to express concerns with its conclusions….

According to Hanushek’s analysis, the panel’s thorough examination of multiple studies is not evident in its conclusions.

“Instead of weighing the full evidence before it in the neutral manner expected of an NRC committee, the panel selectively uses available evidence and then twists it into bizarre, one might say biased, conclusions,” Hanushek wrote.

The anti-testing bias, he says, comes from the fact that “nobody in the schools wants people looking over their shoulders.”

Hanushek, an economist, claims that the .08 standard deviation increase in student learning is not as insignificant as the report makes it sound. According to his calculations, the benefits of such gains outweigh the costs: that amount of learning, he claims, translates to a value of $14 trillion. He notes that if testing is expanded at the expense of $100 per student, the rate of return on that investment is 9,189 percent. Hanushek criticized the report for not giving enough attention to the benefits NCLB provided disadvantaged students.

The report, Hanushek said, hid that evidence.

“They had that in their report, but it’s buried behind a line of discussion that’s led everybody who’s ever read it to conclude that test-based accountability is a bad idea,” he said. Hanushek reacted strongly, he said, because of the “complacency of many policymakers” who say education should be improved but that there are no effective options.

http://www.huffingtonpost.com/2011/11/23/eric-hanushek-rebuts-much_n_1108690.html?ref=email_share

Citation:

Grinding the Antitesting Ax: More bias than evidence behind NRC panel’s conclusions
Eric A. Hanushek,Education Next, WINTER 2012 / VOL. 12, NO. 2

Incentives and Test-Based Accountability in Education: A report from the National Research Council Checked by Eric A. Hanushek

http://educationnext.org/grinding-the-antitesting-ax/

One of the reasons why Hanushek’s critique is so important, aside from the implications that testing has under No Child Left Behind, is the push to use student test results in teacher evaluation. Valerie Strauss has an article in the Washington Post about a study which questions the use of student testing in the teacher evaluation process and the article includes links to the full report. In Study Blast Popular Teacher Evaluation Method Strauss reports:

Student standardized test scores are not reliable indicators of how effective any teacher is in the classroom, not even with the addition of new “value-added” methods, according to a study released today. It calls on policymakers and educators to stop using test scores as a central factor in holding teachers accountable.

Value-added modeling” is indeed all the rage in teacher evaluation: The Obama administration supports it, and the Los Angles Times used it to grade more than 6,000 California teachers in a controversial project. States are changing laws in order to make standardized tests an important part of teacher evaluation.

Unfortunately, this rush is being done without evidence that it works well. The study, by the Economic Policy Institute, a nonpartisan nonprofit think tank based in Washington, concludes that heavy reliance on VAM methods should not dominate high-stakes decisions about teacher evaluation and pay.

Here is the report link

Sarah Garland of the Hechinger Report has written the article, Should value-added teacher ratings be adjusted for poverty?

In Washington, D.C., one of the first places in the country to use value-added teacher ratings to fire teachers, teacher-union president Nathan Saunders likes to point to the following statistic as proof that the ratings are flawed: Ward 8, one of the poorest areas of the city, has only 5 percent of the teachers defined as effective under the new evaluation system known as IMPACT, but more than a quarter of the ineffective ones. Ward 3, encompassing some of the city’s more affluent neighborhoods, has nearly a quarter of the best teachers, but only 8 percent of the worst.

The discrepancy highlights an ongoing debate about the value-added test scores that an increasing number of states—soon to include Florida—are using to evaluate teachers. Are the best, most experienced D.C. teachers concentrated in the wealthiest schools, while the worst are concentrated in the poorest schools? Or does the statistical model ignore the possibility that it’s more difficult to teach a roomfull of impoverished children?

Saunders thinks it’s harder for teachers in high-poverty schools. “The fact that kids show up to school hungry and distracted and they have no eyeglasses and can’t see the board, it doesn’t even acknowledge that,” he said.

http://hechingerreport.org/content/should-value-added-teacher-ratings-be-adjusted-for-poverty_6899/

The question is what do test results mean and more importantly, how are test scores to be used? Wainer’s book attempts to analyze these questions.

Citation:

Should value-added teacher ratings be adjusted for poverty?

Sarah Garland, The Hechinger Report, November 22, 2011

http://hechingerreport.org/content/should-value-added-teacher-ratings-be-adjusted-for-poverty_6899/

Every population of kids is different and they arrive at school at various points on the ready to learn continuum. Schools and teachers must be accountable, but there should be various measures of judging teacher effectiveness for a particular population of children. Perhaps, more time and effort should be spent in developing a strong principal corps and giving principals the training and assistance in evaluation and mentoring techniques. There should be evaluation measures which look at where children are on the learning continuum and design a program to address that child’s needs.

Dr. Wilda says this about that ©