Friday, September 11, 2015

NPRM: Escape for Many, Scant Relief for Those Left Behind

While the NPRM might do much to reduce the number of projects requiring IRB review, it would do little to improve the quality of review for those projects for which it is still required. This is a retreat from the more ambitious plans of the 2011 advance notice of proposed rulemaking.


[This post will be cross-posted to the Petrie-Flom Center's Bill of Health, which is conducting an online NPRM Symposium.]


Fewer reviews

A great many of the proposals in the NPRM would reduce IRB jurisdiction and workload. These include:


  • Exclusion of some types of studies, such as oral history.
  • Changes to exemption procedures, to encourage alternatives to the present system, in which many institutions regard an exemption determination as a “level of review.”
  • Allowing a single IRB to approve a multi-site study.
  • Elimination of continuing review for most expedited studies.

That’s all well and good. IRBs and IRB staffs are overworked, often spending much of their time completing paperwork for low-risk studies. Both researchers and reviewers deserve relief.


But what about those studies that still face full-board review?

Single IRB review, but no consistency


The NPRM offers no reforms to improve the quality of IRB decision-making, but it could make IRB arbitrariness harder to spot.


Right now, some of the best evidence that IRBs make arbitrary decisions comes when identical proposals are submitted to multiple IRBs. Occasionally this happens as part of a deliberate test, as when Laura Stark asked eighteen IRB chairs how they would handle a proposal to study hiring discrimination, and got “very different” answers.


More commonly, we learn about inconsistency when actual researchers want to do work at multiple sites. By submitting identical proposals to multiple IRBs, they perform a natural experiment, and learn how inconsistent and arbitrary IRB decisions can be. A classic example, cited first the ANPRM and now the NPRM, is Green et al., which found “Wide variation in standards applied to review and approval of IRB applications.”


The NPRM addresses such studies by reducing the number of IRBs that must be consulted, but not by making the decision-making any better.


Imagine a weather station whose operator complains of having to check a hundred thermometers, which vary widely in the temperatures they report. Allowing the operator to check a single thermometer will greatly reduce his burden, but it won’t do anything to address the lack of reliable information from any of the instruments.


Public scrutiny of consent forms, but not IRB decisions


The NPRM suggests mandatory posting of consent forms for clinical trials, “so that anyone drafting a consent form will do so knowing that it will eventually be subject to public scrutiny.” But it does not explain how the public will distinguish the poor consent forms drafted by researchers from those good ones mangled in response to IRB demands. Will PRIM&R continue to claim that “IRBs neither interact with subjects nor write consent forms?”


More significantly, the NPRM does not propose the publication of IRB decisions themselves, though public scrutiny of these documents could do a great more to address the arbitrariness the NPRM is designed to combat.


More consistent guidance, but even less of it


Critics of the current system have noted how hard it is to get timely, reliable guidance out of the federal government. As Klitzman notes, a call to OHRP produces just “vague generalities.” SACHRP is occasionally better, sometimes offering specific advice on matters like internet research. But SACHRP, an HHS body, is primarily focused on medical research.


In an effort to promote consistency across agencies in the interpretation of the Common Rule, the NPRM would require “Federal guidance on the requirements of this policy shall be issued only after consultation, for the purpose of harmonization (to the extent appropriate), with other Federal departments and agencies that have adopted this policy, unless such consultation is not feasible.”


But rather than establish a body representative of the various agencies promulgating the rules, and of the various disciplines subject to them, the budget section of the proposal suggests that it would be the job of “Three [OHRP] staff people at the GS–14 level to draft new guidance and revise old guidance.”


This strikes me as a poor mechanism for providing the timely, consistent guidance researchers and IRBs need. A better choice would be to fund a widely representative expert body like Canada’s Panel on Research Ethics, and both empower and require it to provide prompt, written, public responses to IRB queries.


Revision of expedited categories, but not of the regulations


The NPRM promises that the list of activities deemed minimal risk will be updated “at least every 8 years.” While that’s lovely, it also suggests that the NPRM framers do not expect that the rest of the regulations will be regularly revised, or at least not more frequently than once a generation. And that’s a big disappointment. Canada has shown the value of more frequent revision, with a wholesale revision after 12 years, and a significant update after only four.


No centralized reporting


The ANPRM (question 69) suggested having “all data on adverse events and unanticipated problems collected in a central database accessible by all pertinent Federal agencies.” Explaining the ANPRM to the Presidential Commission for the Study of Bioethical Issues, ANPRM architect Ezekiel Emanuel stressed his wish for "a learning process that would be constant and dynamic to reflect actual risk, which I took to be a reference to the database plan.


The NPRM notes that most commentators opposed the central database as burdensome and impractical, and so the NPRM takes it off the table. OK, but in its absence, it offers no other means for IRBs to learn or any requirement that they base their decisions on evidence.


No appeals


Finally, the NPRM fails to mention one of the key proposals of the ANPRM: “a requirement that every institution must provide an appropriate appeal mechanism.” As I noted in 2011, this suggestion was the closest the ANPRM came to addressing the issue of IRB incompetence. Its omission from the NPRM is a severe disappointment.


Eliminating distractions isn't enough


The NPRM argues that “cumbersome and outdated regulatory standards overwhelm and distract oversight bodies and other stakeholders from appropriately addressing the real risks and benefits of research.”


That is no doubt true, and reducing burdens is a necessary step toward better IRB decision-making. But it is not a sufficient step. We have substantial documentation of IRB ineptitude, and no corresponding evidence of IRB effectiveness. Even if the NPRM’s drafters were unwilling to question the core premises of the IRB system, could they not have taken some steps toward improving those reviews it will continue to generate?

No comments: