IRB Chair: "Nobody Really Knows" If IRBs Do Any Good

A former chair of the University of Wisconsin-Madison Education and Social/Behavioral Science IRB guesses that IRB review “most likely” protects subjects from harm, but concedes that “nobody really knows.” He also notes that it consumes tens of thousands of hours of work, mostly by researchers, at his university each year.

[Kenneth R. Mayer, “Working through the Unworkable? The View from Inside an Institutional Review Board,” PS: Political Science & Politics 49, no. 02 (April 2016): 289–93, doi:10.1017/S1049096516000226.]

AAHRPP Claims IRBs Rarely Disapprove Research

The Association for the Accreditation of Human Research Protection Programs (AAHRPP) notes that of the organizations it surveyed (it doesn't report how many or how they were selected) 63 percent "did not disapprove any protocols in the past year" and only "13.2 percent disapproved two or more percent." The AAHRPP uses these figures to argue that the "perception that IRBs disapprove a significant portion of research protocols submitted is not true."

I wonder whom they are accusing of holding this perception. Serious observers of IRBs know that a board need not formally reject a proposal to make it impossible for the researcher to proceed. Look at what happened at Linda Thornton. Look at what happened to Kathryn Edgerton-Tarpley's student. Or the anonymous researcher who wanted to study how parents discipline their children. In these cases, researchers abandoned projects in the face of IRB intransigence, allowing the IRB to claim it had not disapproved the project.

I am therefore unsurprised by AAHRPP's figures. I am surprised that AAHRPP thinks that formal disapproval is a meaningful measure of the IRB suppression of research, and by its stooping to this straw man argument.

AAHRPP Organizations Check Fewer Boxes

The AAHRPP has reported that only 29 percent of its accredited organizations (not counting VA facilities, which don't have the option to uncheck boxes) pledged to apply all parts of the Common Rule to research not directly sponsored by the federal government. However, 72 percent of organizations that left at least one box unchecked still "applied the DHHS regulations to all research regardless of funding source." Thus, for a great many organizations, unchecking a box or two remains a way to escape federal oversight without offering researchers any additional freedom.

AAHRPP and PRIM&R Plan Conferences

Both the Association for the Accreditation of Human Research Protection Programs, Inc ., (AAHRPP) and Public Responsibility in Medicine and Research (PRIM&R) plan conferences next month. AAHRPP's conference, "Breaking Down Barriers, will be held in Washington, April 6-8, then PRIM&R meets in Boston, April 28-29, for a "Social, Behavioral & Educational Research Conference."

Texas's All-Star IRB Report

In February 2008, the University of Texas System formed an IRB Task Force to examine ways to improve IRB operations throughout the UT System. In April 2009, that task force issued its report: "IRB TASK FORCE REPORT: Trust, Integrity, and Responsibility in the Conduct of Human Subjects Research."

I recently obtained a copy of that report. While it has not previously been posted on the web, a University of Texas official assured me that a final policy report like this is public information under Texas law. So to make this important public document freely available, I have posted a copy on my website. (See link above.)

The report offers an exceptionally thorough and thoughtful consideration of how IRBs should work at great research universities. While some of its recommendations may be inapplicable to univerisities that are not part of larger systems or do not operate a medical campus, many of the task force's procedures and recommendations offer a model for others. I salute all those who were involved in the report's preparation.

In particular, I commend the following elements of the UT task force and its report.

Procedures

The UT system carefully set up the task force in a way that would build respect for its findings. In particular, it managed to:

1. Represent Multiple Disciplines

UT included task force members "representing a variety of academic disciplines," among them SACRHP member Lisa Leiden. (3, 24) It also sought help from a range of consultants, including such "national experts" Tina Gunsalus; John Heldens, Moira Keene, Dan Nelson, Ivor Pritchard, and Marjorie Speers. (8)

2. Include Stakeholders

At each UT campus, the task force solicited comments from university officials, IRB chairs, investigators, and other interested parties. (8)

3. Allow Adequate Time

The UT task force completed its deliberations over the course of a year, from February 2008 to February 2009, then released its report in April 2009. A report with so broad a scope cannot be rushed.

Recommendations

Careful investigation led to thoughtful recommendations. The report suggests that the UT system:

1. Employ Faculty Expertise

The UT report recognizes that researchers are often expert in a particular area of human subjects research, and it recommends that faculty experts be identified to prepare standards for specific types of research (e.g., research involving subjects with impaired capacity, internet research) and to consult on individual projects as needed. (10)

2. Utilize Flexibility and Empirical Evidence

The report recommends that the university encourage IRB staff to provide "an efficient level of regulatory review compatible with adequate protection of human subjects," rather than the most stringent level of review. In particular, it suggests that IRB staff and members employ empirical evidence when determining risk, by consulting experts or scholarly literature. (11) The report also suggests that IRBs rely strongly on experts when determining the scientific soundness of a proposal. (14)

3. Uncheck the Box

The UT report notes that, according to Speers, fewer than 50 percent of AAHRP accredited institutions check all the box on their federal-wide assurances, and that it is "primarily the major research universities that are considering unchecking the box," thus maximizing their flexibility in handling projects not directly funded by a Common Rule agency. (12) (More on this in an upcoming post.)

4. Diversify the IRB

The UT report recommends that "Institutional Officials should ensure that the institution's disciplines are well represented in the IRB." (18)

5. Provide IRB Oversight

The UT report recommends that institutions "consider the implementation of a research ombudsperson to increase the opportunity for rapid resolution of issues involving human subjects research." (21) It also suggests an ongoing "IRB Advisory Group" to implement recommendations and assist with human subject policy issues. (23)

6. Define Key Terms

The UT Report includes definitions of key terms. Among other things, these definitions make clear that information-gathering interviews, service surveys, and classroom activities may not meet the definition of human subjects research, and that biography and oral history interviews do not meet that definition. (27) As I mentioned earlier, this recommendation has already led to the deregulation of oral history at the University of Texas at Austin.

7. Explore Alternatives

The report include an appendix on "alternative IRB models," based on the November 2006 "National Conference on Alternative IRB Models." While it is promising that the task force is willing to consider such alternatives, I would have liked it to pick up on that conference's call for "further exploration" of models for social and behavioral research. The University of Pennsylvania policy on research in the sociobehavioral sciences might be such a model.

Goal

Most importantly, the UT task force understood the IRB problem as something larger than an administrative challenge. While it explored ways to increase IRB "effectiveness, efficiency, and productivity," it went beyond such managerial concerns to probe big questions about "IRB authority, mission, and functions," the "unnecessary obstruction of research and lapses of effective human subject protection," and best practices from the available literature. (8)

The task force hoped to foster a "culture of conscience" rather than a "culture of compliance," and it understood that conscience cannot be dictated from above. (20) If other universities also seek to promote a culture of conscience, they must give a voice to all those involved with human subjects research.

AAHRPP Urges Extra Care When Talking to Pregnant Women

As I mentioned in October, Marjorie Speers, president of the Association for the Accreditation of Human Research Protection Programs (AAHRPP), has stated that her organization permits institutions it accredits to submit Federalwide Assurances (FWAs) that do not pledge to impose federal regulations on all research not directly funded by the federal government. In otherwords, institutions may "uncheck the box" without losing accreditation.

At the same time, Speers insisted that "if the boxes are unchecked, we hold the organization to have equivalent protections in place for all research." In my earlier post, I asked what an equivalent exemption might look like.

AAHRPP takes a stab at answering this in the Winter 2010 issue of AAHRPP Advance.

"To Check or Uncheck the FWA Boxes" (p. 6) notes that "The regulations that come into play when the FWA boxes are checked were written primarily with clinical research in mind. Thus, if an organization checks Subpart B of the FWA, for example, it will not be allowed to conduct SBER studies that involve pregnant women, because any study involving pregnant women is required to advance biomedical knowledge."

To give institutions more flexibility, the article claims,

AAHRPP has designed its standards to apply as much to protecting participants in SBER as they apply to protecting those in biomedical research. When organizations have unchecked the boxes, AAHRPP allows them to provide protections that are appropriate to the level and nature of the risk involved in the study and meaningful to the type of research. Under the accreditation standards, an organization could require that the results of a SBER study that includes pregnant women, for example, must contribute to general knowledge or knowledge that is beneficial to society, rather than to biomedical knowledge.

While any concession is welcome, this one is unimpressive. The regulation in question, 45 CFR 46.204 states that "pregnant women or fetuses may be involved in research" only if the research holds out the prospect of direct benefit to the pregnant women or fetus or if "risk to the fetus is not greater than minimal and the purpose of the research is the development of important biomedical knowledge that cannot be obtained by any other means."

Changing "biomedical" to "general" here yields a requirement that researchers meet a higher standard if they want to interview, survey, or observe pregnant women than if they want to interview, survey, or observe non-pregnant women, prospective fathers, or other autonomous adults. Such a requirement violates the rights of both researchers and women.

While I appreciate AAHRPP's understanding that much of the Common Rule is inappropriate when applied to non-biomedical research, the example it has chosen suggests that as an organization, AAHRPP still lacks a basic grasp of some of the issues surrounding IRB review of social science research. What is needed is not mere tinkering with the Common Rule, but wholesale reconsideration of the rights and responsibilities of social scientists.

I thank Rob Townsend for bringing this to my attention.

AAHRPP Retreats from "Objective Data"

In July I posted my comments on AAHRPP's Proposed Revised Standards. At the time, I applauded Element I.5.B for insisting that "based on objective data, the Organization identifies strengths and weaknesses of the Human Research Protection Program, makes improvements, when necessary, and monitors the effectiveness of the improvements."

How disappointing, then, to find that the Final Revised Accreditation Standards omit the phrase about objective data. Are we to infer that AAHRPP considers objective data too difficult a standard, and wants institutions to base their programs on subjective impressions? Of course, most of the IRB regime is based on such guesswork, but I had thought that AAHRPP seeks to raise the level of IRB review.

AAHRPP Policy on FWAs Remains Blurry

Back in July, I reported on the AAHRPP's ambiguous position on whether the institutions it accredits may "uncheck the box" on their federalwide assurances.

AAHRPP's new Final Revised Accreditation Standards fail to resolve this ambiguity. They require that an accredited organization apply "its HRPP [Human Research Protection Program] to all research regardless of funding source, type of research, or place of conduct of the research," but do not state not whether that HRPP must track federal regulations in all cases.

Interviewed for the October 2009 Report on Research Compliance, AAHRPP President Marjorie Speers had this to say:

We believe an organization must protect participants in all of the human research it conducts, whether or not it receives federal funding . . . As an accrediting organization, we don't have an opinion on whether or not an institution should 'check the box,' on their FWAs to OHRP. If an institution 'checks the box,' then we hold the institution to follow the regulations to all research to which 'the box' applies. If the boxes are unchecked, we hold the organization to have equivalent protections in place for all research.

This does little to clarify matters. What are "equivalent protections" to those specified in federal regulations? Were AAHRPP site visitors correct to tell the University of California "that in order for a human research protection program to be accredited, it must apply the Common Rule and its subparts to all human research at the institution, irrespective of funding"? Or can a university add new categories for exemption and expedited review, as advocated by Lisa Leiden, and consider those equivalent to the federal categories?

Unchecking the box is one of the leading proposed remedies for IRB overreach. It is a pity that AAHRPP has missed this opportunity to address this movement more directly.

AAHRPP and the Unchecked Box

Regular readers of this blog likely know that most United States universities submit "federalwide assurances" (FWAs) pledging to abide by the Common Rule for research funded directly by federal agencies that have adopted that rule.

Section 4 of the standard assurance includes an optional pledge that "This Institution elects to apply . . . to all of its human subjects research regardless of the source of support, except for research that is covered by a separate assurance" either the Common Rule or the Common Rule and subparts B, C, and D of 45 CFR 46. Institutions that check this box--as seems to have been common in the past--with one stroke of the pen eliminate one of the major concessions made by federal regulators in 1981, when they promised that non-federally-funded research would not be regulated.

Recently, however, at least 164 universities have "unchecked the box," declining to promise to apply the regulations to all research. The American Association of University Professors has strongly recommended that universities uncheck the box as a first step toward devising procedures less burdensome than those specified in the regulations.

Malcolm Feeley has noted that unchecking the box could also yield important empirical data:

If there are few reports of negative consequences . . . they might encourage national officials to rethink the need for such an expansive regulatory system . . . On the other hand, if opt-out results in increased problems, the findings might help convince Katz, Dingwall, me, and still others of the value of IRBs.

Nor are such comments confined to outsiders. At the July 16, 2008, meeting of the Secretary's Advisory Committee on Human Research Protections, committee member Lisa Leiden of the University of Texas system spoke of her own interest in freeing nonfunded research from direct federal regulation:

We have talked about limiting the federal wide assurances, unchecking the box, and I believe the position that we're going to be taking is to advocate in a gentle way thinking about doing that. We have heard both sides of the story or maybe just a few sides, but we think that there are certainly some advantages. And one of the advantages might be . . . what can we do with the expedited review level. It seems that there is a lot of flexibility in that, and we might be able to increase some of that by unchecking the boxes and adding different categories for that.

Unchecking the box is therefore one of the most promising incremental reforms now on the table. This is why I was disappointed to see that the AAHRPP's proposed revised standards, described in my previous post, seem to preclude this option.

A correspondent questioned this assertion, noting that AAHRPP president Majorie Speers had mentioned unchecking the box in her presentation, "Finding Flexibility in the Regulations." But there's nothing in the slides to suggest that AAHRPP or Speers approves of such a practice, and a July 2008 memo from the University of California states that AAHRPP site visitors have told university administrators "that in order for a human research protection program to be accredited, it must apply the Common Rule and its subparts to all human research at the institution, irrespective of funding."

Either AAHRPP forbids accredited organizations from unchecking the box, or its policies are so unclear that its site visitors are giving out bad information. Either way, I suggest that the revised standards permit unchecking the box as a means of reform.

AAHRPP Proposes Revised Standards

Robert Townsend, PhD, kindly alerted me to the Proposed Revised Accreditation Standards of the Association for the Accreditation of Human Research Protection Programs (AAHRPP). The revisions are largely cosmetic, grouping many of the existing standards under new headings. As far as the review of social science and humanities research goes, I see no drastic departures from previous AAHRPP positions. This is a pity, since the standards need more substantive revision to meet the goals that AAHRPP has set for itself.

AAHRPP is accepting comments until July 30. My comment follows.

AAHRPP Calls for Research on IRBs and Behavioral and Social Science

Rob Townsend kindly alerted me to the Winter 2008 issue of AAHRPP Advance, published by the Association for the Accreditation of Human Research Protection Programs, which features the lead article, "IRBs and Behavioral and Social Science Research: Finding the Middle Ground." The article concedes that "many behavioral and social scientists feel constrained by a system that seems tilted toward biomedical research and, therefore, neither understands nor reflects their concerns." And it reports the interest of Drs. J. Michael Oakes, "an author and frequent lecturer on IRB review of behavioral and social science research," and Howard Silver, Executive Director of the Consortium of Social Science Associations, in addressing some of those concerns.

Oakes and Silver "encourage researchers to investigate scientifically IRB oversight of behavioral and social science research. Such research could determine whether IRBs are consistent in applying the federal regulations, whether IRBs are taking advantage of the flexibility that’s built into the regulations, and whether relationships between IRBs and social scientists are less strained on campuses that have separate IRBs to review behavioral and social science research. The resulting data could shed light on ways to relieve tensions between these two groups."

That sounds good, but it's a bit disappointing that the article does not acknowledge the considerable research already completed on this topic, much of it already cited on this blog. Nor does it remark on the curiosity that IRB oversight has continued for four decades without anyone knowing if it does any good.

Moreover, the article states as fact some beliefs that should be investigated with just the sort of research it calls for. I hope Oakes, Silver, and AAHRPP will allow data to challenge some of their own presuppositions.

Here are some questions that could be answered by further research.

1. When did IRB review of social science go bad, and why?

AAHRPP thinks it already knows the answer to this one. The article claims that "Tensions began building in the late 1990s in response to increased government scrutiny of research involving human participants," and that "the regulations have not changed. What’s new is their enforcement and, in many instances, that enforcement is overdue." I am at work on a history of IRB review of social science and humanities research, and I think more research can challenge this view.

The first part of the AAHRPP claim is doubtful; social scientists have protested IRB regulations since 1966, and tensions have waxed and waned since then. If we take this longer view, then the assertion about the immutability of the regulations is wrong; the regulations--first promulgated in 1974--changed twice, in 1981 and 1991. And the 1991 revisions greatly expanded the reach of IRBs. The 1981 regulations exempted survey, interview, and observational research unless it “deals with sensitive aspects of the subject’s own behavior, such as illegal conduct, drug use, sexual behavior, or survey or interview procedures is use of alcohol" and if "the subject’s responses, if they became known outside the research, could reasonably place the subject at risk of criminal or civil liability or be damaging to the subject’s financial standing or employability. The 1991 regulations, in contrast, eliminated the "sensitive aspects" clause and added potential harms to reputation to the list of triggers for IRB review. These changes were made over the objection of social scientists. And they set the stage for the conflict of the 1990s and today.

The claim that "what's new is [the regulations'] enforcement," is only half-true. Also new is the guidance issued by OPRR/OHRP since 1995 that reversed previous policies.

Finally, the article claims that "enforcement is overdue." Really? What errors did social scientists commit in the 1980s--a decade of relatively light regulation?

2. Why do IRBs sometimes delay or prohibit social science research?

Dr. Oakes states, “IRB members are not those folks who are looking to thwart your study. They are peer researchers who have a job to do.” But clearly some IRB members are looking to thwart studies, or else studies wouldn't get thwarted as often as they do. The question is how many IRB members do this, and why.

One part of this question concerns membership. Oakes's claim that IRB members "are peer researchers" depends on an odd definition of peers. In the NIH peer review process, for example, proposals are reviewed by study sections whose members are chosen for their expertise. The NIH's Center for Scientific Review requires, among other things, that

"* Expertise is the paramount consideration when developing/updating a study section roster.

"* Each scientific area reviewed by the study section needs appropriate expert representation."

IRBs theoretically must include experts on each type of research reviewed, but Oakes knows as well as I that this requirement is often ignored. Additional research might indicate how often a researcher faces an IRB with no expertise in the methods under review.

Then, of course, some IRB members are not researchers at all, but the "one member whose primary concerns are in nonscientific areas" required by the regulations. As Laura Stark's dissertation suggests, these members can be particularly undisciplined in their meddling.

3. What types of research now fall subject to IRB review?

Like PRIM&R, AAHRPP thinks that IRBs review only two kinds of scholarship: biomedical research, and something called "behavioral and social science research." The article states, "AAHRPP’s Founding Members, Board of Directors, Council on Accreditation, and Supporting Members all include representatives of organizations engaged in behavioral and social science research."

This statement suggests the fallacy of the undistributed middle term:

* Ehnographers are represented by organizations engaged in behavioral and social science research.
* Organizations engaged in behavioral and social science research have a voice in AAHRPP.
* Therefore, ethnographers are represented by organizations that have a voice in AAHRPP.

The second premise is "undistributed," since it is not true that all organizations engaged in behavioral and social science research have a voice in AAHRPP.

Here's a counter example:

* Countries in South America, Africa, and South Asia are not in North America or Europe.
* Countries from parts of the world other than North America and Europe are permanent members of the UN security council and the G-8.
* Therefore, countries in South America, Africa, and South Asia are permanent members of the UN Security Council and the G-8.

In committing this fallacy, AAHRPP lumps together a dozen or more scholarly disciplines--each with its own history, methods, and ethics--into a single category: "behavioral and social science research." The AAHRPP website does not list the disciplinary affiliations of members of its Board of Directors, Council on Accreditation, or list of site visitors, but if there's a journalist, historian, or folklorist in the lot, I'll be surprised.

To take the example I know best, oral historians do not expect psychologists, social workers, or education researchers to understand or represent their interests. AAHRPP (like PRIM&R) should find out how many disciplines are now subject to review, and include representatives from all of them.

4. What models of ethical review exist, and what models might we imagine?

The article asks "whether relationships between IRBs and social scientists are less strained on campuses that have separate IRBs to review behavioral and social science research." But that is only one of several alternative systems in place on various campuses. For example, Macquarie University delegates ethical review to a number of subcommittees with special expertise in certain fields. And the University of Pennsylvania allows researchers using some social science methods to forego "a fixed research protocol." And we can imagine even more models, some of which would require redrafting present regulations, others of which might not.

I appreciate AAHRPP's call for research, and I hope it agrees that research is most valuable when the answers are not predetermined.