IRBs "Jerk Around" Education Research

Debra Viadero reports on the recent conference of the American Educational Research Association: "Security Checks of U.S. Education Contractors to Change," Education Week, 2 April 2008.

She includes the following description of one session:

Under federal human-protections laws, studies that involve human subjects—a category that includes most research in education—first have to be approved by institutional review boards, or IRBs, based at researchers’ home universities or research organizations.

But that process can sometimes be fraught with frustration and distrust . . . .

“While I was behind the curtain, it seemed to me that our board was quite reasonable,” said Frederick D. Erickson, a professor of anthropology in education at the University of California, Los Angeles. He has served on three institutional review boards over the course of his career. “Now, I’ve got a project of my own in expedited institutional review,” he added, “and it’s being jerked around in ways that make my blood boil.”

Some problems with the process, said Melissa S. Anderson, an associate professor of higher education at the University of Minnesota- Twin Cities who has studied IRBs, is that researchers often disagree with the boards’ judgments or may be skeptical of their authority.

“It’s, ‘What right do they have to tell me whether or not I can do research?’ ” she said. “The issue of peer review when peers aren’t seen as peers is … a sticking point.”

“This can lead to IRB shopping,” Ms. Anderson added, which is what occurs when researchers working on a study involving multiple universities try to figure out which one’s board is most likely to approve their project. “That’s becoming increasingly common and problematic.”

In a national survey of scientists that Ms. Anderson and her colleagues conducted last year, 5 percent of respondents admitted to having ignored or circumvented human-research requirements sometime in the previous three years. When medical researchers were removed from the sample, that percentage rose to 8 percent.

Trauma-Based Research Is Less Risky Than Imagined

The March 2008 issue of the Journal of Empirical Research in Human Research Ethics is out. As with previous issues of this journal, several articles present empirical evidence that challenges the assumptions used by many IRBs.

The lead editoral, Joan E. Sieber's "Protecting the Vulnerable: Who Are They?" provides a good summation of some of the the findings:

In this issue of JERHRE, five articles demonstrate the importance of applying an empirical approach to understanding vulnerability. Each article demonstrates a fallacy of using a simple subpopulation approach, and the importance of a more reasoned, nuanced and empirical evaluation of vulnerability.

Luebbert, Tait, Chibnall, and Deshields show how the labels we apply to subpopulations can mislead. They found that ethics committee members view psychiatric subjects as having greater vulnerability to coercion and less decisional capacity than medical subjects, even when the medical illness is of a severity likely to engender serious psychiatric comorbidities.

Three articles (DePrince and Chu; Chu, DePrince and Weinzierl; and Schwerdtfeger and Goff) evaluate the vulnerability of trauma victims, including children and young pregnant women, to research that focuses on their past traumas. Some have argued that such research “retraumatizes” the victims. However, all three studies found that trauma victims experience such research participation as distinctly beneficial.

In their article, "The Effects of Trauma-Focused Research on Pregnant Female Participants," Kami L. Schwerdtfeger and Briana S. Nelson Goff conclude from their review of the existing literature that

although trauma-based research may produce intense emotions, it is not re-traumatizing nor does it cause harm to participants. Studies involving a variety of trauma survivors found that participation in the research was not overwhelming or distressing and was generally an experience that participants would be willing to repeat.

Their own study found this also to be true of pregnant women.

The possibility that interviewing may traumatize narrators has been used as a chief justification for IRB review of oral history. (See, for example, Taylor Atkins's comments in Kanya Balakrishna, "Humanities Research May See More Rules," Yale Daily News, 17 April 2007.) But empirical research suggests that this possibility is rather small. It is therefore probable that by deterring interviews with trauma survivors, IRBs are significantly more likely to deny them a positive experience than to protect them from harm.

New URL

Perceptive readers may have noted a slight change in this blog. I have moved it to the custom domain, institutionalreviewblog.com. The old domain, institutionalreviewblog.blogspot.com, should still work as well.

Maureen Fitzgerald's Ethics Project

Anthropologist and George Mason University alumna Maureen Fitzgerald, now affiliated with the University of Sydney, is the director of "An Analysis of Research Ethics and the Ethical Review Process as Culture and Cultural Process," an ongoing investigation more succinctly named "The Ethics Project."

Between 2001 and 2005 she and her associates observed the workings of 29 ethics committees in Australia, Canada, New Zealand, the United Kingdom, and the United States--not finding significant differences across national boundaries. They have published several papers based on this work, most of which are available on the project's website. Since the themes of many of these publications overlap, I will focus my comments on three articles that I found particularly helpful.

Peter Moskos on What to Tell Your IRB

Sociologist Peter Moskos is author of Cop in the Hood, a book about police work he wrote based on his fieldwork as a Baltimore police cadet and police officer. He maintains a blog concerning the issues raised in the book (a splendid marriage of old media and new), and on February 12 he commented about the ethical issues raised by Sudhir Venkatesh's Gang Leader for a Day. In that posting, "Outing the insiders," Moskos wrote,

I’ve never been a fan of the I.R.B. Few professor are. I don’t think that overt non-experimental academic researchers should need approval to observe and interact with most human subjects. We’re not giving out experimental drugs. We’re not running experiments. We’re watching and talking and living. I don’t even like the term “human subjects.” It’s dehumanizing. They're people, damnit! It’s condescending to think that adults aren’t smart enough to make their own decisions about what to say to whom. And if they’re not, well, such is life.

Nor am I convinced that research subjects who harm others deserve institutional protection. I believe academics should act under a code similar to journalists. But federal law disagrees with me. And the press has explicit constitutional protection that professors don’t.

AAHRPP Calls for Research on IRBs and Behavioral and Social Science

Rob Townsend kindly alerted me to the Winter 2008 issue of AAHRPP Advance, published by the Association for the Accreditation of Human Research Protection Programs, which features the lead article, "IRBs and Behavioral and Social Science Research: Finding the Middle Ground." The article concedes that "many behavioral and social scientists feel constrained by a system that seems tilted toward biomedical research and, therefore, neither understands nor reflects their concerns." And it reports the interest of Drs. J. Michael Oakes, "an author and frequent lecturer on IRB review of behavioral and social science research," and Howard Silver, Executive Director of the Consortium of Social Science Associations, in addressing some of those concerns.

Oakes and Silver "encourage researchers to investigate scientifically IRB oversight of behavioral and social science research. Such research could determine whether IRBs are consistent in applying the federal regulations, whether IRBs are taking advantage of the flexibility that’s built into the regulations, and whether relationships between IRBs and social scientists are less strained on campuses that have separate IRBs to review behavioral and social science research. The resulting data could shed light on ways to relieve tensions between these two groups."

That sounds good, but it's a bit disappointing that the article does not acknowledge the considerable research already completed on this topic, much of it already cited on this blog. Nor does it remark on the curiosity that IRB oversight has continued for four decades without anyone knowing if it does any good.

Moreover, the article states as fact some beliefs that should be investigated with just the sort of research it calls for. I hope Oakes, Silver, and AAHRPP will allow data to challenge some of their own presuppositions.

Here are some questions that could be answered by further research.

1. When did IRB review of social science go bad, and why?

AAHRPP thinks it already knows the answer to this one. The article claims that "Tensions began building in the late 1990s in response to increased government scrutiny of research involving human participants," and that "the regulations have not changed. What’s new is their enforcement and, in many instances, that enforcement is overdue." I am at work on a history of IRB review of social science and humanities research, and I think more research can challenge this view.

The first part of the AAHRPP claim is doubtful; social scientists have protested IRB regulations since 1966, and tensions have waxed and waned since then. If we take this longer view, then the assertion about the immutability of the regulations is wrong; the regulations--first promulgated in 1974--changed twice, in 1981 and 1991. And the 1991 revisions greatly expanded the reach of IRBs. The 1981 regulations exempted survey, interview, and observational research unless it “deals with sensitive aspects of the subject’s own behavior, such as illegal conduct, drug use, sexual behavior, or survey or interview procedures is use of alcohol" and if "the subject’s responses, if they became known outside the research, could reasonably place the subject at risk of criminal or civil liability or be damaging to the subject’s financial standing or employability. The 1991 regulations, in contrast, eliminated the "sensitive aspects" clause and added potential harms to reputation to the list of triggers for IRB review. These changes were made over the objection of social scientists. And they set the stage for the conflict of the 1990s and today.

The claim that "what's new is [the regulations'] enforcement," is only half-true. Also new is the guidance issued by OPRR/OHRP since 1995 that reversed previous policies.

Finally, the article claims that "enforcement is overdue." Really? What errors did social scientists commit in the 1980s--a decade of relatively light regulation?

2. Why do IRBs sometimes delay or prohibit social science research?

Dr. Oakes states, “IRB members are not those folks who are looking to thwart your study. They are peer researchers who have a job to do.” But clearly some IRB members are looking to thwart studies, or else studies wouldn't get thwarted as often as they do. The question is how many IRB members do this, and why.

One part of this question concerns membership. Oakes's claim that IRB members "are peer researchers" depends on an odd definition of peers. In the NIH peer review process, for example, proposals are reviewed by study sections whose members are chosen for their expertise. The NIH's Center for Scientific Review requires, among other things, that

"* Expertise is the paramount consideration when developing/updating a study section roster.

"* Each scientific area reviewed by the study section needs appropriate expert representation."

IRBs theoretically must include experts on each type of research reviewed, but Oakes knows as well as I that this requirement is often ignored. Additional research might indicate how often a researcher faces an IRB with no expertise in the methods under review.

Then, of course, some IRB members are not researchers at all, but the "one member whose primary concerns are in nonscientific areas" required by the regulations. As Laura Stark's dissertation suggests, these members can be particularly undisciplined in their meddling.

3. What types of research now fall subject to IRB review?

Like PRIM&R, AAHRPP thinks that IRBs review only two kinds of scholarship: biomedical research, and something called "behavioral and social science research." The article states, "AAHRPP’s Founding Members, Board of Directors, Council on Accreditation, and Supporting Members all include representatives of organizations engaged in behavioral and social science research."

This statement suggests the fallacy of the undistributed middle term:

* Ehnographers are represented by organizations engaged in behavioral and social science research.
* Organizations engaged in behavioral and social science research have a voice in AAHRPP.
* Therefore, ethnographers are represented by organizations that have a voice in AAHRPP.

The second premise is "undistributed," since it is not true that all organizations engaged in behavioral and social science research have a voice in AAHRPP.

Here's a counter example:

* Countries in South America, Africa, and South Asia are not in North America or Europe.
* Countries from parts of the world other than North America and Europe are permanent members of the UN security council and the G-8.
* Therefore, countries in South America, Africa, and South Asia are permanent members of the UN Security Council and the G-8.

In committing this fallacy, AAHRPP lumps together a dozen or more scholarly disciplines--each with its own history, methods, and ethics--into a single category: "behavioral and social science research." The AAHRPP website does not list the disciplinary affiliations of members of its Board of Directors, Council on Accreditation, or list of site visitors, but if there's a journalist, historian, or folklorist in the lot, I'll be surprised.

To take the example I know best, oral historians do not expect psychologists, social workers, or education researchers to understand or represent their interests. AAHRPP (like PRIM&R) should find out how many disciplines are now subject to review, and include representatives from all of them.

4. What models of ethical review exist, and what models might we imagine?

The article asks "whether relationships between IRBs and social scientists are less strained on campuses that have separate IRBs to review behavioral and social science research." But that is only one of several alternative systems in place on various campuses. For example, Macquarie University delegates ethical review to a number of subcommittees with special expertise in certain fields. And the University of Pennsylvania allows researchers using some social science methods to forego "a fixed research protocol." And we can imagine even more models, some of which would require redrafting present regulations, others of which might not.

I appreciate AAHRPP's call for research, and I hope it agrees that research is most valuable when the answers are not predetermined.

Historians Flood OHRP With Comments

In response to the October 2007 announcement in the Federal Register calling for comments about the existing guidance on expedited review, oral historians and their allies have flooded OHRP with complaints about IRB review of oral history and requests for an unambiguous exemption.

As the original announcement noted, comments sent in response to Federal Register notices are a public record. Mr. Glen Drew and Ms. Toni Goodwin of OHRP kindly sent me copies of all 65 comments on expedited review. Of these, 38 commented on oral history or folklore, with all but one of those seeking exclusion for such research.

The comments came from a wide range of scholars. University historians ranged in rank from graduate students to chaired professors. Non-university historians included those working for federal and state agencies and for private companies. Historians of science and medicine—among those most familiar with the medical research that led to the current regulatory scheme—were particularly vocal. The American Historical Association weighed in against IRB review—reversing its 1998 stance—as did the American Association for the History of Medicine, the American Folklore Society, and the Society of American Archivists. Scholars in sociology, English, psychology, medicine, and American studies also called for oral history's exclusion from IRB jurisdiction, as did one IRB chair.

These scholars' complaints about IRB review of oral history will be familiar to those who have followed this controversy. Many noted IRBs' demands that scholars submit questions to be approved in advance, a practice that outrages oral historians who pride themselves on their ability to improvise questions in response to their research and the stories they hear.

Of the 38 comments, only one did not condemn IRB jurisdiction over oral history. Yet even that comment, by Claytee White of the Oral History Research Center at UNLV, does not endorse IRB review as practiced at most universities. Rather, White notes that she spends only 15 minutes clearing each project, suggesting blanket approval for her work rather than the project-by-project review that is the focus of most complaints.

The comments acknowledge that "memory can be painful," but they also point out that "historians are professionally obliged to ask our interview partners probing questions independent of the benefit or harm for the interviewee." Because they do not expect IRBs to understand this, historians—and the IRB chair—suggest that other oral historians would be better equipped to judge oral history projects than IRBs with few or no historians as members.

The scholars do not think IRB members mean-hearted, just hopelessly unfamiliar with the practice of oral history. "University IRBs do not have the necessary background to appreciate that oral history research is different from other research involving human subjects," wrote one. Nor do they expect this to change. A historian of medicine notes that "as long as IRB members are active, overworked faculty volunteering their time, they will be unable to track the nuances of a style of research they see very rarely."

For this reason, the historians do not seek modification of the current system, but an unambiguous removal of oral history from IRB jurisdiction. At least 14 comments, including the American Historical Association's, endorsed the 2006 recommendation of the American Association of University Professors "that research on autonomous adults whose methodology consists entirely in collecting data by surveys, conducting interviews, or observing behavior in public places, be exempt from the requirement of IRB review—straightforwardly exempt, with no provisos, and no requirement of IRB approval of the exemption."

As I noted in my own comment, the present guidance was adopted in 1998 in response to six comments about oral history, one of which included the American Historical Association's endorsement of oral history's inclusion on the list of methods eligible for expedited review. Now that the AHA and 36 others have called for the wholesale exclusion of oral history from IRB review, I hope OHRP will be as responsive as its predecessor was a decade ago.

Here are excerpts from the comments on oral history in PDF form.

Ethics Yes, IRBs No

The blog "Law and Letters" features a posting from "Belle Lettre" entitled Venkatesh's Gang Leader For a Day and IRBs. The author asks whether the ethical problems raised by Sudhir Venkatesh's book form an argument for IRB review of ethnography.

As the comments make clear, however, Lettre has uncritically equated IRB-approved research and ethical research. IRB critics do care about research ethics, but we question whether IRB review is either necessary or sufficient to ensure adherence to scholarly ethics and the law.

Must Employees Consent?

Two recent items concerning IRBs and studies of job performance caught my eye.

The more prominent item was a December 30, 2007, New York Times op-ed, "A Lifesaving Checklist" by Dr. Atul Gawande. Gawande describes efforts by the Johns Hopkins University and the Michigan Health and Hospital Association to offer five-step checklists to intensive-care providers, to help them remember to wash their hands and take other steps to avoid infection. OHRP shut the project down, even though it had been ruled exempt by the Hopkins IRB. (So much for local control.)

Although I try to keep this blog focused on issues concerning the humanities and social sciences, and to avoid issues concerning strictly medical research, I was struck by OHRP's determination that "the subjects of the research were both the healthcare providers at the participating ICUs and their patients." Are healthcare providers really human subjects in a case like this?

A similar question arose in a non-medical setting a few weeks later. In her pseudonymous blog on Education Week, eduwonkette complained that New York City teachers were being studied without their consent and without the approval of an IRB. Since eduwonkette herself concedes that "teachers did not need to consent in this case, as they are government employees and their employers can collect whatever data they want," I don't understand why she thought an IRB should be involved. But it does suggest that the question of studying employee effectiveness is not limited to medical contexts.

Poking around in my research notes, I find that at the July 1977 meeting of the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, Dr. Karen Lebacqz raised just this issue as a hypothetical:

"Suppose you want to do a study that involves changing the delivery of services in a mental institution ward. The patients in the institution are the ones that you want to study in terms of what the impact is on their recovery rates and so on. The staff who work on that ward will also be very directly impacted by the research. Do they constitute subjects, even though the design and the purpose of the research is not set up to study the impact on them of what you are going to do?"

While Albert Jonsen initially suggested that the staff would constitute subjects, other commissioners disagreed:

Lebacqz. “There is a very direct impact on the licensed staff.

Ryan. “Then the administrator can force it on them as a condition of employment.

Turtle. “Yes, I was going to say, that is the whole key, it is a condition of employment. Many things that, while I am a great libertarian here, lots of things I ask the people who work for me to do, I don’t ask them for their informed consent.”

[National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, Transcript, Meeting #32, July 1977 (Bethesda: The Commission, 1977), 133-135.]

After this exchange, the commission moved onto another subject, suggesting a consensus that employees are not human subjects whose informed consent is required. In its recent determination letters on the checklist project, OHRP has ruled the opposite.

I don't expect the compliance officers at OHRP to be familiar with the commission's deliberations, and I fault the commission for not defining human subjects more carefully. But it's worth noting how far OHRP has strayed from the commission whose work nominally informs its decisions.

How IRBs Decide--Badly: A Comment on Laura Stark's "Morality in Science"

Laura Stark's recent essay in Law & Society Review led me to her 2006 Princeton University dissertation, "Morality in Science: How Research Is Evaluated in the Age of Human Subjects Regulation." The heart of the dissertation is her description of the workings of three university IRBs--one in a medical school and two at universities without medical schools--based on recordings of their meetings and her direct observation of the IRBs at work. It makes for fascinating reading, and I applaud Stark for her achievement even as I disagree with her conclusions.

Stark claims to be neutral about IRBs' ability to perform their stated task: protecting the subjects of research. She writes, "My goal is not to judge the 'fairness' and 'effectiveness' of IRBs myself." (7) And she correctly notes that the ethical acceptability of an IRB-approved project is a "social truth," not an empirical one. (244) But her tone is generally sympathetic to the IRBs. For example, she writes that IRBs' "forms of evaluation provide directed, tangible ways for board members to carrying out their review process, given the practical difficulty of applying unmediated, abstract ethics principles," making the IRB members sound like heroes who have achieved a workable system against the odds. (186)

Indeed, in some cases she reports, IRBs seem to be doing some good. For example, a physiologist and a nurse had a fruitful debate about the need for a quick medical screening of subjects in an exercise study (197-200). That's an example of an IRB with multiple experts on a single type of research--something I hope is reasonably common in much biomedical research. But most of Stark's observations are distressing in ways I don't think she appreciates. Here, then, are some of the actions she observed, along with my reasons for finding in them an indictment of the IRB system as presently run.