IRBs vs Law and Society

The Law and Society Association (LSA) has posted “The Impact of Institutional Review Boards (IRBs) on Law & Society Researchers," a 2007 report by the association's Membership and Professional Issues Committee.

In the spring of 2007, the committee put out a general call for comments from association members, receiving 24 replies. Committee members also interviewed association members, taking special interest in those who had served on IRBs. This produced some accounts of frustrating encounters that led to the destruction of research, especially cases involving research on crime and punishment, a special concern of the association.

One particularly sad story came from "Respondent 019":

I knew when completing my [questionnaire] that I would have some difficulty getting my topic approved because it related to a protected population. I completed the [questionnaire] honestly and accurately and finally heard back that I had to make substantial revisions to my proposal. Not only did I take all of the IRB’s recommendations for review into account when completing a second questionnaire, but I even had frequent contact with the compliance coordinator and the secretary to make sure I was doing everything correctly. I submitted a second [questionnaire] that covered all the areas that caused problems with my first proposal. Finally, I heard back, and I was rejected on a whole other set of criteria that IRB never mentioned when they rejected me the first time. Finally I ended up changing my topic enough so that I no longer had to deal with IRB because they had pushed back the start date of my research so far that I couldn’t risk not getting approved again.

Other accounts describe IRB blocakge of research concerning mental health facilities that house sex offenders and prisons in the United States and Turkey.

The report concludes with three sets of recommendations:

First, it wants to restrict IRB jurisdiction: "the LSA should strive to minimize the scope of the IRBs regulations over non-behavioral studies and make the procedures of approving behavioral studies as smooth and expedient as possible."

Second, the report calls for a program of research and education, ranging from conference panels and publication to a statement of best practices for research. The goal is "a nuanced and contextual view of the IRB process, one that moves away from hard and fast 'rules' for most social science research, allowing for optimal protection of human subjects without inhibiting research goals."

Finally, the report claims that "the most successful IRBs (in terms of 'customer satisfaction') are those with a decentralized 'sub-board' system," citing UCLA and Macalester as examples. I find this unpersuasive, given that the report is based on part of the work of Jack Katz of UCLA, who seems pretty unsatisfied with his IRB.

While not directly inconsistent, these three recommendations are in some tension with each other, as well as with the evidence presented in the report and positions taken by the LSA. The report's collection of horror stories--unrelieved by any reports of IRB contributions to ethical research--calls into question the propriety of any IRB review of social research, and the report's analysis suggests that there is no legal requirement for such review. Put together, these sections of the report support Katz's conclusion that "the optional decision to push all ethical review of social science and humanistic research through a prior review sieve is not only massively inefficient, it is also counterproductive where risks are most serious." ["Toward a Natural History of Ethical Censorship," Law & Society Review 41 (December 2007), 807.]

By contrast, the calls for additional research, a more flexible IRB process, and decentralization all resemble the stance of Felice Levine and others, who advocate working within the current system.

In particular, the report raises the question of LSA's position on the scope of the "behavioral research"mentioned in the National Research Act and other forms of interactions with human beings. "There has been no mention, or expressed intention regarding human subject research in law and the social sciences," the report notes, "and clearly not all research in law and society is 'behavioral.'" If this is the case, then many of OHRP's policies and interpretations of the Common Rule lack a statutory basis. Yet in December 2007, five months after the report was presented to the association, the Association signed onto Levine's comment to OHRP, in which she wrote of "social and behavioral sciences (SBS)," conflating the very categories the report wishes to keep distinct.

The July 2008 newsletter of the association mentions the report, but it does not state whether the association has accepted its committee's position or taken any action in response. I hope the LSA will continue work in this field, and that it will consider further the question of whether all survey, interview, and observational research should be made subject to a law governing "behavioral research."

OHRP Continues Indiana-Bloomington Investigation

In September I reported on the OHRP investigation of Indiana University-Bloomington, ably covered by the Bloomington Herald-Times. Along with several stories, an editorial, and at least two op-eds, the newspaper posted heavily redacted copies of the OHRP letter and the IU reply.

On October 3, I submitted a Freedom of Information Act request for the unredacted OHRP complaint. This week, I received a reply, dated November 17, stating that "the subject matter of your request is the subject of an open and ongoing investigation. Release of any additional information at this time could reasonably be expected to interfere with ongoing proceedings." Hence, I received no information. We still don't know what Indiana-Bloomington did to bring down the federal hammer, or if the hammer will strike again.

Meanwhile, the crackdown has disrupted research, especially for social scientists. As the Herald-Times reported on October 8 (Nicole Brooks, "IU Research Oversight Office Has More Staff, But Projects Still Delayed"),

The need to “attend to the compliance issue speedily” led to Bloomington researchers using the same proposal forms as IUPUI faculty, according to [Research Affairs Committee chairman Stephen] Burns. These forms are designed for medical research, and are “more complex than needed,” especially for social science researchers.

This has caused some faculty — and students — to not bother with some research projects, Burns said. And some students are changing their thesis topics so they don’t include human subjects research, he said.

Before the compliance issue came into play, when Bloomington campus faculty used their own form and not IUPUI’s, this was a problem, Burns said. Research topics are becoming more and more diverse, and the divide between what information is necessary for different kinds of research is widening, he said.

The upshot is that faculty and students at a major research university are abandoning their research because of secret allegations against their university's administration. It's all very well for OHRP to claim (as Ivor Pritchard did at the October SACHRP meeting) that OHRP enforcement actions are rare. But even the rare crackdown, if as severe as this one, is enough to have IRBs nationwide quaking in fear, and putting regulatory compliance above all other considerations.

Can IRBs Handle the Nuances of Anthropological Ethics?

In a November 14 essay in the Chronicle of Higher Education ("New Ethical Challenges for Anthropologists"), Carolyn Fluehr-Lobban reports on the work of the Commission on Engagement of Anthropology with the U.S. Security Community, of which she is a member. The commission was established by the American Anthropological Association in 2006, in response to complaints that anthropologists had acted unethically by participating in the Department of Defense's Human Terrain System and other national security programs.

Fluehr-Lobban offers a carefully nuanced discussion of the questions of secret research and research that harms its subjects. Both require subtlely. As she notes,

even in agencies well known for their secrecy, like the Central Intelligence Agency, such terms as "transparency" and "disclosure" have become more common, and secrecy less easy to define; "classified" government documents can be accessed by scholars and journalists, while truly top-secret materials deal with intelligence research rather than anthropology. Moreover, scholars may not be able to discern whether their work contains secret material when projects are compartmentalized, and when their contribution is only a segment of a project whose wider mission is unknown. In short, anthropologists who provide "subject matter expertise" may not know the direct or indirect impact of their engagement.

Likewise, harm can be hard to predict, or even define. As Fluehr-Lobban explains,

Anthropologists have been deployed to Iraq and Afghanistan as part of Human Terrain Teams embedded with combat troops. Part of the work they do unquestionably causes harm to some people -- but it may prevent harm to others. In addition, we know so little about what the teams do, or the projects they are part of, that objective evaluation is impossible at present.

What does this tell us about IRB review of social science research? Well, that's not clear either.

On the one hand, the commission asks anthropologists to "be assured that adequate, objective review of the project has been conducted, ideally by external reviewers."

On the other hand, Fluehr-Lobban suggests that such review would require expertise not found on the typical IRB. She concludes,

Ideally, decision making occurs in a group process where the relevant disciplinary, cultural, and government-agency stakeholders are at the table . . . Consultation with professionals in related disciplines who have been grappling with issues of engagement -- for example, psychologists who have debated their role in identifying what would constitute "soft torture" and their alleged involvement in interrogations in Abu Ghraib and Guantánamo -- is also recommended, as well as with those who have been historically engaged without serious controversy -- for example, political scientists working as consultants on terrorism for defense and intelligence agencies.

Perhaps the best advice will come from one's own disciplinary colleagues. To that end, a group, "Friends of the Committee on Ethics," may be set up to offer informal, private advice about research ethics. As we think through the various issues that secrecy and doing no harm demand of us, such a committee will have an important role to play in helping define increasingly complex anthropological practice.

These latter recommendations do suggest a role for interdisciplinary consultation, but they are no endorsement for the current IRB system, which makes no provision for assuring review by relevant stakeholders, professionals who have grappled with the issues, or disciplinary colleagues. I hope that Fluehr-Lobban's commission will explore the implications of its findings for the appropriate mechanisms of ethical review.

(Thanks to the Research Ethics Blog for bringing this to my attention.)

Comments Oppose New Regulations on Training

As reported by the Chronicle of Higher Education, OHRP recently released the replies it had received in response to its July call for comments on education and training requirements. I thank Chronicle reporter David Glenn and OHRP associate director Michael Carome for supplying me with copies of the comments.

As I see it, the comments pose three main questions.

Chronicle of Higher Education on OHRP Training Comments

The Chronicle of Higher Education quotes your faithful blogger in "Scholars Mull Rules for Training in Research Ethics," by David Glenn, 4 November 2008.

The story concerns the eighty or so comments received in response to OHRP's July call for comments on education and training requirements. Glenn notes that by and large, the comments were skeptical about the need for new guidance, and particularly skeptical about regulations. As he reports, "the American Association for the Advancement of Science, the Association of American Universities, the Association of American Medical Colleges, and a consortium of large social-science organizations [all] said that before the federal government issues new rules, it should carefully study whether training actually improves researchers' conduct."

I will offer some of my own comments on the comments in coming posts.

Report from SACHRP, October 2008

Today I attended the public meeting of the Secretary's Advisory Committee on Human Research Protections (SACHRP). Most of the day's discussion concerned the slow pace with which SACHRP's recommendations over the years have been implemented in any way; some have been sitting for years without action. OHRP officials offered detailed explanations of the complexity of policy-making process and OHRP's lack of resources. While these issues help explain OHRP's neglect of questions important to the social sciences and humanities, today's discussion had little of direct importance to scholars in those fields.

Here are a few tidbits of potential significance.

What is the difference between guidance and regulation?

Christian Mahler, a lawyer with HHS's Office of General Counsel, explained that only regulations can be enforced; as a legal matter, institutions are free to ignore OHRP guidance documents. But he and other officials acknowledged that in practice, the difference may not be great. As acting OHRP director Ivor Pritchard conceded, "when we issue guidance . . . people look at every phrase, clause, use of punctuation to see what was meant by OHRP.” He noted that institutional officials may believe the safest course is to comply with all OHRP guidance.

Pritchard said that OHRP tries to differentiate, when drafting guidance, between "must" statements (that indicate OHRP's interpretations of the regulations) and "should" statements that can be ignored if an institution has good reason. He also stated that the best course might be to issue guidance documents that offer multiple ways to comply with the regulations, though it's not clear that OHRP has ever issued such guidance.

Finally, Mahler pointed to the Office of Management and Budget's 2007 "Final Bulletin for Agency Good Guidance Practices.” That bulletin notes that

The courts, Congress, and other authorities have emphasized that rules which do not merely interpret existing law or announce tentative policy positions but which establish new policy positions that the agency treats as binding must comply with the [Admistrative Procedure Act]’s notice-and-comment requirements, regardless of how they initially are labeled. More general concerns also have been raised that agency guidance practices should be better informed and more transparent, fair and accountable. Poorly designed or misused guidance documents can impose significant costs or limit the freedom of the public.

Despite that last sentence, the bulletin seems designed more to limit regulation of economic affairs than to safeguard the freedom of the public. Still, if its approach were followed, OHRP might not be able to get away with so many arbitrary decisions.

What is research?

In February 2007, then-OHRP director Bernard Schwetz told the New York Times that OHRP would, by the end of 2007, issue guidance on what is and is not research under the regulations. OHRP is almost a year overdue in keeping that promise, but apparently it is still at work. Pritchard noted that "We have been working on a guidance document on the definition of ‘research’ for several years now.” Pritchard did not indicate how far along that process is, or whether OHRP will solicit public comment before promulgating that document.

What is an ideal consent process?

In her presentation, Elizabeth Bankert, co-chair of SACHRP's Subpart A Subcommittee, addressed the problem of IRB insistence on long, complex consent forms. She noted that while the regulations requiring consent forms have not changed since 1991, IRBs have been demanding ever more detailed forms, and have gained a reputation for "wordsmithing" and "nit-picking." Not only does this erode investigator trust and respect for IRBs, but it also "diminishe[s] the consent process for subjects."

Bankert--drawing on work by subcommittee member Gary Chadwick--challenged the presumption that "the form must contain every piece of information and in the same detail as required in the consent process," which leads to such long forms. She called upon OHRP and FDA to endorse the use of shortened consent forms: about 3-4 pages for a clinical trial, and just one for "surveys, etc."

Bankert offered these suggestions as a way to decrease some burdens on IRBs and investigators, but she then suggested that IRBs would still need to review the consent process. She did not explain how the same incentives for nit-picking wouldn't lead to endless fretting about the consent process, rather than the consent form. As SACHRP member Jeffrey Botkin pointed out, IRBs and investigators turn to long forms out of self-preservation. And as David Foster noted, FDA inspectors and AAHRPP accreditors have demanded ever longer forms. Before Bankert and Chadwick can address the problem of silly forms, they need to understand the system that produces them.

Predictably, almost all the discussion of consent forms centered around clinical trials. For example, Bankert offered sample executive summaries for a drug trial and a request to store blood or tissue for future research, but no comparable document for a social science project. And one committee member, expressing her concerns, forgot to speak of "human subjects" and instead started talking about protecting "patients." Given the complexity of the issue, and the dominance of SACRHP by medical researchers and bioethicists, I would expect that any change will be designed for clinical trials, and then imposed on social researchers.

Menikoff to Head OHRP

The Department of Health and Human Services has announced the appointment of Dr. Jerry Menikoff as the director of the Office for Human Research Protections (OHRP). Menikoff holds degrees in medicine, law, and public policy, and has served as a law professor and public official.

A quick glance at Dr. Menikoff's CV suggests that while he has published extensively on the ethics of medical experimentation, he is written little on questions of the regulation of social science. The exception is his article, "Where’s the Law? Uncovering The Truth About IRBs and Censorship," Northwestern University Law Review 101 (2007): 791-799, which I described in a January 2007 blog entry. In that article, Menikoff suggested that the 46.101 exemptions were sufficient to avoid any conflict with the First Amendment.

Menikoff does understand that many IRBs have abused social researchers. "There are surely too many instances in which IRBs and others fail to understand, and properly administer, the regulations," he writes. (793, n. 9) Later, in a response to an Inside Higher Ed article about this blog, he wrote, "As to social science and behavioral studies, I do support reforms to relax the rules somewhat, though the claims that the system constitutes censorship under the U.S. Constitution are overkill."

It is not clear, however, that Menikoff understands how rarely the 46.101 exemptions are applied, and how much responsibility OHRP bears in their disappearance. And in his Northwestern piece, Menikoff praises OHRP for "concluding . . . that much of the work performed by oral historians, in sitting down with people and getting information from them, does not fall within the category of doing 'research.'" (798) He seems not to know that OHRP so contradicted itself that its pronouncement had no effect on most universities.

If Menikoff uses his position to revive 45 CFR 46.101, to restore the agreement with oral historians, and to correct other lapses between the regulations as written and as enforced by OHRP, his accession to the OHRP directorship will be great news not only for scholars in the social sciences and humanities, but also to participants in medical research, since resources can be reprogrammed for their protection. Under the leadership of a lawyer, perhaps OHRP will begin to obey the law.

45 CFR 46.101 Is Still Dead

At the September AAAS meeting, I learned of the June 2008 report, "Expedited Review of Social and Behavioral Research Activities," put out by the Social and Behavioral Research Working Group, Human Subjects Research Subcommittee, Committee on Science, National Science and Technology Council. The brief report (11 pages, including a great deal of white space) offers thirteen "brief illustrative examples" of "social and behavioral research activities [that] qualify for expedited review . . . assuming that they also meet the standard of minimal risk."

While presenting itself as an effort to help researchers, administrators, and reviewers "avoid needless misunderstanding and delays in the review process," the document threatens to add misunderstandings and delays by suggesting review for many projects that should be exempted from IRB oversight.

Here are the thirteen examples, to which I have added numbers for discussion purposes. The categories listed at the end indicate the reasons the working group believes the examples are eligible for expedited review, based on the 1998 list of categories.

AAAS Hears Clashing Views on IRBs

The American Association for the Advancement of Science (AAAS) has posted a summary of a September 22 meeting on IRBs and the social sciences, in which I took part.

As the press release, "AAAS Meeting Explores Ways to Improve Ethics Panels that Oversee Social Science Research," notes, "from both researchers and administrators alike, there was general agreement that the system can and should work better," but participants disagreed about what a better system would look like.

Social researchers recounted some horror stories, the most vivid of which was Gigi Gronvall's:

Gigi Gronvall, a senior associate at the Center for Biosecurity of the University of Pittsburgh Medical Center, recalled her efforts to do a survey of scientists who do dual-use research in biology, such as work on viruses that might have military as well as civilian application. The IRB at Johns Hopkins University, where the biosecurity center was located at the time, asked Gronvall to give a three-page consent form to all those she interviewed. It included a warning that the respondent, in agreeing to answer Gronvall's questions, ran the risk of being investigated by government agencies, being "exploited by hostile entities," or even being kidnapped.

The IRB's members, Gronvall said, "were totally over-identifying with my subject population." The result was a six-month delay in the survey project, during which Gronvall almost lost her funding. "The IRB should be on your side," she said. "That's not how I felt during this."

Gronvall said that blocking or delaying research on a controversial topic can mean that it will be explored only in the news media, without any IRB-style protections for those being interviewed.

In addition to such personal experiences, some participants warned of fundamental flaws in the IRB system. I described the origins of IRB review as the work of physicians, psychologists, and bioethicists who had no understanding of the methods and ethics of social scientists, and assembled no evidence of widespread abuse by them. Joan Sieber, editor of the Journal of Empirical Research on Human Research Ethics, suggested that the horror stories are not abberations, but common.

IRBs had their defenders, particularly from scholars and consultants with a background in medicine. Anne N. Hirshfield, associate vice president for health research, compliance and technology transfer at the George Washington University, claimed that "IRBs can only work if there is mutual attention to a common goal—conducting ethical research that protects the rights and welfare of participants." But she also argued that "it is the obligation of the P.I. to know what the IRB needs and to stage the argument. Give the citations to show that your work is not risky." In other words, researchers must prove a negative, while IRBs are free to conjure up scientist-kidnappers.

Perhaps the most impressive presentation was that of Janet DiPietro, associate dean for research at Johns Hopkins University's Bloomberg School of Public Health, who described her efforts to reform the IRB there. If all IRBs were managed by someone as thoughtful as she, we'd have far fewer complaints. But there aren't enough DiPietros to go around, so her achievements do not make a good case for granting administrators nationwide such power.

The press release concludes with a statement from Mark Frankel, staff officer for the AAAS's Committee on Scientific Freedom and Responsibility:

Some Committee members believe that the balance is awry because IRB's are imposing unwarranted and arbitrary demands on proposed research by social and behavioral scientists. This raises serious issues related to scientific freedom insofar as such actions lead to potentially valuable research that is inappropriately altered, unduly delayed, or not done at all.

While noncommital about specific policy recommendations, this recognition that IRB review is a threat to scientific freedom is in itself an important finding. I look forward to further AAAS efforts to explore this problem and contribute to its resolution.

Treeroom Trade

John Mueller kindly alerted me to Gautam Naik, "Switzerland's Green Power Revolution: Ethicists Ponder Plants' Rights," Wall Street Journal, 10 October 2008.

Naik reports that the government of Switzerland has required researchers to "conduct their research without trampling on a plant's dignity," based on an April 2008 treatise, The Dignity of Living Beings With Regard to Plants.

The treatise itself notes that the members of the Federal Ethics Committee on Non-Human Biotechnology could not reach consensus on most of the moral questions they considered, and some worried about over-regulation. Nevertheless, "The Committee members unanimously consider an arbitrary harm caused to plants to be morally impermissible. This kind of treatment would include, e.g. decapitation of wild flowers at the roadside without rational reason," though even then the committee members disagreed over why such decapitation is impermissible. Nor does the report lay out criteria for a "rational reason." What if it's fun to decapitate wildflowers?

[Disclosure: As a boy, I liked to knock the heads off dandelions with a stick. Guess I won't be getting a visa to Switzerland any time soon.]

Naik's article focuses on the threat to genetic research, but if there's anything that four decades of human subjects regulation has taught us, it's that government officials refuse to recognize distinctions between lab experimentation and social science. Perhaps we should expect restrictions on the observation, surveying, and interviewing of plants as well. In six months Washington's cherry blossoms will be out, and I pity all those poor trees, left helpless as hundreds of thousands of people come to gawk at their exposed genitals.