Friday, January 26, 2007

Still no evidence?

Twenty-six years ago today, on 26 January 1981, the Department of Health and Human Services published in the Federal Register its "Final Regulations Amending Basic HHS Policy for the Protection of Human Research Subjects." (PDF) The department rejected a proposal by several social-science organizations to "exempt all research utilizing legally competent subjects if that research involved neither deceit nor intrusion upon the subject’s person, nor the denial or withholding of accustomed or necessary resources." It justified this decision in empirical terms, stating that "despite some general comments that the regulations would impede social research, the Department has been presented no evidence that social science research that may present risks to subjects has been unduly hampered by the requirement for IRB review and approval" (emphasis added).

What's striking is the department's suggestion that evidence of undue hampering might have changed its decision on the regulations. Twenty-six years later, it's a good deal easier to find such evidence. See, for example,
  • Jennifer Howard, "Oral History Under Review," Chronicle of Higher Education, 10 November 2006.
  • AAUP, "Research on Human Subjects: Academic Freedom and the Institutional Review Board," (2006)
  • Robert L. Kerr, "Unconstitutional Review Board? Considering a First Amendment Challenge to IRB Regulation of Journalistic Research Methods," Communication Law & Policy 11 (2006): 393–447.
  • American Ethnologist 33, No. 4 (November 2006)
  • Journal of Applied Communication Research 33, No. 3, (August 2005).
  • Patricia Loomis Marshall, "Human Subjects Protections, Institutional Review Boards, and Cultural Anthropological Research" Anthropological Quarterly 76 (Spring 2003): 269-285
Even scholars who have served on IRBs despair when confronted by all the stories of inappropriate review.

Is it time for the federal government to revisit its decision?

Sunday, January 21, 2007

The Language of Exemption

In an earlier posting, I asked

How many examples must critics accumulate before Sieber, Menikoff, and others will ask if these problems are not the fault of individual IRBs, but of the application of rules designed for experimental research to non-experimental research?

A correspondent comments:

I do not think 'experimental' versus 'non-experimental' gets at the problem you are trying to address. The rules are not really designed for experiments. For example, the 'Tuskegee Experiment', despite the name, was not an experiment.

I am not sure I am ready to concede this point, but the comment does highlight the need for language that precisely delineates the kind of research I and others would like to exempt from IRB review. I therefore present four proposals, made over the years, that would exempt qualitative interview research:

1. In 1979, the Department of Health, Education, and Welfare (HEW) announced planned revisions to 45 CFR 46 (the federal regulation requiring IRBs). Overall, the proposal would have greatly expanded the reach of IRBs, but one of the possibilities it considered would have exempted "survey activities involving solely product or marketing research, journalistic research, historical research, studies of organizations, public opinion polls, or management evaluations, in which the potential for invasion of privacy is absent or minimal." (Federal Register, 14 August 1979, PDF)

2. The announcement provoked outcry from a variety of scholarly organizations. The American Association for the Advancement of Science "resolved that the Association urge that further government regulation designed to protect the human subjects of scientific inquiry not be made applicable to studies that involve no more than the free exchange of information between adult subjects, competent to make their own decisions, and the scientist."
(American Association for the Advancement of Science, "AAAS Resolution: Protection of Human Subjects of Research," Center for the Study of Ethics in the Professions at IIT, 14 August 1979.

3. In 1980, several groups endorsed the American Council on Education's demand that the regulations include the proviso,
“These regulations do not apply to research using legally competent subjects that involves neither deceit nor intrusion upon the subject’s person nor denial or withholding of accustomed or necessary resources.”
[J. W. Peltason, "Comment on the Proposed Regulations from Higher Education and Professional Social Science Associations," IRB: Ethics and Human Research 2 (February 1980), 10, JSTOR ]

4. I haven't traced all the proposals since then, but in 2006, the American Association of University Professors recommended "that research on autonomous adults whose methodology consists entirely in collecting data by surveys, conducting interviews, or observing behavior in public places, be exempt from the requirement of IRB review—straightforwardly exempt, with no provisos, and no requirement of IRB approval of the exemption." (AAUP, "Research on Human Subjects: Academic Freedom and the Institutional Review Board," 2006)

Since any of these proposals would get historians off the hook, they all strike me as improvements over the status quo. But there are significant differences among them.

The HEW proposal is the most limited. By listing by name only a few disciplines to be exempted, it would have kept IRB review of disciplines, such as folklore, even further removed from the biomedical and behavioral research that was the target of the National Research Act of 1974. Far better to list methodologies, such as interviewing. Moreover, from the recent debate over the meanings of "generalizable" and "in general," we can now see that the proposal would have left unresolved the question of who gets to determine when "the potential for invasion of privacy is absent or minimal."

The 1979 AAAS proposal and the 2006 AAUP proposal are quite similar to each other, in that they both stress the right to talk to competent adults. However, the AAAS is less precise in its language. Many researchers who have found themselves subject to IRB review do not consider ourselves "scientists." Moreover, the AAUP recognizes the need for self-exemption. If only IRBs (and their designees) can exempt researchers from IRB review, then researchers will remain subject to the caprice of the IRBs.

A comparison of the AAUP proposal to that of the ACE raises an interesting issue. Since the AAUP's exemption targets "methodology [that] consists entirely" of surveys, interviews, and observation, it would not encompass research that withheld resources, such as the penicillin denied the subjects of the Tuskegee study. But what about deception, which can be part of surveys and interviews? The AAUP's report is ambiguous on the issue. On the one hand, it raises the specter of Milgram's obedience experiments (which relied on deception) to suggest that "an across-the-board exemption for all social science research is arguably overbroad." On the other hand, it cites Keith Humphreys and Michelle L. Brandt to note that we have no empirical evidence that IRBs protect human subjects. Thus, even if we concede that research using deception raises significant ethical problems, that's no reason to assume that IRB review can identify or resolve those problems.

The AAUP's proposal still needs a bit of tweaking on this question, but it is very close to a plausible amendment to the current regulations.

Friday, January 19, 2007

Special Issues

One of my hopes for this blog is that it will bring together viewpoints from a variety of disciplines. To this end, I am grateful to have been alerted to two recent journal issues with multiple articles about IRB review:

* Journal of Social Distress and the Homeless, Volume 15, Number 1 (January 2006)
* American Ethnologist, Vol. 33, No. 4 (November 2006)

I plan to start reading and blogging these issues shortly.

Inside Higher Ed

This blog is five weeks old and already famous; see Paul D. Thacker, "Reviewing the Reviewers," Inside Higher Ed, 19 January 2007.

Thursday, January 18, 2007

John Mueller

John Mueller of the University of Calgary kindly alerted me to his page on Research Ethics, which offers a skeptical look at IRBs' utility.

Tuesday, January 16, 2007

IRB demands destruction of tapes

Kathy Staley of Appalachian State University writes on H-Oral:

I'm currently doing an oral history project on my university's LGBT population and have run into some interesting challenges. Older people (50+ years) requiring pseudonyms. My IRB approval required several hoops to be jumped through (such as no 3rd party names are to be listed in the transcripts and my tapes are to be destroyed) to ensure strict confidentiality. I'm completely ill about this and although I disagree with IRBs being required for oral histories philosophically and my particular requirements, I want to graduate and my thesis adviser required me to do it too.

There is a case to be made for blanking out the names of third parties; I regard this as an ethical question yet to be sufficiently explored. The National Human Research Protections Advisory Committee's "Clarification of the Status of Third Parties When Referenced by Human Subjects in Research" is not particularly helpful. It states that researchers don't need informed consent from third parties, but does not explain how they should regard those parties' privacy.

The demand to destroy tapes, in contrast, is just wrong. Tapes can be sealed for decades; historians are patient folks. Here an IRB has forced a historian to violate her professional ethics.

Menikoff, "Where's the Law?"

Another paper from the 2006 conference, "Censorship and Institutional Review Boards," has appeared on SSRN: Jerry Menikoff, "Where's the Law? Uncovering the Truth About IRBs and Censorship."

Most of the article repeats claims made by Joan E. Sieber, Stuart Plattner, and Philip Rubin in their 2002 essay, "How (Not) to Regulate Social and Behavioral Research," Professional Ethics Report (Spring 2002), 1-4. They argue that any abuses by IRBs are the fault of individual IRBs, not the regulations governing them. They conclude, therefore, that "IRBs and researchers can return to true interpretation of the Belmont Report under the Common Rule, if they make use of the flexibility it offers for reasonable interpretation of its requirements." Similarly, Menikoff believes that "both the regulators, and the regulations they enforce, reflect a system that, properly understood and implemented, already imposes a fairly minimal burden on individual researchers in the area of social and behavioral research."

These arguments suggest that inflexible, inappropriate impositions by IRBs are anomalies in a fundamentally sound system. But for several years now, complaints have poured in from scholars in anthropology, communications, folklore, history, law, and sociology; from universities across the country; from researchers at the undergraduate, graduate, and faculty level. And these are just the most outrageous cases; the August 2005 issue of the Journal of Applied Communication Research, devoted entirely to the question of IRB review, suggests that many researchers suffer in silence. To blame all of these problems on individual IRBs strikes me as unfair; the pattern suggests a problem in the system itself.

How many examples must critics accumulate before Sieber, Menikoff, and others will ask if these problems are not the fault of individual IRBs, but of the application of rules designed for experimental research to non-experimental research?

Menikoff's second argument is that while IRB review of the social sciences may be bad policy, "individual researchers are not subject to a burden rising to the level of constitutional concern." Perhaps not, but since Menikoff cites no case law on what does constitute censorship, it's hard to understand how he reached this conclusion. How many questions must a board turn down before you can call it a ban?

A more complete discussion of this question can be found in Robert L. Kerr, "Unconstitutional Review Board? Considering a First Amendment Challenge to IRB Regulation of Journalistic Research Methods," Communication Law & Policy 11 (2006): 393–447. Kerr concedes that the courts have yet to offer clear guidance, but he notes that

“In a variety of contexts” the Supreme Court has declared that “even though the governmental purpose be legitimate and substantial, that purpose cannot be pursued by means that broadly stifle fundamental personal liberties when the end can be more narrowly achieved.” . . . In the interest of addressing concerns arising from abuse in biomedical research, the IRB regulations reach so far as to prevent scholars from utilizing journalistic research methods that are fully protected in other contexts. The breadth of the regulations means that scholars are not able to conduct interviews at the time that they may be most available or most fruitful, because they generally must wait at least a number of weeks before their request for IRB approval can be processed. If an interview subject is no longer available after the wait, the delay imposed by the IRB might as well be a ban. Even when permission is eventually granted, the interview sought may be lost for good during the waiting period.

Since the Northwestern conference was held in April 2006, it's likely that contributors had not had a chance to read Kerr's thoughtful paper. I hope that future legal scholarship on this issue will build on his work. I would also like to hear some explanations of why no IRB has, to my knowledge, dared assert authority to review university journalism programs.

Wednesday, January 10, 2007

"Generalizable" revisited

One persistent question about the need for IRB oversight of oral history is whether oral history interviews come under the jurisdiction of 45 CFR 46, which defines research as a "systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge." Any project not "designed to develop or contribute to generalizable knowledge" is therefore excluded from the requirement for review.

In October 2003, Dr. Michael Carome, associate director of the Office of Human Research Protection, U.S. Department of Health and Human Services (OHRP), sought to answer that question in a discussion with representatives of the UCLA Office for Protection of Research Subjects. UCLA prepared an outline of that discussion, and Carome subsequently confirmed it in correspondence to Northern Illinois University. Carome offered descriptions of three hypothetical projects, explaining that one of them would be excluded from review, while the other two would require IRB approval. Because these were hypothetical, it was impossible to tease out the distinctions among them, so the guidance was unclear.

In 2004, however, OHRP offered a concrete example of non-generalizable oral history. OHRP staff, including director Bernard A. Schwetz, conducted a series of videotaped interviews with former members and staff of the the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. Recordings and transcripts of those interviews, entitled the Belmont Oral History Project, are online at

I recently asked OHRP if the project had been reviewed by an IRB, and received the following reply:

Date: Wed, 10 Jan 2007 09:39:42 -0500
Subject: RE: Belmont Report Oral History
Sender: "Nellis, Kevin (HHS/OPHS)"
To: "Zachary M. Schrag"
Cc: "ElHinnawy, Patricia C (HHS/OPHS)"

Dear Dr. Schrag,

OHRP determined that obtaining oral histories of members and staff of the National Commission did not represent research as defined at 45 CFR 46.102(d) because the activity was not a systematic investigation, nor was it intended to contribute to generalizable knowledge. This oral history activity was designed merely to preserve a set of individuals' recollections; therefore, this activity was not subject to IRB review. The interviewers had no specific training related to this activity; and those interviewed did not sign a consent document.

I am delighted that you found the interviews informative and helpful. If you have any more questions regarding the protection of human subjects in research or need clarifications on this response, please do not hesitate to contact OHRP.

Best Regards,

Kevin L. Nellis, M.S., M.T. (A.S.C.P.)
Public Health Analyst
Division of Education and Development
Office of Human Research Protections

Zach comments:

Based on this project, then, recorded interviews do not represent research as defined at 45 CFR 46.102(d) even when:

  • interviewers ask some standardized questions (several narrators are asked, for example, how much time they spent deliberating about social and behavioral, as opposed to biomedical, science).
  • interviewers ask prepared questions tailored to the narrators' individual experiences.
  • interviewers ask follow-up questions based on the narrators' replies to initial questions.
  • interviewers ask narrators to draw conclusions about the lasting significance of their work.
  • recordings and transcripts are posted on the Internet and labeled "Oral History Archive."
  • portions of the interviews are used in an interpretive work of history (in this case a video
    arguing that the 1970s "marked a turning point in how human research was controlled in the United States," and that the authors of the Belmont Report should be thanked "for their help in guiding our decisions far into the future").
  • the interviews appear as part of a website that draws on history to inform present policy ("Today, the Belmont Report continues as an essential reference for institutional review boards (IRBs) that review HHS-conducted or -supported human subjects research proposals involving human subjects, in order to ensure that the research meets the ethical foundations of the regulations").

Sunday, January 7, 2007

Nancy Janovicek offers a Canadian perspective

Historian Nancy Janovicek of the University of Calgary has written a thoughtful description of the problems faced by Canadian oral historians as they face ethical review by scholars "not well versed in historical methodologies." The article, "Oral History and Ethical Practice: Toward Effective Policies and Procedures," is slated to appear in the Journal of Academic Ethics, but it has already been posted at the SpringerLink website.

In Canada, institutional review boards are known as "research ethics boards" (REBs), and the Canadian equivalent of the Common Rule is the Tri-Council Policy Statment: Ethical Conduct for Research Involving Humans. But though the names are different, the effects are the same. Janovicek's complaints will be familiar to anyone who has followed the American debate over IRB review of oral history, but her account is particularly eloquent on two points.

First, she notes ethics boards' obsession with confidentiality and anonymity, despite the wishes of both researchers and subjects. When Janovicek told one narrator of this poicy, "a lesbian activist who has struggled to maintain lesbian visibility in her communinity replied, 'I like my name.'" (This point would have resonated better had Janovicek named the activist in this article.)

Second, Janovicek complains of policies that seek to prevent the exploitation of Aboriginal communities by requiring reseachers to "acquire written approval from Band Councils in order to interview members of the community." But, she asks, "what happens when leaders act as gatekeepers for research about less powerful people in their communities?"

Both of these arguments show how well-meaning policies crafted for medical research can strip power away from autonomous adults who are not only able to decide whether they want to speak to a historian, but are able to seize the opportunity to challenge existing power structures.

My one cavil concerns Janovicek's statements that "the American Historical Association has argued successfully that oral histories should not be subject to review by Institutional Review Boards" and that, in the United States, "oral history projects are exempt from IRB review." As and have noted, the AHA's 2003 declaration of victory was premature, and the debate continues on both sides of the border.

Wednesday, January 3, 2007

Why not make IRB review voluntary?

One of the best arguments I have read in favor of IRB review of interview and survey research is Joan E. Sieber, "Privacy and Confidentiality: As Related to Human Research in Social and Behavioral Science," in National Bioethics Advisory Commission, Ethical and Policy Issues in Research Involving Human Participants, volume II (The Commission, 2001).

Sieber, a psychologist who has studied the ethics of research since the early 1980s, provides some compelling examples of cases in which a researcher might have trouble deciding what levels of confidentiality are appropriate and how to achieve them. For example, she lists the complex networks of laws that might require the disclosure of research data. And she points out the technical difficulties of knowing what data--age, job title, and the like--might be harmless in one case but sufficient to identify individuals in others. While I do not agree with all of her judgments, were I planning research in some of the sensitive areas she describes, I would want to have her advice or that of someone like her.

Sieber concedes, however, that many or most IRBs are unable to provide such guidance:

The regulations of human research, as currently written, give little hint of how finely the protocol and informed consent relationship must be crafted in response to the manifold aspects of privacy and confidentiality in social research. Worse, they do not allude to the background of concepts and plans that would underlie such an effective protocol and relationship. Remarkably, some IRBs function quite effectively despite these ambiguities, through wise interpretation by IRB members who are well schooled in ethical problem solving and whose scientific training has provided relevant research competencies. Such a fortunate confluence of education, competency, and effort is not the norm, however. Nor can such outstanding performance reasonably be expected of most IRBs, which are woefully overworked and under-budgeted.

She continues:

There is now a literature of virtually hundreds of approaches to protecting privacy or assuring confidentiality. This literature is rarely sought out by IRBs, researchers, or teachers of research methods. Most are not even aware that it exists. . . .

Many IRBs have little sense of the range of privacy issues they should consider or of the range of solutions that might be employed in the service of valid and ethical research. Many IRB chairs, members, and staff persons are not in a position to effectively guide or teach their clientele, or to gain the respect of their clientele.

In other words, many IRBs know less about research ethics than the researchers themselves, and are likely to do more harm than good when they intervene.

Sieber proposes to remedy this problem with a "web-based educational resource . . . that will guide the ethical problem solving in research. Itshould not be offered as an official regulation, or interpretation of regulations, but as a user-friendly educational resource that will challenge IRBs, researchers, teachers, and students to improve their ability to craft solutions to ethical and methodological problems." Instead of judging protocols by guesswork, IRBs could apply scholarly standards, backing up their recommendations with citations to empirical research. In this vein, Sieber now edits a new journal, the Journal of Empirical Research on Human Research Ethics, which presumably seeks to provide the kind of knowledge she identified as crucial in 2001.

Unfortunately, Sieber fails to address the institutional barriers to her recommendations. As Carpenter and Hyman note, IRBs have little incentive to base their decisions on scholarly ethics, rather than regulations. If the federal government audits them, it will judge them on their compliance with procedures, not the soundness of their advice. And since the researchers at their institutions form a captive audience, they have no real reason to "gain the respect of their clientele," as Sieber puts it.

If Sieber really believes in the power of IRBs to educate researchers, she should advocate voluntary review. Good librarians do not need federal requirements to attract researchers; they need sticks to keep the researchers away. The same is true of other university offices that offer research assistance, such as writing centers and statistical or computer training departments. Why not make IRB review voluntary?

Skilled IRBs would lose nothing, since serious researchers would continue to seek their advice. Unskilled IRBs, in contrast, would have to shape up by educating themselves in the ways Sieber recommends. And neither researchers nor their subjects would lose anything as researchers quit consulting incompetent boards.

The system would not be perfect, but it would impose a quality control now lacking.