Political Science Perspectives on IRBs

The July 2008 issue of PS, the journal of the American Political Science Association, offers a five-part symposium: "Protecting Human Research Participants, IRBs, and Political Science Redux." Over the next few days I plan to comment on each article, but for now it's safe to say that while each author offers a different diagnosis and prescription, none thinks that the current system is working well.

The Psychologist Who Would Be Journalist

Back in August 2007, I mentioned the controversy surrounding the book The Man Who Would be Queen (Washington: Joseph Henry Press, 2003) by J. Michael Bailey, Professor of Psychology, Northwestern University. At the time, Professor Alice Domurat Dreger, also of Northwestern, had just posted a draft article on the controversy. Now that article, along with twenty-three commentaries and a reply from Dreger, has appeared in the June 2008 issue of the Archives of Sexual Behavior.

Dreger's article, the commentaries, and Dreger's response focus on big questions about the nature of transsexuality, the definitions of science, power relationships in research, and the ground rules of scholarly debates. Only a handful take up the smaller question of whether—as a matter of law and as a matter of ethics--Bailey should have sought IRB approval prior to writing his book. But that's the question that falls within the scope of this blog.

Oral Historians Draw Conclusions, Inform Policy, and Generalize Findings

In the lead story of today's New York Times ("Occupation Plan for Iraq Faulted in Army History"), Michael R. Gordon reports on a new 700-page official history of the early occupation of Iraq, produed by the Army’s Combined Arms Center at Fort Leavenworth. As Gordon reports, "the study is based on 200 interviews conducted by military historians and includes long quotations from active or recently retired officers." He notes that "the study is an attempt by the Army to tell the story of one of the most contentious periods in its history to military experts — and to itself." It draws important conclusions with policy implications, finding, for example, that "the military means employed were sufficient to destroy the Saddam regime; they were not sufficient to replace it with the type of nation-state the United States wished to see in its place.”

This sounds suspiciously like the kind of project comprising generalizable research as defined by OHRP's Michael Carome in his October 2003 discussion with the UCLA Office for Protection of Research Subjects (as reported by UCLA.) In that conversation, Carome noted that

Systematic investigations involving open-ended interviews that are designed to develop or contribute to generalizable knowledge (e.g., designed to draw conclusions, inform policy, or generalize findings) WOULD constitute "research" as defined by HHS regulations at 45 CFR 46.

[Example]: An open ended interview of surviving Gulf War veterans to document their experiences and to draw conclusions about their experiences, inform policy, or generalize findings.

Except for the fact that it's the wrong Gulf War, the Army study nicely fits Carome's example of research requiring review.

Fortunately for federal historians, no one else in the federal government seems to share Carome's view on this matter. I know of no federal agency, executive or legislative, that requires IRB review for oral histories conducted by its employees. As reported on this blog, even OHRP officials did not submit to IRB review when conducting oral history research.

Maybe Dr. Carome will try to discipline the researchers at Fort Leavenworth. Him and what army?

IRB Disciplines and Punishes a Qualitative Researcher

Tara Star Johnson reports her experiences in "Qualitative Research in Question: A Narrative of Disciplinary Power With/in the IRB," Qualitative Inquiry 14 (March 2008): 212-232.

Johnson left teaching high school to pursue a PhD in Language Education at the University of Georgia. As she completed her preparatory work, she found "no qualitative studies investigating the phenomenon of sexual dynamics in the classroom." She decided, for her dissertation work, "to address this void in educational research through in-depth interviewing of teachers who have experienced desire for and/or from students to trace how these attractions happen and open the door for dialogue about embodiment, desire, and sexuality in education." Her professors were encouraging, and her advisor accompanied her to her appointment with the IRB.

After waiting an hour and a half beyond their scheduled appointment, Johnson and her advisor finally met with about twenty members of the IRB. The chair listed several restrictions, which Johnson found disappointing, but "not unreasonable or completely unexpected." Then the fun began.

Music Educator Finds IRBs Inconsistent, Restrictive, and Burdensome

Rhoda Bernard kindly alerted me to Linda C. Thornton, "The Role of IRBs in Music Education Research," in Linda K. Thompson and Mark Robin Campbell, eds., Diverse Methodologies in the Study of Music Teaching and Learning (Charlotte, North Carolina: Information Age, 2008), 201-214.

Thornton (along with co-author Martin Bergee) wanted to survey music education majors at the 26 top university programs to ask why they had chosen music education as a profession. She writes, "no personal information regarding race, habits, or preferences was being collected—only descriptive data such as each student's major instrument (saxophone, voice, etc.), age, and anticipated year of graduation." She dutifully submitted her proposal to her local IRB, and then the trouble began.

Thornton's own IRB forbade the researchers from surveying students at their own institutions, then imposed requirements suitable for a survey on sexuality or criminal activity. Most significantly, it required Thornton to seek permission from the IRBs at the 24 universities remaining in her pool.

Nine of the 24 accepted the proposal as approved by Thornton's IRB, including one which noted it had a reciprocity agreement in place. Of the remaining 15, several imposed burdensome requirements, ranging from small changes in the informed consent letter (which then needed to re-approved by the original IRB), and the requirement that the instructor at the local institution, who was just going to distribute and collect questionnaires, be certified in human subjects research. Application forms ranged from two pages to eight; at least one IRB demanded to know the exact number of music education majors in every school to be surveyed. The result was that the researchers dropped many of the schools they hoped to study, cutting their sample from several thousand to 250.

This sad story touches on two points: inconsistency, and regulatory exemptions.

Participatory Research Meets the IRB

Participatory Research and Participatory Action Research are approaches that seek to include the people studied as participants in framing and answering questions. In Participatory Action Research, researchers also seek to work with participants to effect change, rather than merely identifying problems. For reasons unknown to me, many researchers who embrace these approaches are affiliated with the discipline of geography. In 2006, a number of American and British participatory researchers frustrated by ethics committees found themselves discussing the problem at meetings of the the Association of American Geographers and the Royal Geographical Society/Institute of British Geographers. The result was a special thematic issue (volume 6, number 3, 2007) of ACME: An International E-Journal for Critical Geographies. The ten essays show that participatory researchers think hard about ethics, and as a result often find themselves struggling with ethics committees that do not.

In their introductory essay, "Participatory Ethics: Politics, Practices, Institutions," Caitlin Cahill, Farhana Sultana, and Rachel Pain explain that

As participatory researchers, we pursue research and other activities with communities (or traditional research ‘subjects’) as collaborating partners, with the primary goal of working towards positive changes on issues identified by the collective. We try to engage in all aspects of research - research questions, the choice and design of methods, the analysis of data, the presentation of findings, and the pursuit of follow up action - as collaborative projects which require negotiation between the different parties. So the complex challenge of negotiating ‘ethics’ – as multiple and contested, and whether in institutional or everyday spaces – is central to our research process and inquiry. (305)

Unfortunately, that kind of thoughtful approach to research ethics does not easily fit into the standardized, medical model of ethics committee in the United States and Britain. As the authors note, "researchers seem increasingly subject to a restrictive, inflexible and top-down view of what ‘ethics’ should be, via the codes of human subject panels which we are expected to adhere to." (307)

The remaining essays express frustrations that will be familiar to any social scientist who has followed the IRB debates. Deborah Martin complains that IRBs

conceptualize research participants as “subjects” who face potential harm and exploitation in the research process . . . [Participatory research], however, seeks to redefine the researcher-subject model, conceptualizing research as a collaborative, negotiated process in which the direction and benefits of the research are as much a product of the participants’ involvement as the researcher’s. (322)

Sarah Elwood notes that "institutional ethics assume that ethical problems and risks can be identified before they occur, can be identified outside the context of the research situation, and that rules for ethical practices can be universal." (331) Kye Askins served on an ethics committee and was dismayed that the process emphasized forms and paperwork over training students to think ethically. (358) And Megan Blake complains that "the easy camaraderie born of friendship and underpinned by trust is undermined by [committee's] implicit assumption that the research may lead to harm, exploitation or suffering for those involved." (417)

Two essays present outrageous IRB behavior. Blake's experience at the University of Sheffield sounds more or less comical:

I was required to get a [criminal background check] before I could research the food practices of my children and their friends, and had to have my friends sign confidentiality and copyright agreements as I served them a cup of tea and a biscuit in my home. If I followed strictly the guidelines on anonyminity, I would also be required to ignore the details that I know about my friends as individuals when I analyse their accounts. (417)

More seriously, Matt Bradley spent months meeting with IRB staff and the chair, only to have his project--a documentary film--rejected on the grounds that "there is risk that people in the community might be upset about the portrait that has been painted." (340) This is what lawyers call viewpoint discrimination; Bradley was free to tell a happy story, but barred from telling an angry one. Yet even as it guessed about this risk to a community, the IRB refused to compare it to any possible benefit to that community, as opposed to individual participants or to scholarly knowledge. He concludes that far from manifesting respect for persons,

the IRB’s insistence on anonymity in my case smacked of the paternalism and control . . . Evident in the communications I received from the committee is the notion that the people whom I was involving in my research are not smart enough to make decisions for themselves or to understand the implications and possible repercussions of their decisions. Even more problematic, however, is the notion that the committee is smart enough to make these decisions for the ‘subjects’ and will make choices both about what I can or cannot collect from them and how they can represent themselves. (347)

Other complaints are more specific to the authors' commitment to participatory research. Elwood enlisted non-scholars as "community map makers" in a participatory project. Though these map makers were, in effect, co-authors,her IRB wanted their names stripped from the maps. (333) Eventually, the authors agreed to remove the names from maps printed in academic publications. Thus, the IRB denied the map makers credit for their work. Caitlin Cahill dislikes the Belmont Report's admonition to "do no harm," on the grounds that for a participatory-action researcher, harmlessness alone is an abdication of responsibility. (366) Peter Hopkins complains that "the detached, disembodied and ‘tick-box’ approach adopted by many ethics committees often renders absent the positionalities of the researchers, downplaying the significance of researchers’ life experiences, biographies and complex identities.” (387) While he does not describe in detail his experience with committee approval, his article suggests that he was far better prepared by his own reading than by any guidance from a committee. (389)

The authors do find some benefits to the process. Elwood notes, "discussing how to address the IRB’s concerns forced us to consider more specifically how the research process might affect participants whose experiences might differ dramatically from our own." (332) And rather than thrust an IRB-mandated consent form at her interviewees, she "begin[s] the process by trying to initiate discussion about the history and politics of informed consent in research, reasons why universities require researchers to follow certain protocols, or what the process recognizes and what it might leave out." (336) Thus, she concludes, "Institutional rules for ethical practice in research and systematic oversight of researchers, however partial and frustrating they may be, ensure that all university-based research has at least one forum where the ethics and human impacts of its activities must be considered." (337) Even Bradley, whose project was derailed ten years ago, claims that "the frustrations and different needs of qualitative and action researchers have been heard on many campuses," though for evidence he relies on an article noting positive developments on just two campuses. (347)

Despite the problems they have faced, the scholars represented in this issue appear remarkably hopeful, resilient, and determined to think seriously about research ethics, and they all seek to reform the ethics committee system rather than to escape from it. I hope the committees will prove worthy of their confidence.

Michael Rowe on Situational Ethics

Bill Hart kindly referred me to Michael Rowe, "Tripping Over Molehills: Ethics and the Ethnography of Police Work," International Journal of Social Research Methodology 10 (February 2007): 37-48. Rowe, a criminologist at Victoria University, New Zealand, explains the ethical challenges he faced as a participant observer with British police in 2004.

Rowe comes across as an unusually conscientious researcher. He writes, "while designing the project, and discussing it with academic colleagues and gatekeepers within the police service, I read many of the methodological texts on ethnography and accounts by previous researchers who had used similar methods with the police." (38) Some of this preparation was overkill, in that Rowe did not himself face some of the greatest challenges of previous researchers, like witnessing excessive force, false charges, or corruption. But it did sensitize him to some important issues about police work, and the result seems to have been a careful study, respectful at once of the police, the citizens with whom they interacted, and the scholarly pursuit of truth.

Rowe concludes that generic ethical advice and rigid rules are poor guides to researchers doing fieldwork:

It is the nature of ethnographic research that the principles contained in methodological textbooks or professional codes of conduct will be stretched and perhaps distorted as they are applied in dynamic situations. Since policing is unpredictable, the ethical dilemmas police researchers might face cannot be easily anticipated . . . If an absolute code of ethics is not feasible, researchers must be prepared to be reflexive in terms of ethical dilemmas and the methodological difficulties experienced in securing informed consent and meaningful access to research subjects. (48)

Though Rowe does not mention ethics committees in this article, it is striking how much his preparation diverged from the typical requirements of IRBs, at least in the United States. Rowe's experience points to the benefit of reading as specifically as possible in preparation for fieldwork. But the standardized training programs required by most IRBs, such as the CITI Program, present highly generic information about such topics as informed consent, and nothing about topics as specific as police ethnography. And while Rowe emphasizes the researcher's need to remain flexible, IRBs focus on protocol review. By making researchers pledge in advance what they will and will not do, protocol review reduces, rather than enhances, researchers' flexibility to adapt to unexpected situations. In other words, the IRB system is structured to hamper the kind of ethical preparation that Rowe recommends.

As I've mentioned before, the University of Pennsylvania's policy on evolving research promises to relieve the second part of this problem, since some researchers, at least, are spared the need to file a "fixed research protocol."

But Penn still requires its researchers to "have documented discipline-appropriate education regarding human subject protection, in accordance with certification standards defined by the Vice Provost for Research." While Penn staff have refused me permission to see the approved training modules, from corresponding with people at Penn, I get the sense that they are pretty generic. Rowe's article suggests that such programs are not helpful, and what is really needed is for each researcher to prepare an ethical bibliography, based on problems faced by researchers who have conducted similar work. That way, each researcher would be equipped with the ethical guidance most relevant to her particular case. And in assembling the bibliography she would exercise the very independent judgment she will need in the field.

(Thanks to Rebecca Tushnet for discussing with me the idea of an ethical bibliography.)

How Talking Became Human Subjects Research

The Journal of Policy History has accepted my article, "How Talking Became Human Subjects Research: The Federal Regulation of the Social Sciences, 1965-1991," drawn from my book-in-progress on the history of IRB review of the social sciences and humanities. The article is tentatively scheduled to be published in spring 2009, but in the meantime you can read a draft at SSRN: http://ssrn.com/abstract=1124284. I welcome feedback.

Do UCLA Researchers Have Permission to Read This Blog?

In July 2007, the UCLA Office for Protection of Research Subjects (OPRS) issued a policy statement, "Human Subjects Research Determinations," stating that:

The UCLA OPRS/IRB has the sole authority to determine whether an activity conducted by UCLA faculty, staff, or students (or conducted on UCLA students) meets the regulatory definition of “human subjects research” and therefore requires IRB review and approval or certification of exemption from IRB review. UCLA faculty, staff, and students who intend to conduct activities that might represent “human subjects research” do not have the authority to make an independent determination that UCLA IRB review and approval or certification of exemption is not required.

As a result

Investigators who intend to conduct activities that might represent “human subjects research” must submit a description of the proposed activities to the UCLA OPRS/IRB for a determination of whether UCLA IRB review and approval or certification of exemption is required prior to the UCLA investigator’s involvement in the proposed activities.

Might represent to whom?

This policy can mean one of two things:

1. UCLA researchers should seek IRB permission before drinking a cup of coffee, reading the newspaper, talking with their spouses, or riding the bus. After all, any of these activities "might represent 'human subjects research,'" and the researcher can't be trusted to figure it out.

2. UCLA researchers should not seek IRB permission before drinking a cup of coffee, reading the newspaper, talking with their spouses, or riding the bus. Instead, they must ignore the literal meaning of the policy and instead make an independent determination that UCLA IRB review and approval or certification of exemption is not required.

The OPRS has just taken a step toward the first interpretation. As reported on a UCLA library blog, the new policy statement 42 gives university faculty, staff, and students blanket permission to use data from the U.S. Census and other publicly available datasets. If UCLA researchers must rely on such policy statements to read publicly available, public domain data on the friggin' internet, what can they possibly do without permission?

The 2007 policy statement ambiguously lists as references various state and federal documents, without directly claiming that any of them require or authorize the policy. As far as I can tell, they do neither. As reported on this blog, for example, OHRP staff make independent determinations of what is and is not human subjects research. UCLA's OPRS is just making up powers for itself.

Researchers Honored for Harming Human Subjects

It's Pulitzer Prize season, and once again my dear Columbia University has showered medals on reporters who placed the subjects of their stories at risk of criminal or civil liability or damaged their financial standing, employability, or reputation, all without IRB oversight. This year's board seems to have been particularly bloodthirsty, giving two prizes--rather than the usual one--for investigative reporting, as well as honoring muckraking work with prizes for public service, local reporting, and international reporting.

Since at least the 1970s, IRB critics have asked why such work is honored when a reporter does it but condemned--at least by IRBs--when a scholar is asking the questions. I have yet to find a clear answer from defenders of the system. Here's a typically fuzzy response--Dr. Jeffrey Cohen's statement before the October 2001 meeting of the National Human Research Protections Advisory Committee:

This is a very difficult issue and it borders on the whole issue of the distinction between journalism and research . . . And that is a really murky, murky area. As a matter of fact, it is one of the conversations I had at the Oral History Association because the oral historians are in that same sort of issue. I think that clearly there is a need for more guidance on distinguishing between journalism and research.

I think the courts are doing that. I mean, the courts are addressing what constitutes journalism and the extent and scope of the First Amendment rights, especially in the context of the internet. Publishing something on the internet, does that make it journalism and so forth? And so I think that the human protections movement should look to the courts for guidance on some of that.

There's also a distinction, though, between -- in a sort of common sense way -- between journalism and research. Journalism is done for the public knowledge and for the public good in the sense of providing information the public needs to know. Research has a sort of different context and that is, you know, further -- the development and furthering of generalizable knowledge, which is a somewhat different thing than the public's right to know, although they're blurred.

So I think that it is very clear in practice that the government, institutions and IRBs do have sort of a right or a responsibility particularly when it is focusing not on censorship but on protecting the rights and welfare of the subjects of research to review that, and I think the courts have upheld that. Particularly, I think, was the University of Minnesota case,* which wasn't particularly about human subjects but it was on research integrity. The courts upheld that right as opposed to journalism where infringing on that would be censorship. There is a murky area in between.

There's also the sort of traditional knowledge that your right ends at the tip of my nose. And so that you can't yell fire in a crowded theater. And so there are things that even though we have constitutional rights, we also have the right to protect subjects and that there's a balancing there that needs to be done. It is not that there's an easy answer to that.

The reason that the distinction remains a "really murky, murky area" is that the Department of Health and Human Services has failed, with all of its various commissions and advisory boards, ever to convene a group whose primary mission was to determine the rights and responsibilities of social scientists. (An exception might be the 1966 NIH conference at which social scientists asked to be left alone, but their recommendations were ignored.) Dr. Cohen's statement suggests that in lieu of such an investigation, the matter be turned over to the courts. That may yet happen, but I doubt it will be pretty. The courts can restrain the worst abuses of the present system, but a lawsuit is no substitute for sound policy-making, based on careful fact-finding.

Were regulators to take a serious look at the sort of journalism honored this week, they might find that different modes of inquiry involve different ethical practices, and different ethical goals. But once they concede that, their whole edifice starts to crumble.

See also, "James Weinstein's Anti-Intellectualism."

* Note: I don't know the nature or name of the "University of Minnesota case." I've sent a query to Dr. Cohen, and I hope to replace this footnote with his reply.