More failures of "local precedents"

Laura Stark’s 2012 book, Behind Closed Doors: IRBs and the Making of Ethical Research, devotes a chapter to what Stark calls “local precedents,” her term for “the past decisions that guide board members’ evaluations of subsequent research.” “By drawing on local precedent,” Stark claims, “board members can read new protocols as permutations of studies that they have previously debated and settled based on members’ warrants. The result is that IRBs tend to make decisions that are locally consistent over time.” (47)

But I keep getting stories about IRBs that are locally inconsistent.

NPRM: Escape for Many, Scant Relief for Those Left Behind

While the NPRM might do much to reduce the number of projects requiring IRB review, it would do little to improve the quality of review for those projects for which it is still required. This is a retreat from the more ambitious plans of the 2011 advance notice of proposed rulemaking.

[This post will be cross-posted to the Petrie-Flom Center's Bill of Health, which is conducting an online NPRM Symposium.]

Horror Story Buffet

We end the year with two collections of IRB horror stories.

[Varma, R. “Questioning Professional Autonomy in Qualitative Inquiry.” IEEE Technology and Society Magazine 33, no. 4 (winter 2014): 57–64. doi:10.1109/MTS.2014.2363983; Glenda Droogsma Musoba, Stacy A. Jacob, and Leslie J. Robinson, The Institutional Review Board (IRB) and Faculty: Does the IRB Challenge Faculty Professionalism in the Social Sciences? Qualitative Report 19 (2014), Article 101, 1-14, http://www.nova.edu/ssss/QR/QR19/musoba101.pdf]

NRC Report: Assess Risk Empirically

One theme running throughout the NRC report is the need to replace the worthless gut reactions decried by Ezekiel Emanuel with a system that would base its judgments on the latest empirical evidence. But the report does not present a clear set of reforms that would effect this change without scrapping the current system of local IRB review.

Emmerich on Schrag, Stark, and van den Hoonaard

Nathan Emmerich of Queen's University, UK, reviews Ethical Imperialism, Behind Closed Doors, and The Seduction of Ethics for Research Ethics.

[Emmerich, Nathan. “Between the Accountable and the Auditable: Ethics and Ethical Governance in the Social Sciences.” Research Ethics 9, no. 4 (December 2013): 175–186. doi:10.1177/1747016113510654.]

Emmerich is particularly frustrated by the lack of accountability of research ethics committees:

"The review process renders research accountable whilst, at the same time, erasing any trace of its own accountability (Stark: 73) or, we might say, its own status as an ethical endeavour . . .

"How systems of governance should themselves be held responsible − to researchers, to research participants, and to society as a whole − remains uninterrogated by applied ethical thinking."

Dreger Reviews Stark: It Is Lawyers All The Way Down

Alice Dreger reviews Laura Stark's Behind Closed Doors for the Journal of American History:

Contrary to the self-aggrandizing story bioethicists like to tell about how IRBs arose out of concern for human subjects of research, Stark shows that, when you dig into this history, it is lawyers all the way down . . . She argues that IRB work was decentralized not to make it more ethical, but to protect the NIH from lawsuits. Stark convincingly concludes that IRBs today do not primarily enact ethical principles; they manage procedures.

[Dreger, Alice. “Behind Closed Doors: IRBs and the Making of Ethical Research.” Journal of American History 99, no. 4 (March 2013): 1328–1328. doi:10.1093/jahist/jas666.]

I Review Stark, Behind Closed Doors

The American Journal of Sociology has published my review of Laura Stark's Behind Closed Doors. I describe it as an "illuminating account of how ethics review really works," but note that "Stark’s reluctance to condemn [IRB] behavior sets her apart from other observers of IRBs in action" and that it is "a stretch for Stark to claim that today’s IRBs use 'a decision-making model that stabilized in the 1950s and 1960s.'"

[Zachary M. Schrag, Review of Behind Closed Doors: IRBs and the Making of Ethical Research by Laura Stark. Chicago: University of Chicago Press, 2012. Pp. Viii+229. $85.00 (cloth); $27.50 (paper).” American Journal of Sociology 118, no. 2 (September 2012): 494–496. www.jstor.org/stable/10.1086/664671]

For my comments on Stark's dissertation, on which the book is based, see "How IRBs Decide--Badly: A Comment on Laura Stark's 'Morality in Science.'"

Emmerich Reviews Behind Closed Doors and Ethical Imperialism

Nathan Emmerich of Queen's University, Belfast, finds that Laura Stark's book and my own "together . . . illustrate the nature of ethics as an aspect of research governance fundamentally contributing to our understanding of the phenomena in a manner that goes beyond the relatively limited or restricted consideration offered by applied ethical analysis."

[Nathan Emmerich, Review of Behind Closed Doors: IRBs and the Making of Ethical Research and Ethical Imperialism: Institutional Review Boards and the Social Sciences, 1965-2009, Sociological Research Online 17, Issue 2 (May 2012).]

Against Armchair Ethics: Some Reflections from Petrie-Flom

As followers of my Twitter feed will know, I spent Friday and the first half of the Saturday at The Future of Human Subjects Research Regulation, a conference sponsored by the Petrie-Flom Center for Health Law Policy, Biotechnology, and Bioethics at Harvard Law School.

The conference organizers, led by Professor Glenn Cohen, did a magnifent job bringing together participants with varied views and backgrounds, and as a group we discussed ideas ranging from treating research subjects as workers to including adolescents on IRBs. Those who want the full story should be able to watch videos of the presentations before too long.

I was struck, however, by one recurring theme: the distinction between evidence-based approaches and armchair ethics.

Dingwall: UK May Enact Research Integrity Rules With Little Scholarly Input

Writing on social science space, Robert Dingwall warns that the United Kingdom's Economic and Social Research Council seems to be crafting burdensome regulations on research integrity without defining the problem to be solved, considering the costs of regulation, or consulting scholarly associations:

Inside Higher Ed Interviews Stark

Laura Stark speaks about her new book, Behind Closed Doors, with Inside Higher Ed.

[Mitch Smith, "Behind Closed Doors," Inside Higher Ed, 8 February 2012.]

Stark explains, "It would seem that fairness is not the only criteria used in IRB evaluations."

Stark Reviews Ethical Imperialism

Laura Stark, assistant professor of sociology at Wesleyan University, reviews Ethical Imperialism for the American Journal of Sociology. She finds that the occasional "keen observation . . . is not harnessed to a broader analytic framework or explanatory apparatus."

[Laura Stark, Review of Ethical Imperialism: Institutional Review Boards and the Social Sciences, 1965-2009 by Zachary Schrag, American Journal of Sociology 117 (November 2011): 1019-1021.]

ANPRM's Problem Statement: Helpful but Incomplete

One of the many remarkable sections of the July 26 advance notice of proposed rulemaking (ANPRM) is its admission that the Common Rule is flawed.

(Note: I have added a link to the ANPRM at the top of the link list in the sidebar.)

Since the 1970s, IRB apologists have claimed that federal regulations are flexible enough, and that local IRBs are to blame for any problems. In 2007, for example, Jerry Menikoff quoted with approval Jeffrey Cohen's 2006 claim that "the regulations provide sufficient flexibility for the efficient and appropriate review of minimal risk research. IRB review of such research does not have to be burdensome or unreasonable if IRBs appropriately utilize the flexibility in the regulations." Menikoff reiterated his claim of "flexibility within the system" in his 2009 speech, “The Legal Assault on the Common Rule."

After thirty years of such claims, it is wonderfully refreshing that the ANPRM takes so seriously many of the critiques leveled at the federal regulations themselves. And the ANPRM helpfully organizes those critiques into seven general categories.

On the other hand, ANPRM's problem statement (pages 44513-44514 in the Federal Register version) overlooks some major critiques. Fortunately, some of those critiques are implicitly recognized by some of the ANPRM's proposals.

Elliott Wants to Scrap IRBs

Carl Elliott, author of White Coat, Black Hat: Adventures on the Dark Side of Medicine, calls IRBs "incapable" and wants them replaced.

[Carl Elliott, Useless Pharmaceutical Studies, Real Harm, New York Times, 29 July 2011.]

U of Michigan Reports Some Progress

The University of Michigan has released the results of a 2009 survey of investigator experiences in human research. The survey suggests that matters have improved somewhat since the university launched its HRPP Policy Innovation and Demonstration Initiative in 2007, but that more work remains to be done.

[Survey Research Center, Institute for Social Research, University of Michigan, "2009 Follow-Up Survey of Investigator Experiences in Human Research," December 2010. h/t: Human Research Protections Blog.]

Sociologists Find IRBs Serve Organizational Interests, Not Professional Ethics

Sociologists Carol Heimer (Northwestern) and JuLeigh Petty (Vanderbilt) find that IRBs "substitute bureaucratic ethics for professional ethics."

[Carol A. Heimer and JuLeigh Petty, "Bureaucratic Ethics: IRBs and the Legal Regulation of Human Subjects Research," Annual Review of Law and Social Science 6 (2010): 601-626.]

Much of the article consists of concise, accurate summaries of many of the complaints lodged against IRBs, including some by your humble blogger. (The bibliography lists well over 100 works on IRBs and research ethics.) Heimer and Petty categorize these complaints as "critiques of IRB law as law, critiques of IRBs as regulation, and critiques of IRBs as a system of norm making." Critics have charged that IRBs act lawlessly, do more harm than good, and deny researchers the opportunity to shape the norms that govern them. "IRBs seem to have lost sight of their original objective," Heimer and Petty state, summarizing some of this work. "No longer collective bodies of researchers deliberating together about the ethical dilemmas they encounter, IRBs are instead agents of the university (or research center). Rather than protecting research subjects from harm, they now seem especially focused on protecting universities and research centers."

To these complaints (which they mostly seem to endorse), Heimer and Petty add three of their own.

First, they employ the "lens of inequality," finding that "the regulations fail in part because the research process does not go as the regulators imagine and because the regulations do not address the social sources of the big inequalities. Furthermore, the regulations support inequality when they prevent research on powerful groups who harm others." IRBs fret over the details over consent forms, ignoring evidence that "potential research subjects actually pay little attention to consent forms and later do not even remember the details in them." At the same times, IRBs ignore "structural inqualities" (most notably, "the big inequality that the majority of the global research funds address the health problems of the wealthy few") while perpetuating inequality by preventing social researchers from studying powerful groups, including sellers of loose cigarettes.

I find the section on "the big inequality" the least persuasive part of this article. Heimer and Petty too readily accept the claims of Jill Fisher and Adriana Petryna that (in the words of Heimer and Petty) "the focus on abstract, universal principles in the Belmont Report deflects attention from the structural conditions and inequalities under which the unethical treatment of research subjects has taken place." Fisher and Petryna mischaracterize both the Belmont Report and the Common Rule by claiming (in Petryna's words) that "so long as an investigator [can] document that his or her subjects could deliberate about personal goals and act 'under the direction of such deliberation,' it [is] ultimately up to the subjects themselves to judge the acceptability of the risks they [take]."

In fact, the Belmont Report specifically warns against imposing the burdens of research "upon poor ward patients, while the benefits of improved medical care [flow] primarily to private patients," and the Common Rule requires IRBs to determine that "risks to subjects are reasonable in relation to anticipated benefits" and that "selection of subjects is equitable," independently of ensuring informed consent.

Whether IRBs are able to do this, and to do this without inappropriately restricting a great deal of ethical research, is another question. But it's unfair to charge the National Commission or the authors of the regulations with ignoring the problem of structural inequality and the challenge it poses to a consent-based model.

After the section on the "lens of inequality," Heimer and Petty "look at IRBs through the lens of professions." Noting that "the regulation of human subject research is a growth industry," they warn that rather than cede the power to declare a project exempt (as suggested by accommodationist reformers like Levine and Skedsvold), "IRB professionals [may] defend and perhaps seek to expand their jurisdiction." In doing so, they are protecting "their livelihood, a secure niche on the edges of the research and scholarly world." And they will have help: "OHRP’s focus on documentation helps explain why IRB professionals and not bioethicists are the growth sector in human subjects regulation."

Finally, Heimer and Petty "examine IRBs and research enterprises as organizations." Here they find that "a complex mixture of coercion by the government, fear of loss of funding, individual professional self-interest . . . and a desire not to be seen to be on the wrong side of a key cultural divide" do more to explain the growth of IRBs than do the Nuremberg Trials and other documented cases of unethical research.

They conclude with a grim assessment:

As the regulation of human subjects research has been institutionalized, professional competition and the protection of organizational interests seem to have carried the day. A bureaucratized research ethics is essentially an ethics of documentation. The task of translating the principles of autonomy, beneficence, and justice was never going to be easy. But translations that ignore structural inequalities, delay or reduce valuable research, and substitute bureaucratic ethics for professional ethics may not bring as much progress as we hoped.

First, Do Some Harm, Part II: The AAA Ethics Task Force

In mid-October, the Ethics Task-Force of the American Anthropological Association solicited comments on the following text, a section of a draft Code of Ethics now being written:

Do No Harm

Anthropologists share a primary ethical obligation to avoid doing harm to the lives, communities or environments they study or that may be impacted by their work. This includes not only the avoidance of direct and immediate harm but implies an obligation to weigh carefully the future consequences and impacts of an anthropologist’s work on others. This primary obligation can supersede the goal of seeking new knowledge and can lead to decisions not to undertake or to discontinue a project. Avoidance of harm is a primary ethical obligation, but determining harms and their avoidance in any given situation may be complex.

While anthropologists welcome work benefiting others or increasing the well-being of individuals or communities, determinations regarding what is in the best interests of others or what kinds of efforts are appropriate to increase well-being are complex and value-laden and should reflect sustained discussion with those concerned. Such work should reflect deliberate and thoughtful consideration of both potential unintended consequences and long-term impacts on individuals, communities, identities, tangible and intangible heritage and environments.

As of December 13, 33 people (presumably all anthropologists, but I'm not sure) had posted comments. The comments are often nuanced, making it hard to say whether they endorse the language or not. But they broke down roughly as follows:

Do No Harm

Significantly, the most wholehearted supporters of the "do no harm" proposal are those who uncritically embrace the Belmont Report and the Common Rule. "'Do no harm' is an IRB principle, and so it should be in our code," writes Bethe Hagens. Four other responses, from Chip Colwell-Chanthaphonh, mkline, Robert T Trotter II, and Simon Craddock Lee, all seem to suggest that the AAA code should conform to those documents, without asking much about their origins or their fit to the practices and beliefs of anthropologists.

Four other responses--from Barbara Rose Johnston, Seamus Decker, socect, and Vicki Ina F. Gloer--endorse Hagens's idea that anthropologist should "intend no harm." Despite the Belmont Report's description of "the Hippocratic maxim ”do no harm” [as] a fundamental principle of medical ethics," this form is more faithful to the Belmont's overall section on beneficence.

Do Some Harm

Eight responses--almost as many--appear to reject the "do no harm" idea on the grounds that neutrality is impossible, and anthropologists should not hesitate to harm those who deserve it. "A blanket edict to 'Do No Harm' could easily lead to a professional paralysis when one considers that a few steps away from the person giving you this interview is someone who will not like, will want or need to fight, or will suffer consequences for what is said much further down the line," writes Benjamin Wintersteen. Murray Leaf concurs. "Do no harm is fine as principle of medical practice," he writes, "where you are working with a single individual. It is nearly meaningless when you (we) work with human communities, in which what is good and what is harm is usually in contention. As some of these posts suggests, what we do is often a matter of helping some while undermining the position of others. No harm at all, in such a context, would almost always be also no help at all–and no effect at all."

Bryan Bruns offers an example. "I work, in conjunction with communities and a government agency, to design and support a process in which communities are likely to, in a reasonably democratic way, act to restrain the behavior and thereby (harm) reduce the benefits of a few people (upstream irrigators, large landowners) who currently take advantage of others, it’s not clear how a principle of 'do no harm' would allow any practical engagement."

I would say that the responses by Dimitra Doukas, Joan P Mencher, Moish, Noelle Sullivan, and Ray Scupin all fall in this general category of respecting critical inquiry. Margaret Trawick's comment is harder to categorize. "I have been teaching 'Do no harm' to my students as the first ethical principle for anthropological fieldwork, for many years," she writes. "It is a difficult principle to follow, precisely because you never know what might cause harm, and therefore you have to THINK about what you are doing in the field more carefully than you might in everyday life. Good intentions are not enough. Additionally, 'harm to whom' is a good question . . . Sometimes to protect and advocate for one party (.e.g. Untouchables in India) is to, at the least, offend some other party – e.g. high caste Hindus." Given her understanding of this problem, I'm not sure why she teaches "do no harm" rather than something like "think about whom you are harming."

It's the Wrong Question

An even greater number of responses suggest that, in the words of Carl Kendall, "This principle is way too vague and self-directed to be practically useful." Kendall hints, perhaps cynically, that anthropologists need one set of principles these ethical principles to "pass IRB muster" and a second set "to protect communities and fieldworkers." Carolyn Fluehr-Lobban argues that "'Harm' should be problematized—are there agreed upon universal standards of harm, and where is there discussion of reasonable disagreement."

James Dow rejects the medical language of IRBs: "'Do no harm' is an good ethical principle to be applied to individual social relationships, which we hope that we understand; however, there is a problem when applying it to larger societies and cultures." Likewise, David Samuels writes that "The place where you need to get informed consent is at the point at which you have turned people into characters in your story. The medicalized pre-framing of the IRB process doesn’t cover that at all."

Taken as a whole, the responses suggest that only a minority of those commenting embrace the Belmont Report and the IRB process as enthusiastically as the AAA did in its 2004 statement that presents the active involvement of IRBs as a positive good. I hope the Task Force recognizes this, and takes the opportunity to reconsider the AAA's overall position in regard to IRB review.

[Hat tip to Alice Dreger. For a historical perspective on another discipline's efforts to craft a research ethics code, see Laura Stark, "The Science of Ethics: Deception, the Resilient Self, and the APA Code of Ethics, 1966–1973," Journal of the History of the Behavioral Sciences 46 (Fall 2010): 337–370.]

Dreger Wants to Scrap IRBs

On the heels of Laura Stark's Los Angeles Times op-ed calling for the replacement of local IRBs with centralized boards of experts, historian Alice Dreger has published her own call for a national system of ethics review based on expertise and transparency.

[Alice Dreger, "Nationalizing IRBs for Biomedical Research – and for Justice," Bioethics Forum, 22 October 2010.]

Troubled by her IRB's approval of a project she considers unethical, and by Carl Elliott's White Coat, Black Hat: Adventures on the Dark Side of Medicine, Dreger concludes that the system of local review is ineffective:

We’ve reached the point where many people in medicine and medical ethics don’t even expect IRBs to act as something other than liability shields for their universities. But do patients who come to us only to be turned into subjects know that? Do they know that there is literally a price on their heads put there by research recruiters?

I’ve come to believe we need a radical solution. Maybe what we need is a nationalized system of IRBs for biomedical research, one that operates on the model of circuit courts, so that relationships cannot easily develop between the IRBs and the people seeking approval. This system could be run out of the Office for Human Research Protections and involve districts, similar to the federal courts system. Deliberations would be made transparent, so that all interested parties could understand (and question) decisions being made.

Think of the advantages: the possibility of actually focusing on the protection of human subjects first and foremost, free of conflicts of interest; the possibility of having nothing but trained professionals (not rotating unqualified faculty and staff) sitting on review panels; the possibility of marking biomedical research as clearly different from the social science and educational research unreasonably managed by many IRBs; the possibility of much greater transparency to those interested in seeing what’s going on; the possibility of having multi-center trials obtain a single approval from one centralized IRB, rather than trying to manage approvals from multiple local institutions. And the possibility of shutting down the deeply opaque, highly questionable private IRBs Elliott describes as being increasingly used by universities. (Go ahead, call me a Communist for caring about the Common Rule.)

Her Communist leanings aside, I don't know why Dreger presents her argument as a defense of the Common Rule, which fails to distinguish between biomedical and social research, puts ethics review in the hands of rotating unqualified faculty and staff, and keeps deliberations opaque. But her wish for the kind of coordination and transparency provided by the court system has a long lineage. I've quoted it before, and I'll quote it again:

The review committees work in isolation from one another, and no mechanisms have been established for disseminating whatever knowledge is gained from their individual experiences. Thus, each committee is condemned to repeat the process of finding its own answers. This is not only an overwhelming, unnecessary and unproductive assignment, but also one which most review committees are neither prepared nor willing to assume.

[Jay Katz, testimony, U.S. Senate, Quality of Health Care—Human Experimentation, 1973: Hearings before the Subcommittee on Health of the Committee on Labor and Public Welfare, Part 3 (93d Cong., 1st sess., 1973), 1050].

It is not lack of good intentions or hard work that leads IRBs to restrict ethically sound surveys while permitting unethical experimental surgery. It is the ignorance and isolation identified by Katz in 1973 and still in place today.

Stark Wants to Scrap IRBs

Sociologist Laura Stark is a careful observer of the IRB system, having based her dissertation on archival research and direct observations of three university IRBs. In 2008, I complained that the dissertation, "Morality in Science," reported but failed to condemn bad IRB behavior. In a newly published essay, Stark takes a more critical stance.

[Laura Stark, "Gaps in Medical Research Ethics," Los Angeles Times, 8 October 2010.]

In her essay, Stark traces today's IRB system back to systems established in the 1960s at the NIH Clinical Center, which performed experiments on "hundreds of healthy prisoners, conscientious objectors, unemployed people and students living in their hospital as subjects." She finds that system included two basic flaws: it failed to inform the public about what was going, and it gave no voice to dissenting members of ethics boards. These flaws, she argues, remain in today's system of ethics review.

To remedy them, Stark proposes that the new Presidential Commission for the Study of Bioethical Issues "rebuild the regulations from the ground up." She writes,

New rules should include these changes:

Replace the thousands of local review boards that labor independently at universities and hospitals here and abroad with a small number of ethics-review networks organized around specific research methods rather than around institutions. The networks would be better equipped to handle multi-site studies that are now commonplace, and would remove the political biases of some outlier institutions.

Consider the advantages and disadvantages of outsourcing ethics review to private companies, which review research for a fee.

Finally, empower research participants by posting the results of ethics reviews online. The current system includes community representatives who presumably speak on behalf of research participants, but that's not good enough.

Though the essay does not specifically mention IRB review of research in the social sciences and humanities, the dissertation gives examples in which the lack of transparency and lack of expertise impeded such projects. Thus, Stark's call for expert boards and published results of ethics review could address non-biomedical research as well as the medical research that is her chief concern. And while Stark presents her proposals as ways to ensure better protection for research participants, they could also benefit those researchers who now fall victim to inexpert boards. Because the current system fails both researchers and participants, reform can benefit them both.

Stark should realize that the changes she proposes would require more than "rebuild[ing] the regulations from the ground up," since the requirement for local IRBs is encoded in federal statute, not just regulations. But a wholesale reconsideration of the IRB system by the presidential commission would be a fine first step.

How IRBs Decide--Badly: A Comment on Laura Stark's "Morality in Science"

Laura Stark's recent essay in Law & Society Review led me to her 2006 Princeton University dissertation, "Morality in Science: How Research Is Evaluated in the Age of Human Subjects Regulation." The heart of the dissertation is her description of the workings of three university IRBs--one in a medical school and two at universities without medical schools--based on recordings of their meetings and her direct observation of the IRBs at work. It makes for fascinating reading, and I applaud Stark for her achievement even as I disagree with her conclusions.

Stark claims to be neutral about IRBs' ability to perform their stated task: protecting the subjects of research. She writes, "My goal is not to judge the 'fairness' and 'effectiveness' of IRBs myself." (7) And she correctly notes that the ethical acceptability of an IRB-approved project is a "social truth," not an empirical one. (244) But her tone is generally sympathetic to the IRBs. For example, she writes that IRBs' "forms of evaluation provide directed, tangible ways for board members to carrying out their review process, given the practical difficulty of applying unmediated, abstract ethics principles," making the IRB members sound like heroes who have achieved a workable system against the odds. (186)

Indeed, in some cases she reports, IRBs seem to be doing some good. For example, a physiologist and a nurse had a fruitful debate about the need for a quick medical screening of subjects in an exercise study (197-200). That's an example of an IRB with multiple experts on a single type of research--something I hope is reasonably common in much biomedical research. But most of Stark's observations are distressing in ways I don't think she appreciates. Here, then, are some of the actions she observed, along with my reasons for finding in them an indictment of the IRB system as presently run.