Must Employees Consent?

Two recent items concerning IRBs and studies of job performance caught my eye.

The more prominent item was a December 30, 2007, New York Times op-ed, "A Lifesaving Checklist" by Dr. Atul Gawande. Gawande describes efforts by the Johns Hopkins University and the Michigan Health and Hospital Association to offer five-step checklists to intensive-care providers, to help them remember to wash their hands and take other steps to avoid infection. OHRP shut the project down, even though it had been ruled exempt by the Hopkins IRB. (So much for local control.)

Although I try to keep this blog focused on issues concerning the humanities and social sciences, and to avoid issues concerning strictly medical research, I was struck by OHRP's determination that "the subjects of the research were both the healthcare providers at the participating ICUs and their patients." Are healthcare providers really human subjects in a case like this?

A similar question arose in a non-medical setting a few weeks later. In her pseudonymous blog on Education Week, eduwonkette complained that New York City teachers were being studied without their consent and without the approval of an IRB. Since eduwonkette herself concedes that "teachers did not need to consent in this case, as they are government employees and their employers can collect whatever data they want," I don't understand why she thought an IRB should be involved. But it does suggest that the question of studying employee effectiveness is not limited to medical contexts.

Poking around in my research notes, I find that at the July 1977 meeting of the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, Dr. Karen Lebacqz raised just this issue as a hypothetical:

"Suppose you want to do a study that involves changing the delivery of services in a mental institution ward. The patients in the institution are the ones that you want to study in terms of what the impact is on their recovery rates and so on. The staff who work on that ward will also be very directly impacted by the research. Do they constitute subjects, even though the design and the purpose of the research is not set up to study the impact on them of what you are going to do?"

While Albert Jonsen initially suggested that the staff would constitute subjects, other commissioners disagreed:

Lebacqz. “There is a very direct impact on the licensed staff.

Ryan. “Then the administrator can force it on them as a condition of employment.

Turtle. “Yes, I was going to say, that is the whole key, it is a condition of employment. Many things that, while I am a great libertarian here, lots of things I ask the people who work for me to do, I don’t ask them for their informed consent.”

[National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, Transcript, Meeting #32, July 1977 (Bethesda: The Commission, 1977), 133-135.]

After this exchange, the commission moved onto another subject, suggesting a consensus that employees are not human subjects whose informed consent is required. In its recent determination letters on the checklist project, OHRP has ruled the opposite.

I don't expect the compliance officers at OHRP to be familiar with the commission's deliberations, and I fault the commission for not defining human subjects more carefully. But it's worth noting how far OHRP has strayed from the commission whose work nominally informs its decisions.

How IRBs Decide--Badly: A Comment on Laura Stark's "Morality in Science"

Laura Stark's recent essay in Law & Society Review led me to her 2006 Princeton University dissertation, "Morality in Science: How Research Is Evaluated in the Age of Human Subjects Regulation." The heart of the dissertation is her description of the workings of three university IRBs--one in a medical school and two at universities without medical schools--based on recordings of their meetings and her direct observation of the IRBs at work. It makes for fascinating reading, and I applaud Stark for her achievement even as I disagree with her conclusions.

Stark claims to be neutral about IRBs' ability to perform their stated task: protecting the subjects of research. She writes, "My goal is not to judge the 'fairness' and 'effectiveness' of IRBs myself." (7) And she correctly notes that the ethical acceptability of an IRB-approved project is a "social truth," not an empirical one. (244) But her tone is generally sympathetic to the IRBs. For example, she writes that IRBs' "forms of evaluation provide directed, tangible ways for board members to carrying out their review process, given the practical difficulty of applying unmediated, abstract ethics principles," making the IRB members sound like heroes who have achieved a workable system against the odds. (186)

Indeed, in some cases she reports, IRBs seem to be doing some good. For example, a physiologist and a nurse had a fruitful debate about the need for a quick medical screening of subjects in an exercise study (197-200). That's an example of an IRB with multiple experts on a single type of research--something I hope is reasonably common in much biomedical research. But most of Stark's observations are distressing in ways I don't think she appreciates. Here, then, are some of the actions she observed, along with my reasons for finding in them an indictment of the IRB system as presently run.

NITPICKING

At all three IRBs she observed, Stark saw members judging proposals based on the proportion of spelling and typographical errors in the proposal. She calls such behavior "housekeeping" and excuses it on the grounds that it "was indispensable for IRBs because the apparent degree of care taken in submitting a tidy proposal served as a proxy for an investigator's ability, allowing board members to make judgments about people's reliability as researchers. In this way, ink and paper serve as character witnesses." (173)

Such behavior, I believe, represents what Sir James George Frazer called the practice of "homeopathic magic." As Frazer put it in the Golden Bough, "the magician infers that he can produce any effect he desires merely by imitating it." In this case, the proposal serves as a magic charm, and a tidy proposal guarantees an ethical research project. That IRBs would resort to such practices is strong evidence that they lack the expertise to judge proposals on their merits.

KIBITZING

Stark calls a second kind of evaluation "scientific evaluation," which sounds nice. But her observations confirm the complaints of many researchers that IRB members demand changes in research they do not understand.

For example, when a social scientist sought permission to interview survivors of domestic violence who became community activists, a lay member considered insisting that the researcher also involve women who were not activists, or who remained in abusive relationships. Stark notes that "the investigator explained that she had not intended to do a comparative study and that the idea of constructing a control group and making her hypothesis explicit were antithetical to Grounded Theory," and that a statistician on the IRB rose the the researcher's defense (205). The statistician admirably stated his intention to remain silent when he lacks expertise, but it's clear that not everyone on the IRB possesses such self-restraint. In another case, a board demanded that an investigator justify decisions that, Stark notes, "did not bear directly on protection of human subjects." (179) The investgator became flustered, and almost agreed to changes that would have invalidated the findings, until another board member intervened and shut up his colleague, while the rest of the board laughed.

Stark reports such events gently, stating, "criticism of the quality of an investigator's science often came from expert members who did not belong to the same discipline as the [investigator]," leaving it up to the investigator's disciplinary colleagues to defend the proposal. (180) She writes, "license to evaluate the scientific merit of studies extended beyond what IRB members could justify in the name of research subjects. Science evaluation as a human subjects issue was at times self-consciously melded with criticisms made for the sake of better science." (181) But in her own examples, members' suggestions were at least as likely to degrade the science as improve it.

IGNORING EMPIRICAL EVIDENCE

Stark finds that IRB members who aim for "subject advocacy" rely on their imaginations, rather than empirical evidence about the effects of various types of research. She offers this story:

"an IRB chair (at a board where I did not observe) described to me an episode involving a mental health worker who served as her board's community representative. An investigator had proposed a study on homeless people, which the community representative resisted because she felt that the population was too easily exploited. Her resistance to this research, according to the chair, was symptomatic of a broader problem in which the community representative would 'overly identify with patients and overestimate risks, and not really attend to data to balance her perspective.' In this instance, the community representative 'was not willing to hear the scientific data,' which indicated that interviewing homeless people did not harm them and that in fact interviewing them provided useful data that might aid the group. The community representative instead argued that 'the mere act of interviewing them was putting these folks as risk.' Thus, to this IRB chair, 'critical thinking and stepping back, taking a little distance from an issue, and trying to look at it in an objective fashion just wasn't something [the community representative] was willing to do…Since the committee doesn't function by consensus, we just moved ahead, and I'm certain she felt sidelined." (183)

It's nice to hear that this member was overruled, but I still pity the researchers who must present their proposals to her. Overall, Stark paints a particularly grim picture of the role of lay members on IRBs. It seems that they only get the attention of the researchers on the board once they've been so co-opted that their comments are indistinguishable from those of the "scientist" members.

SUBORNING DISHONESTY

Federal regulations (45 CFR 46.111) require IRBs to determine that "selection of subjects is equitable." While I don't think this is a wise criterion for judging research in the social sciences and the humanities, I can see its importance for medical research. Unfortunately, establishing an equitable selection of subjects is very difficult. In the case Stark observed, the IRB simply encouraged the researcher to lie about his intentions:

"The investigator indicated that it would be difficult in practice to use subjects from the [predominantly minority] location he had just mentioned because of the logistics of the study. Then, Reverend Quinn joined the discussion. Together, they clarified for the investigator what the board was looking for—and why:

'Reverend Quinn: Actually, even by adding the phrase after "efforts will be made to recruit from senior and community centers throughout the state," "including those that serve areas of minority populations." Just something like that, would simply make it clear you are being more proactive that otherwise people would think you were.

'Olin: And that would suggest, too, that we and you are being more vigilant.'" (185)

The language proposed by the IRB would indeed suggest that the researcher would be more vigilant, even when the researcher had little intention of recruiting minority subjects. A better strategy would have been to allow the researcher to honestly state his intentions and proceed with the research as planned, then seek the resources needed to do the difficult work of including minorities.

KILLING RESEARCH

Most of Stark's stories concern petty IRB interventions. There's some tinkering with a consent form (182), and squabbling over whether a phone call asking parents to participate in a follow-up study constitutes "invasion of privacy." (233) This sort of effort to avoid hurt feelings is far removed from the kinds of permanent harms against which federal regulations are meant to protect.

In one case Stark observed, an IRB intervened more seriously, perhaps killing a study. An investigator wanted to ask parents how they disciplined their children and was reluctant to report suspected child abuse to state authorities lest the prospect of such reporting lead parents to lie. An IRB member who knew a child who had been killed by abuse spoke up, and this led the board to refer the project to the university lawyers. So far so good; regardless of the personal experiences of the board members, an investigator in this position should know and follow the applicable laws about reporting child abuse. But, Stark continues, once the referral was made, "after several months with no decision, the investigator abandoned her research plan and withdrew the project from consideration." (211) She doesn't explain who delayed the project--the IRB or the university counsel--but the upshot is that rather than improve a potentially important research project, the process killed it. Moreover, because technically the proposal was withdrawn rather than rejected, the IRB can continue claiming a low rejection rate.

SINKING INTO A RUT

Stark acknowledges that when offered identical proposals to review, IRBs will respond with wild variation. She asked eighteen IRB chairs how they would react to a proposal to test job discrimination by sending black and white applicants to apply for the same job, then waiting to see who would get a call back. She reports that "the chairs diverged dramatically on both the problems that they identified and the modifications that they requested. Because of their distinctive local precedents, IRB chairs’ had dissonant ideas about what risks the standard protocol entailed, to whom, and with what severity, which guided them towards distinctive decisions about whether consent could be waived, whether investigators could get consent without invalidating their data, and whether debriefing should be mandatory or prohibited as a source of harm in its own right." (231)

Stark defends such variation as evidence of "local precedents" that allow predictability within an institution (239). She compares such decision-making to the "pragmatic, not experimental, tradition that was developed in Anglo-American law and medicine during the late nineteenth century." (240) But the best doctors and lawyers of that century shared their knowledge broadly, and read broadly too. Stark offers no examples of an IRB member calling in an outside expert or doing some independent reading.

Stark argues that "the highest priority for IRBs is consistency--not with other IRBs, but with their own prior decisions." (4) She leaves this finding without further comment, neither praising nor condemning the IRBs. But she takes her relativism too far. An IRB that bases its decisions on spelling errors is consistent, predictable, and irrelevant. An IRB that always requires oral historians to destroy their recordings is consistent, predictable--and wrong. And the highest priority of IRBs should be the protection of participants in research. If they are unwilling to shoulder that responsibility, they should disband.

Stark briefly hints at an awareness of this problem when she notes that "IRBs would make more similar judgments if local boards shared decision-making precedents at a national level. Given that IRBs make decisions based on cases, the challenge for a coordinated review system is not to craft more detailed federal regulations, but to train IRB members with a limited set of nationally-shared cases on which boards can base their decisions, as an alternative to local precedents." (243) That sounds like an interesting proposal, but it would require a massive overhaul of the current regime, from OHRP down to local boards. As it stands, Stark has given us a close look at a broken system.

Johns Hopkins Dean: Generalizable Means "Hypothesis-Based"

In his comments on Inside Higher Ed's recent story on oral history and IRBs, Eaton Lattman, Dean of Research at Johns Hopkins Krieger School of Arts and Sciences, writes that "many oral history projects probably do not meet the OHRP definition of research, and are therefore free from the need to go to an IRB for approval. They are not hypothesis-based projects and they do not produce generalizable results."

This interpretation of generalizability is consistent, as best I can tell, with the intent of the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, which first established generalizability as part of the definition of human subjects research. It also resembles Columbia University's recent statement on oral history.

Dean Lattman's comment contrasts with definitions put forward by Michael Carome of OHRP and consultant Jeffrey Cohen, who maintain an oral history project is generalizable research if it lacks a hypothesis but draws conclusions, informs policy, or creates an archive for use by future researchers.

Thus, Hopkins joins Columbia and Northwestern in explicitly freeing oral history--as practiced by oral historians--from IRB review. While such advances are welcome, more welcome still would be a statement along the same lines from OHRP itself.

How State IRB Laws Threaten the Social Sciences: A Comment on Shamoo and Schwartz

Dr. Adil E. Shamoo of the University of Maryland School of Medicine kindly sent me a copy of his new article, "Universal and Uniform Protections of Human Subjects in Research," The American Journal of Bioethics 7 (December 2007): 7-9, co-authored with Jack Schwartz of the Maryland Attorney General’s Office.

The article calls for a federal law "to require that all human subject research in the United States, regardless of funding source or relationship to FDA marketing approval, be undertaken only after IRB review and with the informed consent of subjects." Barring that, it applauds state laws to that effect, such as the one passed in Maryland in 2002, which states that "a person may not conduct research using a human subject unless the person conducts the research in accordance with the federal regulations on the protection of human subjects . . . notwithstanding any provision in the federal regulations on the protection of human subjects that limits the applicability of the federal regulations to certain research . . . ." (13 Maryland Code §13-2002)

Having read the current New Yorker's story on "Guinea-Pigging," which states that 70 percent of drug trials now take place in the private sector, I can see the reason to regulate biomedical research through the state's police power, rather than through funding restrictions. But aside from a quick mention that research sponsored by the National Endowment for the Humanities is not subject to the Common Rule, the article does not mention the law's effect on the humanities and social sciences. This is a bit surprising, given Shamoo's earlier argument in the Chronicle of Higher Education that human subjects regulations "have handicapped researchers whose work poses no threat to humans." This being the case, why would he want to extend those same regulations to countless researchers not previously covered by them? And do the authors expect all journalists and oral historians in the state to be subject to IRB review, the way that many university-affiliated journalists and oral historians are now?

I wrote to both authors, and here is what I learned from them:

* Shamoo does not believe that the law requires Maryland journalists to seek IRB approval of their work.

* Schwartz does not believe that the law requires Maryland oral historians to seek IRB approval of their work, relying on a 22 November 2005 message from Michael Carome of OHRP.

* According to Schwartz, "The Maryland law, like every bill passed by the Maryland Legislature, was reviewed for constitutionality. This overall review did not address whether some hypothetical application of the law, under a specific set of facts, might raise First Amendment or other constitutional concerns."

* According to Schwartz, the Maryland attorney general's office has not brought any enforcement actions since the law's passage in 2002.

This last point is the key; a law that lies dormant for five years is unlikely to have any effect on anybody, so perhaps Shamoo and Schwartz have reason to think it will not handicap researchers more than they already are handicapped by existing federal regulation. Indeed, the legions of journalists, book authors, and market researchers who conduct interview and survey research outside of universities may have little to fear from a law that has not been enforced and might well fail constitutional scrutiny if it were.

But Shamoo and Schwartz ignore the potential effect of the law on social science and humanities researchers within universities, the very ones for whom Shamoo earlier expressed concern. These researchers have led a growing movement to get their universities to agree to review only research funded by the federal government. Such a move would free few, if any, university-affiliated biomedical researchers from oversight, since their expensive research is generally federally funded. Rather, it would free up the very kind of research that Shamoo considers too low-risk to merit review. But if the state imposes the same regulations regardless of the source of funding, university IRBs can close this door.

Thus, if the law has any effect at all, it will not be what Shamoo and Schwartz claim: "universal application of the ethical standards applicable to human subjects research." Instead, it will be the continued regulation of research by university-affiliated researchers while non-affiliated researchers conducting the same activities work in freedom. The result will not be uniformity, but entrenched disparity.

Shamoo's heart is in the right place. As he wrote in his Chronicle piece, "I have long advocated the creation of universal rules for all human-subject research, whether or not it receives federal funds. But an equally urgent reform is to exempt from present and future regulation any research that poses little or no risk to human subjects." Unfortunately, the state of Maryland has not treated exemption of social research as "equally urgent" to the universalization of rules. Thus, the effect of the law, if any, is to eliminate a path to the very type of exemption that Shamoo champions. Once again, regulators write rules in response to serious concerns about medical experimentation, with little or no attention to the social sciences and humanities.

Inside Higher Ed Reports on Comments to OHRP

Scott Jaschik's January 3 Inside Higher Ed story, "Threat Seen to Oral History," reports on the comments submitted to OHRP by the AHA and by me. The story is a helpful summary of key issues, and the comments following it indicate the passion this issue arouses in scholars in a range of fields.