Peter Moskos on What to Tell Your IRB

Sociologist Peter Moskos is author of Cop in the Hood, a book about police work he wrote based on his fieldwork as a Baltimore police cadet and police officer. He maintains a blog concerning the issues raised in the book (a splendid marriage of old media and new), and on February 12 he commented about the ethical issues raised by Sudhir Venkatesh's Gang Leader for a Day. In that posting, "Outing the insiders," Moskos wrote,

I’ve never been a fan of the I.R.B. Few professor are. I don’t think that overt non-experimental academic researchers should need approval to observe and interact with most human subjects. We’re not giving out experimental drugs. We’re not running experiments. We’re watching and talking and living. I don’t even like the term “human subjects.” It’s dehumanizing. They're people, damnit! It’s condescending to think that adults aren’t smart enough to make their own decisions about what to say to whom. And if they’re not, well, such is life.

Nor am I convinced that research subjects who harm others deserve institutional protection. I believe academics should act under a code similar to journalists. But federal law disagrees with me. And the press has explicit constitutional protection that professors don’t.

In a comment on his blog, I pressed him to elaborate on these points, and he graciously responded with a second posting, "More on IRBs." I recommend reading the whole post, but here are some key points:

1. An IRB doesn't have to reject a proposal to stifle research. As Moskos notes, "the simple nuisance and fear of conflict with an IRB limit social-science research."

2. IRBs are set up to review protocols in advance, but that's not how ethnography works. Moskos is grateful to Harvard's IRB for requiring him, at the outset of his fieldwork, to announce to his academy classmates who he was and what he was doing. But as his work progressed, he found keeping his IRBs informed about every change to be so tedious that it wasn't worth the effort. And now that his book is out, he is still in touch with some friends from the department. When does oversight end?

As I noted in August, the University of Pennsylvania has addressed some of these concerns in its policy on evolving research. I would like to learn how that policy is working.

3. IRBs apply the wrong ethical standards. They seek to ensure that no harm comes from research, when, in cases like Venkatesh's, the "risk of some harm from his research was so great as to be virtually inevitable." And IRBs "want a guarantee of confidentiality," which is not appropriate in all circumstances; Moskos did not witness any serious crimes, but he writes that "if, hypothetically, I witnessed a police officer rob and kill, or sexually abuse a 10-year-old child, or anally violate an innocent man with a plunger, I would feel little compunction legally and ethically to violate a vow of confidentiality."

This third argument is the one that most intrigues me, since it accepts the notion that good research can be harmful. Theoretically, IRBs can approve such research, so long as the benefits of the research are commensurate. But Venkatesh eloquently describes the ethnographer's uncertainty that his research will matter, either to the individuals studied or to society as a whole. Thus, it would be hard for most ethnographers to present the kind of cost-benefit analysis required by the Belmont Report.

Moskos' acceptance of harm seems fully consistent with the ethics of the American Sociological Association, but not with the code of the American Anthropological Association, which states that "anthropological researchers must do everything in their power to ensure that their research does not harm the safety, dignity, or privacy of the people with whom they work, conduct research, or perform other professional activities." Thus, Moskos's eloquent statement of his principles illustrates why folklorists, historians, sociologists, and others should reject being lumped together with anthropologists as "social scientists," lest they find themselves subjected to ethical standards not their own.

At the end of his post, Moskos shares some of the language that he used to get IRB approval without compromising his ethical principles, and he encourages other researchers to use this language to avoid promising written consent forms or absolute confidentiality. Whether or not a researcher is subject to IRB oversight, such thoughtful statements about the ethics of research in specific contexts are valuable--far more so than the standardized ethical training mandated by most IRBs. I hope that other researchers will both read Moskos's statement and consider writing their own.

AAHRPP Calls for Research on IRBs and Behavioral and Social Science

Rob Townsend kindly alerted me to the Winter 2008 issue of AAHRPP Advance, published by the Association for the Accreditation of Human Research Protection Programs, which features the lead article, "IRBs and Behavioral and Social Science Research: Finding the Middle Ground." The article concedes that "many behavioral and social scientists feel constrained by a system that seems tilted toward biomedical research and, therefore, neither understands nor reflects their concerns." And it reports the interest of Drs. J. Michael Oakes, "an author and frequent lecturer on IRB review of behavioral and social science research," and Howard Silver, Executive Director of the Consortium of Social Science Associations, in addressing some of those concerns.

Oakes and Silver "encourage researchers to investigate scientifically IRB oversight of behavioral and social science research. Such research could determine whether IRBs are consistent in applying the federal regulations, whether IRBs are taking advantage of the flexibility that’s built into the regulations, and whether relationships between IRBs and social scientists are less strained on campuses that have separate IRBs to review behavioral and social science research. The resulting data could shed light on ways to relieve tensions between these two groups."

That sounds good, but it's a bit disappointing that the article does not acknowledge the considerable research already completed on this topic, much of it already cited on this blog. Nor does it remark on the curiosity that IRB oversight has continued for four decades without anyone knowing if it does any good.

Moreover, the article states as fact some beliefs that should be investigated with just the sort of research it calls for. I hope Oakes, Silver, and AAHRPP will allow data to challenge some of their own presuppositions.

Here are some questions that could be answered by further research.

1. When did IRB review of social science go bad, and why?

AAHRPP thinks it already knows the answer to this one. The article claims that "Tensions began building in the late 1990s in response to increased government scrutiny of research involving human participants," and that "the regulations have not changed. What’s new is their enforcement and, in many instances, that enforcement is overdue." I am at work on a history of IRB review of social science and humanities research, and I think more research can challenge this view.

The first part of the AAHRPP claim is doubtful; social scientists have protested IRB regulations since 1966, and tensions have waxed and waned since then. If we take this longer view, then the assertion about the immutability of the regulations is wrong; the regulations--first promulgated in 1974--changed twice, in 1981 and 1991. And the 1991 revisions greatly expanded the reach of IRBs. The 1981 regulations exempted survey, interview, and observational research unless it “deals with sensitive aspects of the subject’s own behavior, such as illegal conduct, drug use, sexual behavior, or survey or interview procedures is use of alcohol" and if "the subject’s responses, if they became known outside the research, could reasonably place the subject at risk of criminal or civil liability or be damaging to the subject’s financial standing or employability. The 1991 regulations, in contrast, eliminated the "sensitive aspects" clause and added potential harms to reputation to the list of triggers for IRB review. These changes were made over the objection of social scientists. And they set the stage for the conflict of the 1990s and today.

The claim that "what's new is [the regulations'] enforcement," is only half-true. Also new is the guidance issued by OPRR/OHRP since 1995 that reversed previous policies.

Finally, the article claims that "enforcement is overdue." Really? What errors did social scientists commit in the 1980s--a decade of relatively light regulation?

2. Why do IRBs sometimes delay or prohibit social science research?

Dr. Oakes states, “IRB members are not those folks who are looking to thwart your study. They are peer researchers who have a job to do.” But clearly some IRB members are looking to thwart studies, or else studies wouldn't get thwarted as often as they do. The question is how many IRB members do this, and why.

One part of this question concerns membership. Oakes's claim that IRB members "are peer researchers" depends on an odd definition of peers. In the NIH peer review process, for example, proposals are reviewed by study sections whose members are chosen for their expertise. The NIH's Center for Scientific Review requires, among other things, that

"* Expertise is the paramount consideration when developing/updating a study section roster.

"* Each scientific area reviewed by the study section needs appropriate expert representation."

IRBs theoretically must include experts on each type of research reviewed, but Oakes knows as well as I that this requirement is often ignored. Additional research might indicate how often a researcher faces an IRB with no expertise in the methods under review.

Then, of course, some IRB members are not researchers at all, but the "one member whose primary concerns are in nonscientific areas" required by the regulations. As Laura Stark's dissertation suggests, these members can be particularly undisciplined in their meddling.

3. What types of research now fall subject to IRB review?

Like PRIM&R, AAHRPP thinks that IRBs review only two kinds of scholarship: biomedical research, and something called "behavioral and social science research." The article states, "AAHRPP’s Founding Members, Board of Directors, Council on Accreditation, and Supporting Members all include representatives of organizations engaged in behavioral and social science research."

This statement suggests the fallacy of the undistributed middle term:

* Ehnographers are represented by organizations engaged in behavioral and social science research.
* Organizations engaged in behavioral and social science research have a voice in AAHRPP.
* Therefore, ethnographers are represented by organizations that have a voice in AAHRPP.

The second premise is "undistributed," since it is not true that all organizations engaged in behavioral and social science research have a voice in AAHRPP.

Here's a counter example:

* Countries in South America, Africa, and South Asia are not in North America or Europe.
* Countries from parts of the world other than North America and Europe are permanent members of the UN security council and the G-8.
* Therefore, countries in South America, Africa, and South Asia are permanent members of the UN Security Council and the G-8.

In committing this fallacy, AAHRPP lumps together a dozen or more scholarly disciplines--each with its own history, methods, and ethics--into a single category: "behavioral and social science research." The AAHRPP website does not list the disciplinary affiliations of members of its Board of Directors, Council on Accreditation, or list of site visitors, but if there's a journalist, historian, or folklorist in the lot, I'll be surprised.

To take the example I know best, oral historians do not expect psychologists, social workers, or education researchers to understand or represent their interests. AAHRPP (like PRIM&R) should find out how many disciplines are now subject to review, and include representatives from all of them.

4. What models of ethical review exist, and what models might we imagine?

The article asks "whether relationships between IRBs and social scientists are less strained on campuses that have separate IRBs to review behavioral and social science research." But that is only one of several alternative systems in place on various campuses. For example, Macquarie University delegates ethical review to a number of subcommittees with special expertise in certain fields. And the University of Pennsylvania allows researchers using some social science methods to forego "a fixed research protocol." And we can imagine even more models, some of which would require redrafting present regulations, others of which might not.

I appreciate AAHRPP's call for research, and I hope it agrees that research is most valuable when the answers are not predetermined.

Historians Flood OHRP With Comments

In response to the October 2007 announcement in the Federal Register calling for comments about the existing guidance on expedited review, oral historians and their allies have flooded OHRP with complaints about IRB review of oral history and requests for an unambiguous exemption.

As the original announcement noted, comments sent in response to Federal Register notices are a public record. Mr. Glen Drew and Ms. Toni Goodwin of OHRP kindly sent me copies of all 65 comments on expedited review. Of these, 38 commented on oral history or folklore, with all but one of those seeking exclusion for such research.

The comments came from a wide range of scholars. University historians ranged in rank from graduate students to chaired professors. Non-university historians included those working for federal and state agencies and for private companies. Historians of science and medicine—among those most familiar with the medical research that led to the current regulatory scheme—were particularly vocal. The American Historical Association weighed in against IRB review—reversing its 1998 stance—as did the American Association for the History of Medicine, the American Folklore Society, and the Society of American Archivists. Scholars in sociology, English, psychology, medicine, and American studies also called for oral history's exclusion from IRB jurisdiction, as did one IRB chair.

These scholars' complaints about IRB review of oral history will be familiar to those who have followed this controversy. Many noted IRBs' demands that scholars submit questions to be approved in advance, a practice that outrages oral historians who pride themselves on their ability to improvise questions in response to their research and the stories they hear.

Of the 38 comments, only one did not condemn IRB jurisdiction over oral history. Yet even that comment, by Claytee White of the Oral History Research Center at UNLV, does not endorse IRB review as practiced at most universities. Rather, White notes that she spends only 15 minutes clearing each project, suggesting blanket approval for her work rather than the project-by-project review that is the focus of most complaints.

The comments acknowledge that "memory can be painful," but they also point out that "historians are professionally obliged to ask our interview partners probing questions independent of the benefit or harm for the interviewee." Because they do not expect IRBs to understand this, historians—and the IRB chair—suggest that other oral historians would be better equipped to judge oral history projects than IRBs with few or no historians as members.

The scholars do not think IRB members mean-hearted, just hopelessly unfamiliar with the practice of oral history. "University IRBs do not have the necessary background to appreciate that oral history research is different from other research involving human subjects," wrote one. Nor do they expect this to change. A historian of medicine notes that "as long as IRB members are active, overworked faculty volunteering their time, they will be unable to track the nuances of a style of research they see very rarely."

For this reason, the historians do not seek modification of the current system, but an unambiguous removal of oral history from IRB jurisdiction. At least 14 comments, including the American Historical Association's, endorsed the 2006 recommendation of the American Association of University Professors "that research on autonomous adults whose methodology consists entirely in collecting data by surveys, conducting interviews, or observing behavior in public places, be exempt from the requirement of IRB review—straightforwardly exempt, with no provisos, and no requirement of IRB approval of the exemption."

As I noted in my own comment, the present guidance was adopted in 1998 in response to six comments about oral history, one of which included the American Historical Association's endorsement of oral history's inclusion on the list of methods eligible for expedited review. Now that the AHA and 36 others have called for the wholesale exclusion of oral history from IRB review, I hope OHRP will be as responsive as its predecessor was a decade ago.

Here are excerpts from the comments on oral history in PDF form.

Ethics Yes, IRBs No

The blog "Law and Letters" features a posting from "Belle Lettre" entitled Venkatesh's Gang Leader For a Day and IRBs. The author asks whether the ethical problems raised by Sudhir Venkatesh's book form an argument for IRB review of ethnography.

As the comments make clear, however, Lettre has uncritically equated IRB-approved research and ethical research. IRB critics do care about research ethics, but we question whether IRB review is either necessary or sufficient to ensure adherence to scholarly ethics and the law.