Montana Political Scientist: IRB Process Is "Cumbersome, Inadequate" for Political Research

The Montana Commissioner of Political Practices (COPP), Jonathan Motl, has determined “that there are sufficient facts to show that Stanford, Dartmouth and/or its researchers violated Montana campaign practice laws requiring registration, reporting and disclosure of independent expenditures” in October 2014, when they mailed pre-election postcards to 102,780 registered voters. The postcards were part of an effort to see if certain claims about candidates’ place on an ideological spectrum would affect voting patterns.

As part of the investigation, Motl engaged Carroll College political science professor Jeremy Johnson to determine whether the Stanford and Dartmouth researchers had violated IRB requirements. Johnson finds that they did, but also that the Dartmouth IRBs could have approved the studies without addressing the most serious ethical issues they raised. “The fundamental problem with the IRB process,” he writes, “is the narrow focus on protecting the individual subject. Concerns about human subjects in the aggregate often do not even occur to researchers, faculty, and staff involved in the IRB.”

[McCulloch v. Stanford and Dartmouth, Commissioner of Political Practices of the State of Montana, No. COPP 2014-CFP–046, Decision Findind Sufficient Facts to Demonstrate a Violation of Montana’s Campaign Practice Laws, 11 May 2015. h/t Chris Lawrence.]

NRC Report: Where's the Freedom?

My biggest disappointment with the new NRC report is its silence on the question of academic and personal freedom.

What Can One University Do?

A few weeks ago, a correspondent asked me what reforms individual universities can implement while awaiting systemic, regulatory reform. It's an excellent question, so here's a roundup from material previously covered on the blog.

No university has adopted all of these measures, and at least one of these measures has not been adopted by any. But most of them are in place already, and there's no reason they can't spread.

Inside Higher Ed Reports on AAUP Recommendations

Inside Higher Ed presents the key points of the newly finalized AAUP report.

[Carl Straumsheim, “AAUP Recommends More Researcher Autonomy in IRB Reform," Inside Higher Ed, March 6, 2013. http://www.insidehighered.com/news/2013/03/06/aaup-recommends-more-researcher-autonomy-irb-reform.
]

AAUP Publishes Final Report, Regulation of Research on Human Subjects: Academic Freedom and the Institutional Review Board

The American Association of University Professors has published the final version of Regulation of Research on Human Subjects: Academic Freedom and the Institutional Review Board, prepared by a subcommittee (of which I am a member) of the Association’s Committee A on Academic Freedom and Tenure.

The report explains,

As things now stand, the IRB system assembles local committees whose members have no special competence in assessing research projects in the wide range of disciplines they are called on to assess, whose approval is required for an only minimally restricted range of research projects and who are invited to bring to bear in assessing them an only minimally restricted body of what they take to be information, who are only minimally restricted in the demands they may make on the researchers, and whose judgments about whether to permit the research to be carried out at all are, in most institutions, final. When one steps back from it, one can find oneself amazed that such an institution has developed on university campuses across the country.

Marakowitz Reviews Ethical Imperialism

Ellen Marakowitz, a lecturer in the Department of Anthropology at Columbia University and a member of the Columbia IRB, reviews Ethical Imperialism for Academe, the magazine of the American Association of University Professors. She finds it "a well-documented history of the impact of institutional review boards on social science research in the United States," but wishes I had paid more attention to IRBs' potential to help researchers "think critically about their practices, including potential risks to subjects."

[Ellen Marakowitz, "Regulated Research" (Review of Ethical Imperialism: Institutional Review Boards and the Social Sciences, 1965–2009 by Zachary M. Schrag), Academe November-December 2012.]

AAUP Posts New IRB Report

The American Association of University Professors has posted "Regulation of Research on Human Subjects: Academic Freedom and the Institutional Review Board," a report prepared by a subcommittee of the Committee A on Academic Freedom and Tenure. I served on that subcommittee, which was chaired by Professor Judith Jarvis Thomson of MIT.

The AAUP press release explains,

In July 2011, the federal government took what may be the first step toward the most substantial change in the regulations since 1981, issuing an advance notice of proposed rulemaking, or ANPRM. More than 1,100 individuals and associations have submitted formal comments.

The AAUP report notes that “out of respect for liberty, it is normally expected that government regulation of behavior will consist in listing what is forbidden, all else being permitted.” The report goes on to suggest ways in which the regulations could be rewritten along these lines.

The Association welcomes comments on the report through September 28.

AAUP and Me

I earlier mentioned that I helped draft the AAUP comments in response to the ANPRM. I am pleased to report that I have since joined the AAUP's Subcommittee on Academic Freedom and the Institutional Review Board, which since 1981 has sought to keep IRB oversight within reasonable bounds.

New York Times: Menikoff Promises Not to Restrict Public Information

Today's New York Times reports on historians' and social scientists' hopes and concerns about the ANPRM, especially the prospect of deregulating oral history while restricting the reuse of social science data.

[Patricia Cohen, "Questioning Privacy Protections in Research," New York Times, 24 October 2011.]

AAUP Posts ANPRM Comments

The American Association of University Professors has posted a reply to the ANPRM. I am proud to have contributed to this document.

Who Should Investigate Research Misconduct?

Two recent items do not directly involve IRBs, but they raise broader issues of accountability for research misconduct.

[Erin O'Connor and Maurice Black, "Save Academic Freedom," Inside Higher Ed, 28 February 2011; Alice Dreger, "Darkness's Descent on the American Anthropological Association: A Cautionary Tale," Human Nature (published online 16 February 2011).]

Librarians Ponder IRB Resolution

On June 29, at the American Library Association's Annual Conference, Melora Ranney Norman proposed a "Resolution on Institutional Review Boards and Intellectual Freedom."

Norman, a former chair of the ALA's Intellectual Freedom Round Table, noted that

Despite the fact that walking down the street is more dangerous than any conversation could ever be, on some college and university campuses, assertions of liability or vague, unproven risk are allowed to trump any actual proof of risk or danger, to the detriment of the preservation of knowledge and the human record.

Libraries are all about preserving and providing access to the human record with all its pimples, bumps, and bruises. Many of us have heard a quote attributed to Jo Godwin asserting that "A truly great library contains something in it to offend everyone." If the human record is not created to begin, how can we collect, preserve, and provide access to it?

She then called for the ALA to "[support] the American Historical Association in its position on oral history Institutional Review Board exemption, and [join] with the American Association of University Professors in recommending that 'research on autonomous adults whose methodology consists entirely in collecting data by surveys [or] conducting interviews . . . be exempt from the requirement of IRB review—straightforwardly exempt, with no provisos, and no requirement of IRB approval of the exemption.'"

The ALA Council voted to refer the resolution to the Intellectual Freedom Committee, the Library Research Roundtable, the Library History Roundtable, and the Committee on Professional Ethics. The American Historical Association hopes "they will reconsider the decision and support our efforts after further review."

Norman has also posted a Q & A, IRBs and Intellectual Freedom.

Hat tip: Rob Townsend.

The Systematic Threat to Academic Freedom

Lisa Rasmussen kindly alerted me to her essay, "Problems with Minimal-Risk Research Oversight: A Threat to Academic Freedom?" IRB: Ethics & Human Research 31 (May 2009): 11-16. The essay mostly seeks to rebut the AAUP's 2006 report, "Research on Human Subjects: Academic Freedom and the Institutional Review Board." Rasmussen identifies some important shortcomings in that report, and she raises key questions about the relationship between IRBs and academic freedom. But I am unpersuaded by her central arguments.

Before I address them, I should note the repeated disclaimers within the essay. "I will not settle here the fundamental issue of whether a convincing argument exists that IRB review poses a threat to academic freedom," Rasmussen writes. "A longer explanation of the [AAUP report's] failures is beyond the scope of this paper, but a brief outline is possible." I am disappointed by these limits. Rasmussen devotes significant space to matters peripheral to the question of academic freedom, such as her assertion that researchers whose work was approved by a department--rather than a central IRB--would necessarily merit less legal protection, a claim whose weakness she acknowledges in a footnote. Given only six pages, Rasmussen would have done better to focus on the question posed in her title.

Rasmussen's main argument is that the AAUP report "does not demonstrate that IRBs pose a threat to academic freedom." As she notes, such a demonstration would require a definition of academic freedom, something lacking in the AAUP report. So she offers a passage from the AAUP's "1940 Statement of Principles on Academic Freedom and Tenure": "Institutions of higher education are conducted for the common good and not to further the interest of either the individual teacher or the institution as a whole. The common good depends upon the free search for truth and its free exposition." Emphasizing the grounding of this argument in the search for the "common good," Rasmussen then concludes that "there is a prima facie claim that research can be subjected to assessment regarding whether it threatens to harm the common good via harm to individuals."

I believe this is a misreading of the 1940 Statement, for it suggests that any policy aimed at safeguarding the common good is consistent with academic freedom. For example, she could have written, "there is a prima facie claim that research can be subjected to assessment regarding whether it threatens to harm the common good via the promotion of communist overthrow of the government," and that therefore a prohibition on the use of Marxist analysis is consistent with academic freedom.

A more relevant definition of academic freedom can be drawn from the AAUP's 1915 Declaration of Principles on Academic Freedom and Academic Tenure":

The liberty of the scholar within the university to set forth his conclusions, be they what they may, is conditioned by their being conclusions gained by a scholar’s method and held in a scholar’s spirit; that is to say, they must be the fruits of competent and patient and sincere inquiry, and they should be set forth with dignity, courtesy, and temperateness of language . . .

It is, however . . . inadmissible that the power of determining when departures from the requirements of the scientific spirit and method have occurred, should be vested in bodies not composed of members of the academic profession. Such bodies necessarily lack full competency to judge of those requirements; their intervention can never be exempt from the suspicion that it is dictated by other motives than zeal for the integrity of science; and it is, in any case, unsuitable to the dignity of a great profession that the initial responsibility for the maintenance of its professional standards should not be in the hands of its own members. It follows that university teachers must be prepared to assume this responsibility for themselves.

As Matthew W. Finkin and Robert C. Post write in their new book, For the Common Good: Principles of American Academic Freedom, freedom of research depends on "a framework of accepted professional norms that distinguish research that contributes to knowledge from research that does not." (54) While these two experts on academic freedom decline to offer a firm opinion on the legitimacy of IRBs, they take the AAUP's concerns far more seriously than does Rasmussen (69).

The question, then, is whether IRBs, like the boards of trustees that concerned the authors of the 1915 statement, "lack full competency to judge of [scholarly] requirements." Rasmussen suggests that IRBs merely maintain scholarly standards: "The source of the threat to academic freedom via oversight by one’s colleagues is far from clear," she writes, "especially since researchers undergo peer review for research funding and when submitting their manuscripts for publication." But IRB review is not peer review, since it is conducted mostly by people ignorant of the scholarly methods they are reviewing. (See "Why IRBs Are Not Peer Review," and other posts tagged "peer review.")

To make this a bit more concrete, we can examine the exemplary "horror stories" included in the 2006 AAUP report. Rasmussen rejects these as "unelaborated anecdotes with no documenting citations," rather than examining their implications for academic freedom.

Here's one: "A Caucasian PhD student, seeking to study career expectations in relation to ethnicity, was told by the IRB that African American PhD students could not be interviewed because it might be traumatic for them to be interviewed by the student." Or another: "A campus IRB attempted to deny an MA student her diploma because she did not obtain IRB approval for calling newspaper executives to ask for copies of printed material generally available to the public." No peer review process would impose such conditions. If these are not infringements of academic freedom, then nothing is.

Rasmussen is quite right that we should not equate "inconvenience and hassle with abridgement of academic freedom." Yet nor should we dismiss the abridgement of academic freedom as mere inconvenience and hassle. When IRBs impose conditions on research that prevent researchers from conducting the basic tasks of scholarship--talking to people of varied backgrounds, recording interviews, or telephoning for information--they abridge academic freedom. The more interesting questions are how often this occurs, and why it happens.

Rasmussen presents IRB abuse as a somewhat random process: "IRBs can function well or poorly, and which is true for a given IRB depends on many factors, not least of which are institutional support and member training." This suggests that IRB abuses are individual anomalies, rather than a pattern.

By contrast, the AAUP detects a systematic bias toward the infringement of freedom. This is better developed in the AAUP's 2000 report (cited by Rasmussen), "Institutional Review Boards and Social Science Research." That report includes such observations as "no one is likely to get into trouble for insisting that a research proposal is not exempt" and "no university is likely to want to explain to either the government or the public why its commitment to avoid harming the human subjects of research is limited by the source of funding for the research." In these and other cases, the AAUP recognizes that the IRB system punishes individuals and institutions only for approving research, not for restricting it.

The design flaws in the system have yielded a pattern of abuse. Read Maureen Fitzgerald and Laura Stark, both of whom observed repeated abuses by the IRBs they studied. Read Linda Thornton, whose work was thwarted at 15 of 24 institutions she contacted. Read Jack Katz, who shows that IRBs are particularly likely to pounce on controversial topics. IRBs can function well or poorly, but the system is weighted toward poor function.

Rasmussen acknowledges that poorly designed systems can lead to systematic problems. She concedes that the "lack of an [IRB] appeals process may threaten academic freedom." She also details the way that departmental-level review might systematically hamper research. And she ends her essay with a promising proposal for “template review:"

Disciplines at the national level might formulate templates to guide very common research approaches. For example, a research template for oral historians could stipulate that the researcher will interview individuals, record their answers, refer them to counselors if the questions have provoked strong emotions, procure consent forms, lock the transcripts securely, and identify what will happen to the transcripts at the close of research. IRBs at individual institutions would review the template once and approve it (or even decide to accept any templates from given professional societies). Thus, a researcher would simply submit a form to the IRB stating her agreement to abide by the format of the template. Upon receipt of the form, the IRB would approve the protocol.

If IRBs are not threatening academic freedom, why propose this reform? Inside this proposal is an acknowledgment that disciplinary experts and professional societies in the social sciences and humanities have been excluded from the present IRB system. While such exclusion does not automatically threaten academic freedom, we should not be surprised when it does. For all her skepticism of the AAUP report, Rasmussen has presented her own suggestion that the current system is rotten at the core.

AAUP's Rhoades Takes Soft Line on IRB Training

In an essay on compulsory sexual harassment training ("Sexual Harassment and Group Punishment," Inside Higher Ed, 12 February 2009), the new AAUP general secretary, Gary Rhoades, offers side comments on human subjects research training:

In research universities (where professors’ work routinely involves human subjects, though even there literary and some other scholars are not required to undergo such training), perhaps the most obvious example of this is the human subjects training surrounding research grants and activity. Prior to getting grants approved by the sponsored projects division of a university, an investigator must have undergone human subjects training. Although the training varies by university, there are common patterns nationally. Typically, for example, such training is online, and is not particularly rigorous, to put it mildly. Indeed, the format involves investigators taking an exam by reading some written passages and then answering questions about them. After each section or module the person finds out whether he or she missed too many questions in a section, and proceeds. If they have missed too many questions in a section they simply backtrack, get the same questions in a different order, and retake the quiz, until they pass. A widely used set of exams (which are specified to social/behavioral and biomedical research) are those offered by the Collaborative Institutional Training Initiative, which over 830 institutions and facilities (including a very large number of research universities, and indeed including the University of California at Irvine) utilize. The modules for the CITI quiz typically include three to six questions.

For the most part, although faculty complain about the inconvenience and irrelevance of the training, I do not know of anyone who would suggest that such training should be required only of investigators found to have violated the rights of human subjects. The more important questions of process and principle surround the institutional review board activities that regulate the approval of an investigator’s proposal. Here, serious questions have been raised about compromising investigators’ academic freedom to engage in certain types of research and to research certain subject matter. But the controversy is not, for the most part, about the human subjects training per se. Indeed, I would venture to say that for colleagues in the social and behavioral sciences, among the most common comments and complaints about human subjects training are that it is ineffective, that it does little by way of actually protecting human subjects and seems to be geared more to protecting the institution.

Apparently, Dr. Rhoades is unfamiliar with the widespread, principled opposition to CITI and other online training programs. That is worrisome, if it signals the retreat of AAUP from its longtime leadership in the fight against overly broad human subjects regulations and requirements.