A Defense of RECs

Professor Adam Hedgecoe of Cardiff University kindly pointed me to his article, "Research Ethics Review and the Sociological Research Relationship," Sociology 42 (2008): 873-886.

The article is a response to longstanding criticisms of British research ethics committees (RECs), especially those affiliated with the National Health Service (NHS). For example, Sue Richardson and Miriam McMullan surveyed "UK academic social researchers working in Health, or health services researchers, who had experience of using the NHS research ethics process prior to March 2004," in "Research Ethics in the UK: What Can Sociology Learn from Health?," Sociology 41 (2007): 1115-1132. Fifty-one percent of their respondents reported degrading their research design as a result of the committee approval process, while only 32 percent reported making changes for the better. Overall, 59 percent offered negative comments, while only 15 percent offered positive comments. And Richardson and McMullan set a pretty low bar for a positive comment, counting this: "It’s a lot of paperwork but once you know what is required, it’s acceptable." Overall, it seems, NHS RECs are inhibiting the sociological study of health care in the United Kingdom.

Hedgecoe seeks to rebut this impression, based on his observation of three NHS RECs in 2005 and 2006, and some follow-up interviews. He argues that "NHS RECs are not inherently hostile to social science research, especially qualitative research." (882) The double-negative construction of that thesis suggests Hedgecoe's problem: he's trying to prove that something doesn't happen, or at least not as often as ethics-committee critics believe. That's not an easy task, and I congratulate him for trying. But I find the article unpersuasive.

The article offers mostly generalities, rather than the detailed stories that are the heart of a good ethnography. For example, though telling us he "took extensive notes (including verbatim quotes)" (876), Hedgecoe offers only two vertbatim quotes from committee meetings in the whole piece. It's also short on quantitative data. Hedgecoe notes that he observed 33 committee meetings, but he doesn't say how many ethnographic projects came up for review. He describes only four, one of which, his own, wasn't technically part of his study.

Here are the four:

1. Hedgecoe's own research on RECs.

After begging to have his project reviewed by an REC, Hedgecoe received approval, including permission to sit in on committee meetings without the consent of the researchers applying for approval for their own projects. Instead, he boasts, he needed only to anonymize the people and institutions he observed. (881) Of course, Hedgecoe's determination to anonymize "people, places, and events" may have resulted in the vagueness of his findings, so I'm not sure I count this as a win. Rather, it may be an example of the blandness that critics see as one of the results of ethics review.

2. A cancer ward study:

When reviewing an application proposing qualitative interviewing of nursing staff working on a cancer ward about the impact of their work on their lives, Coastal MREC presented the applicant with a technical bureaucratic problem caused by the researcher’s independent status. The chair’s reaction to the researcher’s concern over this point was to say, 'This is not a problem we are throwing at you, but a problem we are trying think round', and indeed the solution suggested by one member (which was to affiliate with an academic institution the researcher had previously collaborated with) solved this particular problem. (879

Here the committee offers not ethical judgment, but advice on how to deal with an ethically empty bureaucratic rule. Who is forbidding independent research in the UK?

3. "A proposal to study specialist paramedics with a view to evaluating the role, seeing whether it reduced the number of people who were repeatedly admitted to A&E (so called ‘frequent flyers’)." (881)

Did you notice the word "number" in that description? Why is Hedgecoe using a quantitative project to show that RECs are friendly to qualitative research? It sounds to me as though a medical REC would have no trouble fitting this study into the clinical model.

4. Finally, the most interesting case, the "nurses study":

a senior nurse applied to do research as part of an MSc, looking at nurses' attitude towards performance-related pay. Although the study was going to be on the nurse's own team, the committee was generally inclined to approve the application since the results would be restricted to a dissertation. But when the applicant came before the committee it became clear that she had wider goals for the results of this work, including feeding into policy decisions. She also made statements that worried the REC, about wanting to work with 'people I trust and who trust me'. The REC suggested that the applicant study a team at another hospital, to avoid the issues of conflict of interest, but the applicant was not happy, claiming the REC was 'stifling research'. The committee directed her to the head of nursing research and apparently the applicant went so far as to complain to COREC, NRES’s predecessor. (879)

Hedgecoe applauds the REC, describing its decision as "deft" and writing that "researchers are often oblivious to the potentially coercive nature of supervisors asking those they manage to take part in research, especially research which may require them to reflect on their practice." (880)

I'm not persuaded that the REC made the right call; it seems like the nurses were denied a chance to collaborate in research and to shape policy in their workplace. But the bigger question, which Hedgecoe does not explore, is why the researcher felt stifled, rather than enlightened, by the guidance she received. Wouldn't an effective committee be able to persuade her that her project was flawed, rather than leading her to file a formal complaint?

Hedgecoe's silence on this question may result from his research design. While he interviewed "policy-makers, pharmaceutical industry researchers and executives, and, most pertinently for this article, members of research ethics committees themselves," it is not clear that he spoke with any of the social science researchers who sought approval from the committees he observed. They might have a thing or two to say about the way they were treated. (875) But Hedgecoe's comments about getting informed consent from everyone except researchers suggests he avoided them.

That's a pity. Hedgecoe cites with approval the "rigorous empirical data gathering" of Bradford Gray (874), without absorbing one of Gray's key arguments: it's not enough to observe an ethics committee; one must also talk to those affected by its work. For Gray, that primarily meant research participants, but it also meant researchers, at least in his work for the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. (I'm afraid I don't have handy a copy of Gray's 1975 book, Human Subjects in Medical Experimentation, so I can't recall if he spoke to researchers for that, or only committee members and research participants.) Instead of talking to researchers, Hedgecoe imposed his own judgment that they were being treated fairly.

I am glad to learn that some NHS RECs allowed two or three projects to proceed without much difficulty. But given the much more extensive data--qualitative and quantitative--in the Richardson and McMullan study, I am little reassured by Hedgecoe's assertions that RECs are doing just fine.

Oral History Update

Linda Shopes revises and expands her 2007 essay, "Negotiating Institutional Review Boards" in a page on the Oral History Association website.

Shopes, who spent years negotiating with federal officials, now despairs of that route: "After more than a decade of largely ineffective advocacy vis-à-vis OHRP and its predecessor, oral historians are not likely to gain many concessions from federal regulators."

I must agree with Shopes's pessimism. As Michael Carome conceded in October 2008, OHRP has taken action on only a handful of the 147 recommendations put forward by the Secretary's Advisory Committee on Human Research Protections. If regulators cannot or will not implement the recommendations of their own official advisory body, they are unlikely to prove more responsive to the concerns of a group of scholars whom they have ignored for years.

Instead of looking to OHRP for relief, Shopes suggests that "if we must live within a regulatory system that is, at best, incongruent with our ways of working, perhaps the best we can do is work within our individual institutions to develop a measure of mutual accommodation." She notes the progress historians have made at Amherst, Columbia, UMKC, Michigan, and Nebraska. Here's hoping the next update of her essay has a longer list.

U. of California Shouldn't Avoid Debate

In my previous post, AAHRPP and the Unchecked Box, I mentioned a 2008 memo, " “Unchecking the Box” on the FWA – Issues and Guidance," by Rebecca Landes, research policy coordinator at the University of California's Office of Research and Graduate Studies.

The memo deserves a second look, since it shows the tensions within a university administration when faced with challenges from social scientists.

On the one hand, the memo acknowledges the complaints:

There is increasing pressure of late from social science, behavioral and humanities researchers to modify IRB review of research in these disciplines. While there may be good reasons to apply different review standards to different types of research, changes in the application of subject protection rules at UC should be effected through systemwide discussion and consensus. Campus by campus modifications to subject protection rules for nonfederally funded research would lead to confusion and chaos.

I do not see why campus-by-campus modifications in this area should sow more confusion than already exists. I doubt, for example, that UCLA's absurd policies were cleared with other campuses before being promulgated. But at least this portion of the memo calls for "systemwide discussion and consensus."

But continue reading, and you get to a section on "Pros and Cons" of promising to apply federal regulations no nonfunded research. And here's one of the "pros": "Avoids opening up the debate on differing protections for different disciplines, e.g., social science, behavioral and humanities research."

So which is the real goal of the University of California administration: to foster "systemwide discussion," or to avoid opening up a debate? Only one choice is worthy of a great university system.

AAHRPP and the Unchecked Box

Regular readers of this blog likely know that most United States universities submit "federalwide assurances" (FWAs) pledging to abide by the Common Rule for research funded directly by federal agencies that have adopted that rule.

Section 4 of the standard assurance includes an optional pledge that "This Institution elects to apply . . . to all of its human subjects research regardless of the source of support, except for research that is covered by a separate assurance" either the Common Rule or the Common Rule and subparts B, C, and D of 45 CFR 46. Institutions that check this box--as seems to have been common in the past--with one stroke of the pen eliminate one of the major concessions made by federal regulators in 1981, when they promised that non-federally-funded research would not be regulated.

Recently, however, at least 164 universities have "unchecked the box," declining to promise to apply the regulations to all research. The American Association of University Professors has strongly recommended that universities uncheck the box as a first step toward devising procedures less burdensome than those specified in the regulations.

Malcolm Feeley has noted that unchecking the box could also yield important empirical data:

If there are few reports of negative consequences . . . they might encourage national officials to rethink the need for such an expansive regulatory system . . . On the other hand, if opt-out results in increased problems, the findings might help convince Katz, Dingwall, me, and still others of the value of IRBs.

Nor are such comments confined to outsiders. At the July 16, 2008, meeting of the Secretary's Advisory Committee on Human Research Protections, committee member Lisa Leiden of the University of Texas system spoke of her own interest in freeing nonfunded research from direct federal regulation:

We have talked about limiting the federal wide assurances, unchecking the box, and I believe the position that we're going to be taking is to advocate in a gentle way thinking about doing that. We have heard both sides of the story or maybe just a few sides, but we think that there are certainly some advantages. And one of the advantages might be . . . what can we do with the expedited review level. It seems that there is a lot of flexibility in that, and we might be able to increase some of that by unchecking the boxes and adding different categories for that.

Unchecking the box is therefore one of the most promising incremental reforms now on the table. This is why I was disappointed to see that the AAHRPP's proposed revised standards, described in my previous post, seem to preclude this option.

A correspondent questioned this assertion, noting that AAHRPP president Majorie Speers had mentioned unchecking the box in her presentation, "Finding Flexibility in the Regulations." But there's nothing in the slides to suggest that AAHRPP or Speers approves of such a practice, and a July 2008 memo from the University of California states that AAHRPP site visitors have told university administrators "that in order for a human research protection program to be accredited, it must apply the Common Rule and its subparts to all human research at the institution, irrespective of funding."

Either AAHRPP forbids accredited organizations from unchecking the box, or its policies are so unclear that its site visitors are giving out bad information. Either way, I suggest that the revised standards permit unchecking the box as a means of reform.

AAHRPP Proposes Revised Standards

Robert Townsend, PhD, kindly alerted me to the Proposed Revised Accreditation Standards of the Association for the Accreditation of Human Research Protection Programs (AAHRPP). The revisions are largely cosmetic, grouping many of the existing standards under new headings. As far as the review of social science and humanities research goes, I see no drastic departures from previous AAHRPP positions. This is a pity, since the standards need more substantive revision to meet the goals that AAHRPP has set for itself.

AAHRPP is accepting comments until July 30. My comment follows.

To the AAHRPP,

Thank you for the opportunity to comment on your proposed revised accreditation standards. If enforced, these standards would provide researchers with some protections against arbitrary actions by IRBs and human research protections staff. But they fail to look at the bigger picture and ask why so elaborate a structure is necessary to oversee areas of research with sparse record of doing wrong, or whose ethical challenges are too unpredictable to be spotted in advance by an ethics committee. Moreover, the standards fail to take seriously the suggestions of some of the most informed critics of the present structure.

Researchers in the social sciences and humanities have long complained that their work was being reviewed by committees that lacked the necessary expertise. I am therefore glad to see that Element II of the new standards requires that "the IRB or EC has and follows written policies and procedures requiring protocols or research plans to be reviewed by individuals with appropriate scientific or scholarly expertise . . . " It also insists that "the IRB or EC is comprised of members to permit appropriate representation at the meeting for the types of research under review, and this is reflected on the IRB or EC roster," a criterion currently violated by most IRBs that review ethnographic and humanities research.

I would like to suggest that Element II be strengthened by requiring IRBs and ECs to base their evaluations on documented benefits and risks. Many committees demand precautions against risks that are largely chimerical, such as the possibility that an interview will re-traumatize victims of earlier trauma. As noted in the March 2008 issue of the Journal of Empirical Research in Human Research Ethics, empirical study suggests that such people are much more likely to be benefited than harmed by interviews. AAHRPP should insist that IRBs and ECs keep current with such literature and base their judgments on its findings. AAHRPP should also prohibit IRBs and ECs from judging proposals on irrelevant criteria, such as the number of typographical errors in a proposal.

The standards also provide a measure of accountability. Element I.5 insists that "based on objective data, the Organization identifies strengths and weaknesses of the Human Research Protection Program, makes improvements, when necessary, and monitors the effectiveness of the improvements." It also provides that "the Organization has and follows written policies and procedures so that Researchers and Research Staff may bring forward to the Organization concerns or suggestions regarding the Human Research Protection Program, including the ethics review process." Again, many universities currently fail to meet these standards.

I would be more encouraged by these standards if AAHRPP would pledge to enforce them. I suggest that as part of its system of maintaining accreditation, AAHRPP establish a mechanism by which researchers can bring violations of these elements directly to the attention of the association. Researchers at accredited institutions should be guaranteed that AAHRPP will investigate their complaints.

More significantly, I am disappointed to see that the proposed standards reject two of the most prominent suggestions for taming overregulation put forward by IRB critics: allowing researchers to apply clearly worded exemptions to their own research, and limiting the application of federal rules to federally-funded research. AAHRPP promises policies and procedures based on "objective criteria and measurable outcomes." What objective criteria suggest that researchers cannot apply the exemptions, or that federal regulations are necessary for all research?

Prominent members of the IRB community, including a former director of OHRP, and current members of SACHRP, have conceded that overregulation and hyper-protectionism are problems that must be addressed. Restoring the exemptions of 45 CFR 46 to their original intent, and allowing institutions to experiment with ways to oversee non-federally-funded research, are among the most constructive suggestions put forward to address these problems. The proposed standards reject these proposals out of hand. In doing so, they fail to meet AAHRPP's professed ideal of performance-based policies.

Finally, while you have not sought comment on your current Accreditation Principles, let me express my hope that you will find some room in them for the promotion of academic freedom. A university or research center that fails to incorporate this ideal into its human research protection system cannot be faithful to its core mission.

The Systematic Threat to Academic Freedom

Lisa Rasmussen kindly alerted me to her essay, "Problems with Minimal-Risk Research Oversight: A Threat to Academic Freedom?" IRB: Ethics & Human Research 31 (May 2009): 11-16. The essay mostly seeks to rebut the AAUP's 2006 report, "Research on Human Subjects: Academic Freedom and the Institutional Review Board." Rasmussen identifies some important shortcomings in that report, and she raises key questions about the relationship between IRBs and academic freedom. But I am unpersuaded by her central arguments.

Before I address them, I should note the repeated disclaimers within the essay. "I will not settle here the fundamental issue of whether a convincing argument exists that IRB review poses a threat to academic freedom," Rasmussen writes. "A longer explanation of the [AAUP report's] failures is beyond the scope of this paper, but a brief outline is possible." I am disappointed by these limits. Rasmussen devotes significant space to matters peripheral to the question of academic freedom, such as her assertion that researchers whose work was approved by a department--rather than a central IRB--would necessarily merit less legal protection, a claim whose weakness she acknowledges in a footnote. Given only six pages, Rasmussen would have done better to focus on the question posed in her title.

Rasmussen's main argument is that the AAUP report "does not demonstrate that IRBs pose a threat to academic freedom." As she notes, such a demonstration would require a definition of academic freedom, something lacking in the AAUP report. So she offers a passage from the AAUP's "1940 Statement of Principles on Academic Freedom and Tenure": "Institutions of higher education are conducted for the common good and not to further the interest of either the individual teacher or the institution as a whole. The common good depends upon the free search for truth and its free exposition." Emphasizing the grounding of this argument in the search for the "common good," Rasmussen then concludes that "there is a prima facie claim that research can be subjected to assessment regarding whether it threatens to harm the common good via harm to individuals."

I believe this is a misreading of the 1940 Statement, for it suggests that any policy aimed at safeguarding the common good is consistent with academic freedom. For example, she could have written, "there is a prima facie claim that research can be subjected to assessment regarding whether it threatens to harm the common good via the promotion of communist overthrow of the government," and that therefore a prohibition on the use of Marxist analysis is consistent with academic freedom.

A more relevant definition of academic freedom can be drawn from the AAUP's 1915 Declaration of Principles on Academic Freedom and Academic Tenure":

The liberty of the scholar within the university to set forth his conclusions, be they what they may, is conditioned by their being conclusions gained by a scholar’s method and held in a scholar’s spirit; that is to say, they must be the fruits of competent and patient and sincere inquiry, and they should be set forth with dignity, courtesy, and temperateness of language . . .

It is, however . . . inadmissible that the power of determining when departures from the requirements of the scientific spirit and method have occurred, should be vested in bodies not composed of members of the academic profession. Such bodies necessarily lack full competency to judge of those requirements; their intervention can never be exempt from the suspicion that it is dictated by other motives than zeal for the integrity of science; and it is, in any case, unsuitable to the dignity of a great profession that the initial responsibility for the maintenance of its professional standards should not be in the hands of its own members. It follows that university teachers must be prepared to assume this responsibility for themselves.

As Matthew W. Finkin and Robert C. Post write in their new book, For the Common Good: Principles of American Academic Freedom, freedom of research depends on "a framework of accepted professional norms that distinguish research that contributes to knowledge from research that does not." (54) While these two experts on academic freedom decline to offer a firm opinion on the legitimacy of IRBs, they take the AAUP's concerns far more seriously than does Rasmussen (69).

The question, then, is whether IRBs, like the boards of trustees that concerned the authors of the 1915 statement, "lack full competency to judge of [scholarly] requirements." Rasmussen suggests that IRBs merely maintain scholarly standards: "The source of the threat to academic freedom via oversight by one’s colleagues is far from clear," she writes, "especially since researchers undergo peer review for research funding and when submitting their manuscripts for publication." But IRB review is not peer review, since it is conducted mostly by people ignorant of the scholarly methods they are reviewing. (See "Why IRBs Are Not Peer Review," and other posts tagged "peer review.")

To make this a bit more concrete, we can examine the exemplary "horror stories" included in the 2006 AAUP report. Rasmussen rejects these as "unelaborated anecdotes with no documenting citations," rather than examining their implications for academic freedom.

Here's one: "A Caucasian PhD student, seeking to study career expectations in relation to ethnicity, was told by the IRB that African American PhD students could not be interviewed because it might be traumatic for them to be interviewed by the student." Or another: "A campus IRB attempted to deny an MA student her diploma because she did not obtain IRB approval for calling newspaper executives to ask for copies of printed material generally available to the public." No peer review process would impose such conditions. If these are not infringements of academic freedom, then nothing is.

Rasmussen is quite right that we should not equate "inconvenience and hassle with abridgement of academic freedom." Yet nor should we dismiss the abridgement of academic freedom as mere inconvenience and hassle. When IRBs impose conditions on research that prevent researchers from conducting the basic tasks of scholarship--talking to people of varied backgrounds, recording interviews, or telephoning for information--they abridge academic freedom. The more interesting questions are how often this occurs, and why it happens.

Rasmussen presents IRB abuse as a somewhat random process: "IRBs can function well or poorly, and which is true for a given IRB depends on many factors, not least of which are institutional support and member training." This suggests that IRB abuses are individual anomalies, rather than a pattern.

By contrast, the AAUP detects a systematic bias toward the infringement of freedom. This is better developed in the AAUP's 2000 report (cited by Rasmussen), "Institutional Review Boards and Social Science Research." That report includes such observations as "no one is likely to get into trouble for insisting that a research proposal is not exempt" and "no university is likely to want to explain to either the government or the public why its commitment to avoid harming the human subjects of research is limited by the source of funding for the research." In these and other cases, the AAUP recognizes that the IRB system punishes individuals and institutions only for approving research, not for restricting it.

The design flaws in the system have yielded a pattern of abuse. Read Maureen Fitzgerald and Laura Stark, both of whom observed repeated abuses by the IRBs they studied. Read Linda Thornton, whose work was thwarted at 15 of 24 institutions she contacted. Read Jack Katz, who shows that IRBs are particularly likely to pounce on controversial topics. IRBs can function well or poorly, but the system is weighted toward poor function.

Rasmussen acknowledges that poorly designed systems can lead to systematic problems. She concedes that the "lack of an [IRB] appeals process may threaten academic freedom." She also details the way that departmental-level review might systematically hamper research. And she ends her essay with a promising proposal for “template review:"

Disciplines at the national level might formulate templates to guide very common research approaches. For example, a research template for oral historians could stipulate that the researcher will interview individuals, record their answers, refer them to counselors if the questions have provoked strong emotions, procure consent forms, lock the transcripts securely, and identify what will happen to the transcripts at the close of research. IRBs at individual institutions would review the template once and approve it (or even decide to accept any templates from given professional societies). Thus, a researcher would simply submit a form to the IRB stating her agreement to abide by the format of the template. Upon receipt of the form, the IRB would approve the protocol.

If IRBs are not threatening academic freedom, why propose this reform? Inside this proposal is an acknowledgment that disciplinary experts and professional societies in the social sciences and humanities have been excluded from the present IRB system. While such exclusion does not automatically threaten academic freedom, we should not be surprised when it does. For all her skepticism of the AAUP report, Rasmussen has presented her own suggestion that the current system is rotten at the core.