Monday, May 28, 2007

Scott Atran, "Research Police – How a University IRB Thwarts Understanding of Terrorism"

Blogger's note: In March the Harvard Crimson mentioned an unpublished essay by ethnographer Scott Atran of the University of Michigan, detailing his complaints about the IRB process. With Dr. Atran's kind permission, I present the complete essay here. ZMS

Thursday, May 24, 2007

Even the Best IRB Can Antagonize Researchers

Judging from Samuel P. Jacobs's story, "Stern Lessons For Terrorism Expert," Harvard Crimson, March 23, 2007, the Harvard IRB is pretty darn good when it comes to non-biomedical research. Policy researcher Jessica E. Stern learned from the IRB to "not learn the names of many of the people she is interviewing—preferring to use pseudonyms—thus protecting the privacy of her interviewees and making her notes less valuable to federal investigators." She states, “Harvard’s IRB is the only one I know of to approve the kind of research I do. They’ve bent over backwards to make what I do possible, which is better than any other IRB.” Law professor Elizabeth Warren has “never encountered an IRB as helpful as Harvard’s."

Yet another researcher finds "the process is so cryptic and idiosyncratic" that "his students often can’t anticipate the reasons why the institutional review board will reject a proposal." And Stern herself, who got valuable help from the IRB, complains that "Before I came to Harvard, I had pretty remarkable interviews with terrorists . . . There are a lot of reasons that those kind of interviews would be hard today. One of them is the post-September 11 environment, but the other is the IRB strictures.” One project, to interview radical black Muslims, died entirely because of the delay in approval. (Note: this is just what Robert Kerr warned us about.)

How can we have the best of both worlds—helpful advice without arbitrary rejections and delays? Voluntary review.

Wednesday, May 23, 2007

Jeffrey Cohen: More Argument Without Evidence

In a new posting on his blog, Jeffrey Cohen writes,

One of the major propositions that the critics of IRB review of social research put forth is that minimal risk social research with competent adults should be completely exempt from IRB review. On the surface, this makes some sense. We're talking about research that is unlikely to harm anyone and where adults can decide for themselves whether to participate. Why do we need to review such research? Based on my experience personally reviewing thousands of research protocols in the social sciences, there is one basic problem with this - researchers are human beings. Human beings are not perfect - they overlook things, make mistakes and and can't be totally objective about their own work. If researchers were perfect, if they always took all of the ethical issues into account when planning and conducting their research, then we wouldn't need IRB review. But they are not perfect - none of us are perfect. So, every research activity needs an independent, objective review.

Characteristically, Cohen offers not a single example of a social research project whose ethical content was improved by his or any IRB review, much less one that could only be improved thanks to the broad definitions and coercive rules now used by IRBs. (And no, forcing interviewers to carry lists of mental-health centers doesn't count.) If he has such examples, he should offer them. If not, a vague reference to "thousands of research protocols" is unlikely to persuade a community of scholars trained to think critically and to weigh evidence.

Monday, May 21, 2007

Zywicki, "Institutional Review Boards as Academic Bureaucracies"

My colleague Todd J. Zywicki, of Mason's law school, has posted on SSRN his contribution to the forthcoming Northwestern Law Review issue on IRBs: "Institutional Review Boards as Academic Bureaucracies: An Economic and Experiential Analysis,"

As stated in the abstract, the article "argues that the problem is that IRBs are fundamentally bureaucracies, and that this bureaucratic structure explains much of their frequent suboptimal decision-making. The poor performance of IRBs is thus not a consequence of those individuals who comprise it, but rather a reflection of their bureaucratic nature. The bureaucratic nature of IRBs appears to do nothing to improve the decisions that they make, while being the source of many of their problems."

Zywicki is right to note that people who gain money, power, and prestige from controlling other people's work have an incentive to define their duties broadly, even as they fret about researchers' conflicts of interest. Ironically, he seems only dimly aware of the extent to which administrators have seized power from researchers. He writes, "IRBs are fundamentally bureaucracies," not understanding that IRBs themselves are committees of researchers, and that the true bureaucracies are the university compliance offices. He notes, "there is reported to be a growing IRB conference circuit," suggesting his unfamiliarity with PRIM&R and its administrator-dominated conferences. And he makes no mention of the creation, in 1999, of the "Certified IRB Professional" as a new kind of administrator, someone who has staked a great deal on the maintenance, if not expansion, of IRBs' reach.

But buried in the essay is another explanation of the IRBs' expansion:

With respect to university bureaucracies such as IRBs, at least some of the growth in their internal administrative burden has been spurred by governmental regulations. In addition, the preoccupation of IRBs with paperwork and forms has been promoted by a regime of “fear” of governmental oversight, “[f]ear by the institution that it will be ‘out of compliance’ with one or more aspects of the paperwork, and so subject to penalty upon audit (be that by the NIH, the Office for Human Research Protection, the US Department of Agriculture, or whatever other organization is involved).”

How much of the growth of IRB staffs is due to the internal dynamic Zywicki stresses compared to the growing external threat of federal penalty? One way to find out would be to compare the growth of compliance regimes to major OHRP enforcement actions. If as Zywicki notes, the Northwestern University Office for the Protection of Research subjects "grew from two full-time professionals in the late 1990s to 25 professionals and an administrative staff of 20 last year," I'd like to know how much of that growth took place as a response to the suspension of sponsored research at Hopkins, Virginia Commonwealth, and other universities.

Without far more research on the actual workings of IRBs and compliance offices around the country, we can't test hypotheses such as Zywicki's.

Saturday, May 19, 2007

A Trimmer PRIM&R

In my previous posting, I detailed the ways that the composition of PRIM&R's board and the operations of its conferences fail to match its stated mission, and I suggested what it would take for PRIM&R to live up to its claim to serve "the full array of individuals and organizations involved in biomedical and social science/behavioral/educational research." The other option, of course, would be for it to scale back its ambitions to serving merely "the full array of individuals and organizations involved in biomedical research," a noble task and one that its biomedical membership is far better equipped to handle.

PRIM&R's adopting this approach would mean recognizing that current regulations were written with biomedical research in mind and have never suited social science. It should therefore use its prestige to advocate broader exemptions from federal regulation for non-biomedical research. Such exemptions might exclude from IRB review:

* Research involving survey or interview procedures, where the subjects are legally competent, and where the investigator identifies himself/herself, and states that he/she is conducting a research survey or interview

* Research involving the observation (including observation by participants) of public behavior in places where there is no recognized expectation of privacy.


* Research involving the observation (including observation by participants) of public behavior in places where there is a recognized expectation of privacy, except where all of the following conditions exist:

(i) Observation are recorded in such a manner that the human subjects can be identified, directly or through identifiers linked to the subjects,

(ii) the observations recorded about the individual, if they became known outside the research, could reasonably place the subject at risk of criminal or civil liability or be damaging to the subject's financial standing or employability, and

(iii) the research deals with sensitive aspects of the subject's own behavior such as illegal conduct, drug use, sexual behavior, or use of alcohol.

I fear that Dr. Cohen will reject such proposed exemptions, taking them as evidence that I have "no interest in making IRB review work better for researchers, but only in eliminating IRB review for [my] research." Before he does so, let me point out that these exemptions are not my creations, but those of a joint subcommittee of the American Association of University Professors' Committee A on Academic Freedom and Tenure and Committee R on Government Relations. They were published as a report on Regulations Governing Research on Human Subjects, Academe, December 1981, 358-370.

A member of the subcommittee, and a signatory to the recommendations, was Sanford Chodosh, MD, the founder of PRIM&R.

Wednesday, May 16, 2007

PRIM&R's Public Responsibilities

Jeffrey Cohen writes, "I don't understand why you continue to criticize a program designed to explain and encourage the use of the flexibility available in the regulation and to promote communication and collaboration between IRBs and investigators to facilitate research." His comment shows that I have not made clear my concern with PRIM&R's constitution and operation. Let me suggest, therefore, that PRIM&R is a semi-official body that fails to conduct itself in a manner commensurate to its power and responsibility.

We first need to understand PRIM&R's special status as a chosen instrument of the federal government. Unlike scholarly associations, whose letters to OHRP get brushed off, PRIM&R has long functioned as an arm of OHRP and its predecessor, OPRR. For example,

  • OHRP promotes and distributes PRIM&R's "Investigator 101" CD-ROM. Similarly, PRIM&R prepared the first edition of the Department of Health and Human Services' IRB Guidebook.
  • Former OPRR head Charles McCarthy serves on the PRIM&R board; Cohen, a former OHRP official, co-organized the last conference; and current OHRP officials participate as PRIM&R conference faculty.
  • OHRP exhibits at PRIM&R conferences, and OHRP officials, particularly Michael Carome, use PRIM&R conferences to offer guidance that then gets broadcast by IRB consultants, such as Dr. Cohen.

Given these connections, an IRB member or staffer could reasonably assume that following PRIM&R's guidance is a good way to avoid sanctions by OHRP. (Indeed, she would be foolish to assume otherwise.) Since it these staffers who then return to their institutions and impose conditions on research there, PRIM&R is a key link in the chain between federal power and the daily work of researchers.

PRIM&R itself recognizes this function. For example, the recent conference provided attendees with documents offering guidance on the regulatory definition of human subjects research and the proper application of the exemptions. Since providing just such guidance is one of the responsibilities of OHRP, the conference organizers must feel pretty confident that they have OHRP's blessing to take over this important role. (Note that unlike real OHRP guidance, PRIM&R's documents are not made public.)

If this is so, what must PRIM&R do to use its power wisely and justly?

1. Identify its constituents

PRIM&R's website states that "since 1974, PRIM&R has served the full array of individuals and organizations involved in biomedical and social science/behavioral/educational research." But what constitutes that full array? Is "social science/behavioral/educational research" one category or three? If the latter, what disciplines come under which category? And are there varieties of research (such as folklore, nonfiction writing, and law) that fit none of those categories, yet that face IRB review?

I would like to see PRIM&R list all the disciplines that come under review by IRBs whose members it trains. Then it could try to make some distinctions, for example, between disciplines that offer therapy or advocacy and those that do not; disciplines that study the body and those that do not, disciplines that use formal scientific protocols and those that do not; and the like. This would lead to a second task:

2. Include a range of disciplinary perspectives

As I mentioned earlier, PRIM&R's board of directors is dominated by researchers and administrators from hospitals and medical schools. They cannot be expected to be expert in the ethics and methodologies of the full range of disciplines in the behavioral sciences, social sciences, humanities, and professions, nor could any group of scholars drawn from any one field. If PRIM&R wants to offer sound guidance to researchers in all fields subject to IRB review, it should include them on every level, from conference panels to editorial boards to the board of directors itself. The goal should be that each field has the chance to shape any guidance that affects that field, so PRIM&R does not again, for example, offer a conference panel on oral history with no historians present.

Since the vast bulk of IRB review does concern biomedical research, I would not expect equal numbers for researchers in other fields. But I do think that folklorists should have as much power to shape PRIM&R's advice on folklore as physicians have to shape PRIM&R's advice on medical research. If this requires a complex committee structure, so be it.

3. Include a range of viewpoints

Prior to the recent conference, Dr. Cohen wrote me, "if you would like to participate in the conference, we'd be happy to work you in to the program (provided you have something constructive to say)." Now he writes, "I can only conclude that you have no interest in making IRB review work better for researchers, but only in eliminating IRB review for your research."

Does that mean that someone who believes in wider exemptions from review has nothing constructive to say, and therefore no place in PRIM&R? This would contradict Cohen's earlier pride in having invited Linda Shopes, who has advocated an oral-history exemption far longer and more effectively than I. And it would contradict the above-mentioned panels at the last conference, which seem to assume that the proper scope of exemptions has yet to be determined.

I suggest that PRIM&R invite participants based on their knowledge, experience, and ability to represent their discipline, not their adherence to a party line, and that it make public its criteria for choosing participants in all of its endeavors. Perhaps I am not the right person to represent my field at PRIM&R conferences, but Taylor Atkins sure as heck isn't either.

4. Ease participation in conferences

Now I get to Dr. Cohen's specific invitation for suggestions for future conferences. I have three:

a. Don't schedule the conference during exam week. Dr. Cohen informs me that many researchers declined his invitation to participate in the conference. My guess would be that a big part of their refusal resulted from the fact that the second week of May is exam week around the nation--not a problem for federal officials, administrators, and consultants, but a big one for active teaching faculty. As it is, I am surprised he got three researchers from the University of Minnesota; it was exam week there too. October and November are pretty busy with conferences as well for many scholars. I suggest February or March might be more fruitful.

b. Announce the conference. I'm not sure how I found out about the 2007 SBER conference, but I know it was only a few weeks before the conference, and that I did not hear about it through any of my usual reading as a historian, such as H-ORALHIST, the list for oral historians. An open call for participants, months in advance, sent to scholarly newsletters and e-mail lists might boost participation.

c. Pay the costs of researchers. Maybe PRIM&R already pays the travel costs of conference faculty, but if not, it should. Most university professors are lucky to get funding to travel to one or two conferences a year, and they want to use this to attend conferences within their own disciplines. With PRIM&R charging hundreds of IRB staffers up to $950 to attend, it should be able to subsidize a few dozen researchers.

These last three suggestions might bring a few more researchers to the next PRIM&R conference, but I think the real problem is far deeper than participation at conferences. If, as Cohen suggests, PRIM&R truly seeks "to promote communication and collaboration between IRBs and investigators to facilitate research," it will have to work much harder to include the many investigators it has so long neglected.

Thursday, May 10, 2007

Boise State University's IRB Makes a Poor First Impression

I received an anonymous comment expressing alarm about the new website of Boise State University's Office of Research Administration.

The site presents the following information about the requirement of IRB review:

Federal, state and university regulations require all research (including surveys and questionnaires) involving human subjects or data collected, directly or through records (i.e. medical records, specimens, educational test results, or legal documents) to be reviewed by an Institutional Review Board (IRB) . . .

If you are a faculty or staff member, or student at Boise State University, and your research involves the use of human subjects (either directly or through records or other data such as specimens or autopsy materials), your research requires human subjects review.

"Research" is "a systematic investigation, including research development, testing, and evaluation, designed to develop or contribute to generalized knowledge." 45 CFR 46.102(d). Research includes surveys and interviews, behavioral investigations, retrospective reviews of medical information, experiments with blood and tissue, and demonstration and service programs and clinical trials. In addition, FDA includes under the definition of reviewable research, any use of a FDA regulated product except for use of a marketed product in the practice of medicine.

Note: Any administrative, departmental or course assignments involving surveys, questionnaires and interviews designed for internal use and operations of the University do not constitute "research" within the meaning of this policy if the information or conclusion of this data is not intended for scholarly publication or for dissemination to persons outside the administrative organization of the University.

That's it. No explanation of what federal and state regulations apply. No hint of the exceptions specified in 45 CFR 46.101, nor an easy way to learn more about the requirements.

The site does offer a link to the university's "Human Research Protection Policy - BSU 6325 B," but that link, as well as others on the site, is broken. (Yesterday I wrote to the e-mail address on the site, pointing out these broken links, but I have not received a reply). There's also a link to an "IRB Guideline Summary," which in turn offers a link to a Word document called "TYPES OF IRB REVIEW AND APPROVAL," which finally lists the exceptions. Since neither of these link titles mention exceptions, a visitor who knows enough to find this document probably knows about the exceptions already.

All told, my correspondent can be forgiven for fearing (I hope mistakenly) that Boise State "seems to require submission to IRB for analysis of any record of human behavior."

Boise State's Office of Research Administration shows how to antagonize researchers before even meeting them. I would like to remind all such offices that researchers are trained to read critically. Offer them complete and accurate information, and cite your sources.

Update: The links were fixed on May 15.

Monday, May 7, 2007

PRIM&R Finds Another Social Researcher

On April 13, I noted that the preliminary faculty list for this week's "Social, Behavioral, Educational Research Conference" included only one social researcher among twenty faculty. One of the conference's organizers, Jeffrey Cohen, replied that "the final list will have more social scientists." Well, the final list includes one more researcher: Steven Pennell, Survey Director, University of Michigan. Since the total list has been expanded to 33 faculty, this brings the representation of social researchers from five percent up to six percent. Many of the panels will feature no such representation at all.

The conference announcement asks, "Why is it often so difficult for IRBs and investigators to work together effectively when reviewing sophisticated, socially sensitive social/behavioral/educational research?" One answer might be PRIM&R's apparent belief that IRB administrators can learn to regulate research without hearing from researchers.