Can We Patch This Flat Tire?

The fourth article in the PS symposium is Felice J. Levine and Paula R. Skedsvold, “Where the Rubber Meets the Road: Aligning IRBs and Research Practice.” Both authors been involved in IRB debates for several years, and this article reflects their sophisticated understanding of some of the issues involved. But for an article published in a political science journal, it is disappointingly insensitive to the power dynamics that govern IRB-researcher relationships.

Unlike symposium participants Tony Porter, Dvora Yanow and Peregrine Schwartz-Shea, Levine and Skedsvold do not question the premise that IRBs help promote ethical research. Instead, they assert that there is no fundamental conflict between IRBs and social science researchers: "federal regulations, professional ethics codes, and research practice may have shared goals but tend to speak with different languages—creating frustration and skepticism in a system that could potentially work quite well if transformations are made." (502) Based on that assertion, they suggest four such transformations, ranging from the bold to the timid.

The Biomedical Ethics Juggernaut

The third contribution to the PS symposium is Tony Porter, "Research Ethics Governance and Political Science in Canada."

Porter laments that "the history of research ethics governance in Canada reveals recurrent concerns expressed by political scientists and other SSH [social sciences and humanities] researchers that indicate the inappropriateness of the [ethics] regime for SSH research, and that also create the impression that the regime is a juggernaut that continues on its trajectory, relatively impervious to criticism." (495)

Porter then offers a helpful capsule history of the debates leading up to Canada's present policy statements. From an American perspective, they look pretty good. In contrast to the Belmont Report, which calls for informed consent and harms-benefit assessment without specifying the types of research to which it applies, Canada's Tri-Council Policy Statement declares:

certain types of research— particularly biographies, artistic criticism or public policy research—may legitimately have a negative effect on organizations or on public figures in, for example, politics, the arts or business. Such research does not require the consent of the subject, and the research should not be blocked merely on the grounds of harms-benefits analysis because of the potentially negative nature of the findings. (496)

Unfortunately, Porter finds that in practice, research ethics boards ignore such guidance. For his own article, he was asked to specify questions in advance, destroy data, and write long explanations of his research plans. And he warns of even stricter regulation ahead.

Porter attributes the imposition of biomedical ethics and regulation on non-biomedical research to the clout that biomedical researchers have in government and universities. There are more of them, they have more money, and they care more about ethics--since they face more serious ethical challenges. As a result, "the growth of a biomedically oriented but unified research ethics regime has appeared as a seemingly unstoppable trend in Canada." (498) Rather dismally, Porter suggests that the only thing that will stop that trend is its own ability to alienate researchers until "opposition on the part of SSH researchers will increase and the legitimacy of the arrangements will be damaged, as will the ability of the regime to elicit the degree of voluntarism and acceptance that is needed to sustain it." (498)

Perhaps for lack of space, Porter does not consider another possibility: that the social sciences will internalize the medical ethics implicit in the "unified research ethics regime." The American Anthropological Association took a big step in this direction in 1998, with the adoption of a code of ethics that comes close to rejecting the idea that research "may legitimately have a negative effect on organizations or on public figures." If the ethics regime grows stronger in Canada and elsewhere, and more social scientists follow the AAA's line, it may be that young people interested in "critical research," as Porter puts it (496), will seek careers in journalism, rather than in university scholarship. To use a Canadian example, if Russel Ogden were writing for a newspaper, no one would be blocking his research.

Ideas on Fieldwork Are Oldies but Goodies

The second article in the PS symposium on IRBs is Dvora Yanow and Peregrine Schwartz-Shea, "Reforming Institutional Review Board Policy: Issues in Implementation and Field Research."

The authors argue that "the character of its implicit research design model, embedded in its historical development, . . . renders IRB policy problematic for ethnographic and other field researchers." (483) Specifically, they contend that ethnographers are likley to have trouble meeting IRB demands that their protocols spell out procedures for selecting subjects, obtaining informed consent, disgusing the identity of participants, balancing risks and benefits, and protecting the data they collect. (489)

Fieldwork, they argue, is just too unpredictable to be planned out so thoroughly in advance. They note,

Field researchers must enter others’ worlds, and are expected to do so with care and respect, and these worlds can be complex, unbounded, and in flux. Instead of rigidly delimited, predesigned protocols laying out research steps that are invariable with respect to persons and time, which subjects can be handed as they step into the world of the medical researcher, field research often requires flexing the research design to accommodate unanticipated persons and personalities and unforeseen conditions.

And, they find,

extending [the Belmont] principles to other, non-experimental research settings without making the underlying mode of science and its methodology explicit and without exploring their suitability to non-experimental scientific modes and methodologies has resulted in a hodgepodge of ethical guidance that is confused and confusing. Those guidelines do not give the many serious ethical problems of field research design and methodologies the sustained attention they deserve. (491)

All of this sounds perfectly sensible. What suprises me a bit is the authors' belief that they are the first to make these arguments:

The proposals that we have seen to date for reforming IRB policy (e.g., Carpenter 2007) all tinker with the existing system. None of them, to the best of our knowledge, has yet identified and engaged the underlying methodological frame—experimental research design—shaping that policy and its implementation. Policy reforms that address resource, organizational, and other features of the existing policy leave that framing and its prosecution in place. The impact of these policies on field research is, however, serious, extending IRB policy to these other forms of research in the absence of systematic evidence of their having harmed research participants. If we are to have policies to ensure the protection of human participants in all areas of research, those policies need to be suited to other than just experimental research designs in ways that are commensurate with their own potential for harms. It is vital that recognition of the misfit between existing experimentally based policy and field research design and methodologies also be on the table in discussions of IRB policy reform. (491)

In fact, ethnographers have been complaining about the imposition of experimental research ethics on non-experimental research for thirty or forty years. Anthropologist Murray Wax, in particular, eloquently distinguished experimental research from fieldwork in just the way that Yanow and Schwartz-Shea do. See, for example, his essay, "On Fieldworkers and Those Exposed to Fieldwork: Federal Regulations and Moral Issues," Human Organization 36 (Fall 1977): 321-28. Indeed, despite a long bibliography, Yanow and Schwartz-Shea cite none of the many IRB critiques written in 1978-1980, when the IRB regulations were being overhauled.

I don't fault Yanow and Schwartz-Shea too much for not knowing this history. It is one of the tasks of the historian to save others from having to reinvent the wheel, and I hope my book, when finished, will make such a contribution.

Yanow and Schwartz-Shea end their article with "A Call for Action," most of which is fairly vague. IRB critics are split between those who seek to "tinker with the existing system," and those who seek to exclude large categories of research from any IRB jurisdiction. Yet it's not even clear on which side of this divide these authors fall. For example, they want APSA to "Issue a statement calling for reform of IRB policy in a substantive way that protects the interests of APSA members." (492) Lovely, but what should such a statement say? They demand reform without defining it.

More promising is their call for more research. They note,

There is much that we do not know about the kind(s) of field research political scientists are doing today . . . We need more systematic, policy-oriented research about members’ field research practices, and we call on APSA to take the lead in conducting or facilitating it . . . (491)

They mention the possibility of an APSA handbook on ethical issues and current regulations.

This sounds a bit like the effort undertaken by the American Psychological Association in the preparation of its 1973 Ethical Principles in the Conduct of Research with Human Participants. As described in the first chapter of that book, rather than sit together and lay down some rules, the drafting committee surveyed the APA membership and assembled thousands of descriptions of real research projects that had raised ethical issues. The descriptions became the basis for an ethical guide directly relevant to the needs and values of the APA's members.

Around the same time, APSA itself undertook a similar effort, on a smaller scale, by conducting a study of actual cases in which researchers faced problems with confidentiality. Unfortunately, the full study seems not to have been published. A brief summary was published as James D. Carroll and Charles R. Knerr, "The APSA Confidentiality in Social Science Research Project: A Final Report," PS 9 (Autumn 1976): 416-419.

Whether or not a detailed ethical study would help ethnographic political scientists with their IRBs, it would be a great resource for scholars who want to do right by the people they study. I hope APSA--and other scholarly societies--will consider such a project.

Human Subject of Biomedical Research Angry!

Peter Klein at Organizations and Markets notes a brief dialogue concerning medical research ethics in The Incredible Hulk. Interestingly, the scientist involved suggests not the weighing of autonomy, beneficence, and justice demanded by the Belmont Report, but rather a prioritization of autonomy, allowing the subject, rather than an ethics committee, to decide whether the potential benefits justify the risks. Some ethicists of the 1970s proposed such a prioritization, but the National Commission rejected it.

The only movie I can think of off the top of my head, in which a comparable scene depicts an ethical debate in the social sciences and humanities, is Songcatcher. I haven't seen the movie, but even in the trailer they're arguing about when research becomes exploitation. Maybe I should watch the whole thing.

When Seligson is Non-Seligson

The first article in the July 2008 PS symposium is Mitchell A. Seligson's “Human Subjects Protection and Large-N Research: When Exempt is Non-Exempt and Research is Non-Research." While it's great to have someone interested in the contradictions of IRB regulations, the absurdity of the present regime seems to have left Seligson hopelessly confused, and his incoherent essay calls for both expansion and contraction of IRB authority.

Rather than trying to outline his argument, let me just list some of the questions to which he poses contradictory answers.

1. Should social science and humanities research follow the Belmont Report?

Early in his essay, Seligson attacks the Belmont Report as irrelevant to social science research, especially survey research. He particularly dislikes its call for an assessment of risks and benefits, noting

the problem of assessing risk is especially vexing for all of those who rely on large-N studies, typically in the field of survey research. Ironically, when only a handful of subjects are used in a campus laboratory-based experiment, the IRB is likely to approve the project with no objection. But survey research, which invariably relies on large-N studies, is viewed with suspicion by many IRBs simply because the risk, however small, is seen as being replicated 1,000 or more times, since most samples strive for confidence intervals of 63% or better. Protocol analysts, who are used to seeing laboratory experiments and focus groups with samples of fewer than 100, are often taken aback when they confront the large sample sizes inherent in most survey research. And when they do, they question why such a large sample is needed. As a result, it is not at all uncommon to have IRB protocol analysts ask survey researchers to cut down their sample sizes. (479)

He also is skeptical of the Common Rule, especially its protections for pregnant women--irrelevant and damaging to survey research. And he quotes--seemingly with approval--the AAUP's 2006 recommendation "that research whose methodology consists entirely of collecting data by surveys, conducting interviews, or observing behavior in public places be exempt from the requirement of IRB review.”

But then Seligson turns around, lamenting that "historians are not only exempt from IRB control, they have no requirement or even need to take human subjects protection training and pass tests on their knowledge of the principles and rules. Literature faculties often have no knowledge at all of human subjects protection." (480) He wants "faculty members in a broad range of institutions to familiarize themselves with the IRB regulations and to take the tests to demonstrate their knowledge of same," including "the Belmont principles." (482)

Why? Why should faculty members be required to familiarize themselves with guidelines that Seligson has told us are inapplicable to their work? Does he just want company in his misery?

2. Can researchers be trusted?

Seligson thinks that IRB regulations did not help survey research, because

Long before human subjects regulations and the invention of IRBs, survey researchers in all fields instinctually knew that by guaranteeing anonymity they would encourage frankness on the part of respondents. . . . Political scientists who carry out surveys have been aware for decades of the importance of guaranteeing anonymity to their subjects. (480)

If this track record weren't enough, he notes that governments and universities trust political scientists to behave ethically in other aspects of their work.

Even though political scientists conducting educational tests and surveys are exempt from federal regulation, they are not, after all, exempt because the federal government believes we cannot be trusted. What is so strange here is that in countless other important ways, we are trusted by that same federal government. When we grade tests taken by our students, we are not allowed to discriminate on the basis of race, creed, national origin, sexual preference, etc. Yet we are not asked to sign a statement saying that we will not discriminate before ~or indeed after! we grade each exam or before we determine final grades. We hold office hours, but are not asked to submit an application prior to each office hour, not even prior to the start of each term, to the affirmative action offices on our campuses that we will not sexually harass students. We submit articles to conferences but are not asked to submit signed statements saying that we did not plagiarize the material. (481)

Since political scientists have proven more or less trustworthy in these areas, Seligson wants IRBs "to stop assuming, . . that we are all guilty of violations of human subjects rights unless we can prove otherwise." (482)

That's all very nice, but he's unwilling to extend the trust to researchers in other fields. He writes,

some humanists may be naive about the risks involved in disclosing names of subjects. One can imagine many kinds of risk to respondents. One such risk is dismissal of employment from an employer who either might not like the views expressed in the oral history or testimonio or deems them harmful to the company’s welfare. Potential employers might look at the oral history information and deny a position based on the statements contained therein. Another risk could be ostracism at work or in one’s neighborhood for expressing politically unpopular views. One can even imagine law enforcement officials using oral histories to prosecute individuals for revelations that suggest criminal behavior. (480)

In other words, Seligson does not trust interview researchers to have the same instinctual knowledge of ethics he ascribes to survey researchers, he ignores oral historians' sixty-year record in favor of hypothetical abuses, and he assumes historians are guilty of violations of human subjects rights unless we can prove otherwise. Perhaps he wants us to get approval before grading tests as well.

3. Can IRBs be trusted?

Overall, Seligson takes a dim view of those in charge of human subjects regulations, whom he terms "overzealous bureaucrats, both federal and on campuses," and wants retrained. (482) He even relays the follwoing anecdote:

A very senior IRB official at one university, in order to impress upon a political science faculty member his omnipotence, asked, “Do you ever use the library to read books about President Bush?” When the response was affirmative, he said, “Unless you file for IRB approval before opening those books, you will be held in violation, since Bush is a human, is living, and the books almost certainly contain personal information.” (480)

I'm willing to believe a lot of bad things about IRBs, but even I can't swallow a story like this without names and dates attached.

Yet while portraying IRB officials as power-mad bureaucrats, Seligson wants to expand their jurisdiction "to cover all studies of any kind that obtain data on living humans." (482) Wouldn't that include a book about President Bush?

Seligson concludes that "the roadmap to the future should be clear." Maybe it should be, but this article isn't helping. Fortunately, the other essays in the symposium are better researched and reasoned.

Research Restrictions Not Confined to IRBs

John Mueller alerts me to Douglas Todd's article, "Academics Fight for B.C. Prof's Right to View Assisted Suicides," Vancouver Sun, 2 July 2008.

The article concerns the case of sociologist Russel Ogden, who studies assisted suicide. His employer, Kwantlen University College, has prohibited him from witnessing assisted suicides, according to the Canadian Association of University Teachers. The association wants scholars to have the opportunity to "understand politically unpopular behaviour."

The twist here is that the research ethics committee (the Canadian term for an IRB) is not to blame. They approved Ogden's research three years ago.

---

Update, July 7. Professor Mueller alerts me to further coverage by the National Post and Inside Higher Ed.

OHRP Seeks Comment on Training and Education Programs

Rob Townsend kindly alerted me to the July 1 announcement in the Federal Register that OHRP is seeking comments on its requirements for human subjects training for investigators and IRB members. The summary follows; the full text for the announcement is online at http://edocket.access.gpo.gov/2008/E8-14917.htm. The deadline for comments is September 29.


[Federal Register: July 1, 2008 (Volume 73, Number 127)] [Notices] [Page 37460-37463] From the Federal Register Online via GPO Access [wais.access.gpo.gov] [DOCID:fr01jy08-43] -----------------------------------------------------------------------
DEPARTMENT OF HEALTH AND HUMAN SERVICES
Request for Information and Comments on the Implementation of Human Subjects Protection Training and Education Programs AGENCY: Department of Health and Human Services, Office of the Secretary, Office of Public Health and Science, Office for Human Research Protections.

ACTION: Notice.

-----------------------------------------------------------------------

SUMMARY: The Office for Human Research Protections (OHRP), Office of Public Health and Science is seeking information and comments from affected entities and individuals about (a) Whether OHRP should issue additional guidance recommending that institutions engaged in human subjects research conducted or supported by the Department of Health and Human Services (HHS) implement training and education programs for certain individuals involved in the conduct, review, or oversight of human subjects research, or (b) whether HHS should develop a regulation requiring the implementation of such training and education programs. This request for information and comment stems from the 1998 report from the HHS Office of Inspector General (OIG) recommending that Federal requirements be enacted to help ensure that investigators and institutional review board (IRB) members be adequately educated about, and sensitized to, human subjects protections. More recently, the Secretary's Advisory Committee on Human Research Protections (SACHRP) recommended that OHRP require institutions to ensure that initial and continuing training is provided for IRB members and staff, investigators, and certain institutional officials. The implementation of such training and education programs might help to ensure that individuals involved in the conduct or review of human subjects research at institutions holding OHRP-approved Federalwide Assurances (FWAs) understand and meet their regulatory responsibilities for protecting human subjects.

DATES: Submit written or electronic comments by September 29, 2008.

ADDRESSES: You may submit comments by any of the following methods: E-mail: humansubjectstraining@hhs.gov. Include ``Human Subjects Protection Training and Education'' in the subject line. Fax: 301-402-2071. Mail/Hand delivery/Courier [For paper, disk, or CD-ROM submissions]: Michael A. Carome, M.D., Captain, U.S. Public Health Service, OHRP, 1101 Wootton Parkway, Suite 200, Rockville, MD 20852. Comments received within the public comment period, including any personal information, will be made available to the public upon request.

FOR FURTHER INFORMATION CONTACT: Michael A. Carome, M.D., Captain, U.S. Public Health Service, OHRP, 1101 Wootton Parkway, Suite 200, Rockville, MD 20852, 240-453-6900; e-mail Michael.Carome@hhs.gov.

Political Science Perspectives on IRBs

The July 2008 issue of PS, the journal of the American Political Science Association, offers a five-part symposium: "Protecting Human Research Participants, IRBs, and Political Science Redux." Over the next few days I plan to comment on each article, but for now it's safe to say that while each author offers a different diagnosis and prescription, none thinks that the current system is working well.

The Psychologist Who Would Be Journalist

Back in August 2007, I mentioned the controversy surrounding the book The Man Who Would be Queen (Washington: Joseph Henry Press, 2003) by J. Michael Bailey, Professor of Psychology, Northwestern University. At the time, Professor Alice Domurat Dreger, also of Northwestern, had just posted a draft article on the controversy. Now that article, along with twenty-three commentaries and a reply from Dreger, has appeared in the June 2008 issue of the Archives of Sexual Behavior.

Dreger's article, the commentaries, and Dreger's response focus on big questions about the nature of transsexuality, the definitions of science, power relationships in research, and the ground rules of scholarly debates. Only a handful take up the smaller question of whether—as a matter of law and as a matter of ethics--Bailey should have sought IRB approval prior to writing his book. But that's the question that falls within the scope of this blog.

Oral Historians Draw Conclusions, Inform Policy, and Generalize Findings

In the lead story of today's New York Times ("Occupation Plan for Iraq Faulted in Army History"), Michael R. Gordon reports on a new 700-page official history of the early occupation of Iraq, produed by the Army’s Combined Arms Center at Fort Leavenworth. As Gordon reports, "the study is based on 200 interviews conducted by military historians and includes long quotations from active or recently retired officers." He notes that "the study is an attempt by the Army to tell the story of one of the most contentious periods in its history to military experts — and to itself." It draws important conclusions with policy implications, finding, for example, that "the military means employed were sufficient to destroy the Saddam regime; they were not sufficient to replace it with the type of nation-state the United States wished to see in its place.”

This sounds suspiciously like the kind of project comprising generalizable research as defined by OHRP's Michael Carome in his October 2003 discussion with the UCLA Office for Protection of Research Subjects (as reported by UCLA.) In that conversation, Carome noted that

Systematic investigations involving open-ended interviews that are designed to develop or contribute to generalizable knowledge (e.g., designed to draw conclusions, inform policy, or generalize findings) WOULD constitute "research" as defined by HHS regulations at 45 CFR 46.

[Example]: An open ended interview of surviving Gulf War veterans to document their experiences and to draw conclusions about their experiences, inform policy, or generalize findings.

Except for the fact that it's the wrong Gulf War, the Army study nicely fits Carome's example of research requiring review.

Fortunately for federal historians, no one else in the federal government seems to share Carome's view on this matter. I know of no federal agency, executive or legislative, that requires IRB review for oral histories conducted by its employees. As reported on this blog, even OHRP officials did not submit to IRB review when conducting oral history research.

Maybe Dr. Carome will try to discipline the researchers at Fort Leavenworth. Him and what army?