On July 16 I attended the second day of the open meeting of the Secretary's Advisory Committee on Human Research Protections (SACHRP, pronounced sack-harp) in my home town of Arlington, Virginia. This was the first time I have observed such a meeting, and I am sure there is much I missed for want of context. But in this and following posts, I will record a few impressions.
The most interesting part of the meeting came at the end, when the committee's chair, Samuel Tilden, invited committee members to participate in "a systems level discussion" of today's human subjects protection regime. Not all committee members offered comments, and I was disappointed that anthropologist Patricia Marshall, the sole social scientist on the committee, did not do so. But the members who did speak displayed a range of viewpoints.
The most enthusiastic advocates of the status quo were Jeffrey Botkin and Daniel Nelson. Botkin described himself as an "unabashed advocate of current system." He noted that IRBs rose in response to documented abuses in medical research, such as those detailed by Henry Beecher in 1966 ["Ethics and Clinical Research," New England Journal of Medicine 274 (16 June 1966): 1354-1360]. Today, he noted, most researchers know the rules. While the system may let an occasional unethical project slip through, there is no "hidden underbelly of unethical research."
This is an important point, and I remain agnostic about whether IRBs are appropriate for medical research. But I am also sure that Dr. Botkin understands that even beneficial drugs can have nasty side effects, and that he would not prescribe the same drug to treat all ailments. I would be interested to know what he considers the social science analogue to Beecher's article. For if we are to judge today's system by its ability to avoid documented problems of the past, we need to know what we are trying to avoid for every type of research we regulate.
Nelson declared that the "Subpart A Subcommittee" he co-chairs decided early in its existence that "there is general consensus that the Common Rule is not 'broken.'" Yet in his system-level talk, he conceded that the power granted by the Common Rule to local IRBs results in arbitrary decisions (he called this "variability") and "well-intended overreaching." He noted that the only sure way to eliminate all risky research is to eliminate all research.
Other committee members, while not calling for changed regulations, were more explicit about current problems. Lisa Leiden, an administrator at the University of Texas, has heard from a lot of upset faculty, and she is looking for ways to relax oversight. This would include "unchecking the box," that is, declining to promise to apply federal standards to research not directly sponsored by a Common Rule agency. Without going into specifics, she suggested that the federal standards are too stringent, and that the University of Texas system, if freed from them, would craft exemptions beyond those now offered by the Common Rule. Overall, she is looking for ways to move from a "culture of compliance to one of conscience."
Liz Bankert, Nelson's co-chair of the subcommittee, also showed her awareness of the overregulation of social research, and her frustration with IRBs' emphasis on regulatory compliance. "I've gone to IRBs all over the country," she reported. "They are thoughtful, sincere, really intelligent groups. To have all this brainpower sucked into the vortex of minimal risk research is not efficient." It also contributes to what Bankert sees as a lack of mutual respect between IRBs and reseachers. She blamed the problems on a "fear factor which has been developing over the past several years."
Both Leiden and Bankert implied that it was the interpretation of the regulations, not the regulations themselves, that caused the problems they have identified. Without saying so explicitly, they seemed to blame the OPRR of the late 1990s for scaring IRBs all over the country into letter-perfect regulatory compliance, at the expense of research ethics.
In contrast, two committee members seemed willing to reconsider the regulations themselves. David Strauss hoped for a system that was "clinically and empirically informed," terms that no one could apply to the regulation of social research. And he recognized that the regulations are not divine revelation. "We shouldn't be reviewing research that we don't think needs to be reviewed because some folks 30 years ago, at the end of a long, hot day, decided to use the word 'generalizable,'" he explained. "We have to have language that makes sense to us."
Finally, Tilden himself described the Common Rule as largely broken. He noted that the 1981 regulations--which have changed only slightly since--were accompanied by the promise that most social research would not have to undergo IRB review. The fact that so few social science projects escape review, he concluded, showed that the exemption system has collapsed. Rather than try to shore it up again, he suggested that concerns about confidentiality be separated from other risks, and that projects whose only risks involved breaches of confidentiality be evaluated only for the adequacy of their protections in that area.
This last proposal interests me, because when scholars talk seriously about the wrongs committed by social science researchers, they almost always come back to questions of confidentiality. If IRBs were restrained from making up other dangers--like interview trauma--and instead limited to more realistic concerns, they could potentially do some good.
In sum, I did not get the impression that, in Nelson's words, "there is general consensus that the Common Rule is not 'broken.'" Strauss and Tilden, in particular, seem to understand that the present system has wandered far from the stated intentions of the authors of the regulations, and from any empirical assessment of the risks of research or the effectiveness of IRBs. I hope they will continue to think about alternative schemes that would keep controls on medical experimentation without allowing federal and campus officials free rein to act on their fears.
Wednesday, July 23, 2008
Thursday, July 17, 2008
Political Scientists to the Rescue?
The final essay in the PS symposium is Sue Tolleson-Rinehart, "A Collision of Noble Goals: Protecting Human Subjects, Improving Health Care, and a Research Agenda for Political Science."
Tolleson-Rinehart addresses the question of quality improvement in health care, such as the checklist publicized by Atul Gawande. As she notes, her "essay is not about the influence of IRBs on political science research" and is therefore largely outside the scope of this blog. That said, she makes some observations relevant to the regulation of social science research.
While sympathetic to individual IRB members, Tolleson-Rinehart takes a dim view of the system as it now operates:
She adds that while her own students submit "substantially similar" protocols, their
This is damning enough, but Tolleson-Rinehart insists that the "point of this vignette is not to criticize IRBs." (509) Rather, she argues that
Finally, Tolleson-Rinehart notes that while the Association for the Accreditation of Human Research Protection Programs and Public Responsibility in Medicine and Research mean well, both emphasize regulatory compliance over actual research ethics. She argues that political scientists can go beyond compliance questions to work on "a common epistemology of the philosophical, ethical, and political foundations of human subjects research." (510)
All of this sounds fine, and I hope that Tolleson-Rinehart and her colleagues get to work on her agenda. But as my recent exchange with Levine and Skedsvold suggests, the most immediate question for political scientists may be to figure out how to make the regulatory system more responsive to developments in research. We seem stuck with a 1974 law and 1991 regulations that cannot be changed, even when everyone agrees they need updating.
Tolleson-Rinehart addresses the question of quality improvement in health care, such as the checklist publicized by Atul Gawande. As she notes, her "essay is not about the influence of IRBs on political science research" and is therefore largely outside the scope of this blog. That said, she makes some observations relevant to the regulation of social science research.
While sympathetic to individual IRB members, Tolleson-Rinehart takes a dim view of the system as it now operates:
IRBs are understandably, and necessarily, driven by their limiting case: the possibility of invasive or dangerous procedures performed with vulnerable individuals or populations, without adequate regard for the hallmarks of protection, respect for persons, beneficence and nonmaleficence, and justice elucidated in what is widely known as the Belmont Report, and adopted from the time of the Belmont Report’s release as our fundamental ethical principles. It is not surprising, given IRBs’ role as the protector of the vulnerable, that the general IRB perspective on “minimal risk” and risk-benefit comparisons is a conservative one.
IRBs are also terrified. All IRB professionals I know work in real fear that their IRBs could be the next ones to cause an entire university’s research to be shut down. Shutdowns in recent years at Harvard, Duke, and Johns Hopkins give IRBs every reason to be fearful of making what OHRP considers to be an error. The reasonable suspicion that researchers regard IRBs as obstacles to, rather than facilitators of, research, must further IRB professionals’ sense of being embattled.
Reviewers on IRB committees are our very hardworking colleagues who are usually not given adequate release time to meet their committee responsibilities, and who are not able to benefit from truly extensive and nationally standardized training, nor do they have anything like a consensus metric for evaluating the spectrum of risk in different research contexts and for different populations.
All these sources of strain might determine the conservative approach to human subject protections. When social science research (including quality-improvement research) occurs in a biomedical context, or when health care and health policy require evaluation, the conservative stance can become dysfunctional. IRB assessments of my own students’ work provide a clear example of one of the ironic and unintended consequences of the absence of agreed upon and broadly understood metrics for assigning risk in different research contexts. IRBs have a comparative lack of familiarity with how social science methods— such as those used in quality-improvement research—may differ from some other methods of clinical research in the risks they pose to subjects. (508)
She adds that while her own students submit "substantially similar" protocols, their
IRB determinations range from declarations that the project is “not human subjects research” at all to “research that requires full consent,” with every intermediate determination. I frequently have students working simultaneously on very similar projects, one of whom must go through a tedious consenting process taking as much as four to five minutes of the beginning of telephone interviews (with the busy elites at the other end chafing at these long and unnecessary prefaces to the first question), while another student researcher is not required to secure any kind of consent at all. The single source of variation across these cases is not the research question or the method, but the IRB reviewers and their familiarity ~or lack thereof ! with in-depth interviewing and the unique protections already available, via one’s position, to the “powerful research subject.” (509)
This is damning enough, but Tolleson-Rinehart insists that the "point of this vignette is not to criticize IRBs." (509) Rather, she argues that
political science is well prepared to analyze and make normative (but evidence-based) recommendations about the politics of human subjects research. We can help define what it is, and the circumstances under which it is generalizable knowledge, even though it may not necessitate a conservative approach to protections. We can construct frameworks to achieve a more precise understanding of how to balance risks and benefits. Those frameworks might even lead us to formulate what would amount to a multidimensional risk scale. Finally, political science can contribute to the construction of theoretical and methodological underpinnings for the content of truly national standards for IRB training curricula. These would improve IRB reviewers’ understanding of different research methods to go beyond mere compliance with federal regulations and become real resources and decision aids for hard-pressed reviewers who may have to evaluate research they aren’t familiar with. (509)
Finally, Tolleson-Rinehart notes that while the Association for the Accreditation of Human Research Protection Programs and Public Responsibility in Medicine and Research mean well, both emphasize regulatory compliance over actual research ethics. She argues that political scientists can go beyond compliance questions to work on "a common epistemology of the philosophical, ethical, and political foundations of human subjects research." (510)
All of this sounds fine, and I hope that Tolleson-Rinehart and her colleagues get to work on her agenda. But as my recent exchange with Levine and Skedsvold suggests, the most immediate question for political scientists may be to figure out how to make the regulatory system more responsive to developments in research. We seem stuck with a 1974 law and 1991 regulations that cannot be changed, even when everyone agrees they need updating.
Monday, July 14, 2008
Can We Patch This Flat Tire?
The fourth article in the PS symposium is Felice J. Levine and Paula R. Skedsvold, “Where the Rubber Meets the Road: Aligning IRBs and Research Practice.” Both authors been involved in IRB debates for several years, and this article reflects their sophisticated understanding of some of the issues involved. But for an article published in a political science journal, it is disappointingly insensitive to the power dynamics that govern IRB-researcher relationships.
Unlike symposium participants Tony Porter, Dvora Yanow and Peregrine Schwartz-Shea, Levine and Skedsvold do not question the premise that IRBs help promote ethical research. Instead, they assert that there is no fundamental conflict between IRBs and social science researchers: "federal regulations, professional ethics codes, and research practice may have shared goals but tend to speak with different languages—creating frustration and skepticism in a system that could potentially work quite well if transformations are made." (502) Based on that assertion, they suggest four such transformations, ranging from the bold to the timid.
The first suggestion is the boldest: establishing IRBs at the level of the individual department or research unit. These departmental IRBs would still meet the requirements of 45 CFR 46, but they would include--and presumably be led by--researchers familiar with the methods under review. As I've written before, for those projects that can benefit from prospective review, I do like the idea of putting the responsibility for review in the hands of scholars who know something about the proposals they encounter.
But Levine and Skedsvold seem naive when they suggest that "the regulations still provide latitude for institutions interested in developing new models for a local human research protection system to do so." (502) Reading the regulations in the absence of OHRP actions and policy statements doesn't tell you much about the real requirements for IRBs. For example, the regulations do not require that a quorum be documented for every IRB action item, but OHRP does. [Norman Fost and Robert J. Levine, "The Dysregulation of Human Subjects Research," JAMA 298 (14 November 2007), 2196.] That kind of picky demand can only be met by expert staff, which tends to move power away from IRB members and toward the administrators who have the time to keep up with all the rules. It also makes it harder to establish multiple IRBs.
Particularly painful is the article's pseudo-historical claim that "in contrast to when IRBs were first established, colleges and universities today are larger and more complex organizations with many more human subjects protocols to review," and therefore decentralization is more needed today than in the past. (503) Nonsense. When the Department of Health, Education, and Welfare first imposed IRB requirements for a wide range of research in the early 1970s, the fields of anthropology, political science, psychology, sociology, and the like were as varied as they are today, with differing ethical codes and methodological practices. To accommodate this diversity, some universities sought just the kind of department-level review that Levine and Skedsvold now propose. But OPRR (OHRP's predecessor) crushed that effort, insisting that all these fields be lumped into a single "behavioral" category. As Donald Chalkley, OPRR's director, put it in 1976,
Would today's OHRP act differently? Here's hoping.
The second suggestion is modest, but still important. Levine and Skedsvold suggest that IRBs be required to report on their efficiency at handling expedited reviews:
While it seems a pity to add another layer of paperwork to the IRB process, quantifying IRB performance might be a start toward IRB accountability. As Atul Gawande has advised, "if you count something interesting to you, I tell you: you will find something interesting."
It's important, however, to find ways to keep IRBs from gaming the statistics. IRBs have killed a lot of projects--by delaying approval until the researcher gives up, or forcing the researcher to modify the proposal beyond recognition--without outright rejecting them. I suggest, then, that the statistics proposed here are only the beginning. Indeed, if we are serious about IRB accountability, we need several measures of effectiveness. To date, we don't have one.
The third suggestion is that IRBs stop reviewing studies that use "public use data files," which contain "only data collected in anonymous form or that has been stripped of direct and indirect identifiers." The authors suggest a certification scheme so that "local IRBs would no longer need to determine that the use of the file by an investigator meets the criteria for exemption from IRB review." (503)
This sounds sensible enough, though I'd like to get more details about what kinds of files would need such certification. As I reported earlier, UCLA believes that its researchers need IRB permission to read blogs (UCLA's Policy 42 includes specific instructions for how to get permission to read a blog, or a letter to the editor.) I'd like Levine and Skedsvold to elaborate definitions to avoid this kind of hyper-regulation.
The final suggestion is the most timid, and the least helpful. Levine and Skedsvold suggest education as a panacea:
This reminds me of earlier calls for simple cooperation between IRBs and researchers, for example, Robert Cleary's 1987 suggestion:
But such calls for communication and discussion ignore the profound differences that divide IRBs and social researchers. Pick any horror story you want--Tony Porter's, Tara Star Johnson's, Scott Atran's--and you'll find a researcher who knows what she's doing confronting an IRB that does not. IRBs cannot educate investigators about the overall system because they don't know very much about the overall system--if the "overall system" is taken to include the ethics and methods of various branches of the social sciences and humanities. Nor do they have much incentive to learn anything.
I refer Levine and Skedsvold to the ethnographic studies of IRBs by Laura Stark and Maureen Fitzgerald. Both found that IRBs' public policies had little bearing on their actual decision processes. An IRB that holds a monthly open meeting to expound on fundamental ethical principles, then retreats behind closed doors to reject proposals based on spelling errors, is not going to instill the kind of respect among researchers that Levine and Skedsvold want.
An alternative would be to require that IRBs document their reasons for decisions on each proposal, creating what Jack Katz has called a system of "legality." Katz explains:
Katz presents legality primarily as a way to protect researchers against IRB censorship, but it would also be an excellent way to provide the kind of education Levine and Skedsvold want. For example, in the early 1970s, the Berkeley IRB assembled a handbook based on 2500 cases it had decided. By showing in concrete, not hypothetical, terms what the IRB considered to require review and what qualities it looked for in a proposal, the handbook both empowered and educated Berkeley researchers. But the handbook was part of a broader effort that excused some researchers from IRB review, and Chalkley's OPRR killed that too.
The authors acknowledge that IRB overregulation emanates from Bethesda, and they conclude that the ball is in OHRP's court.
I agree that OHRP is crucial here; one cannot expect local IRBs--vulnerable as they are to OHRP's wrath--to stick their necks out with new initiatives. But how likely is OHRP to lead the cause of reform?
As Levine and Skedsvold concede, OHRP has ignored earlier calls--going back to 2002--for some of the very reforms they advocate. The authors are silent on why that should be, or why we should expect better treatment now. OHRP's record of ignoring social scientists' calls for reform challenges the overall assumption in this article that OHRP is interested in helping social scientists with their research. Here's a case where they could have used some of the more skeptical perspective of Tony Porter.
Let's face it: the regulation of medical research is a lot more important, in lives and dollars, than the kinds of work that Levine, Skedsvold, and I care about. Whatever the regulations say, whatever "flexibility" they offer, we can expect OHRP to keep its eye on medical research, and to issue rules solely with medical research in mind. Serious reform may take much bolder restructuring than Levine and Skedsvold admit.
Unlike symposium participants Tony Porter, Dvora Yanow and Peregrine Schwartz-Shea, Levine and Skedsvold do not question the premise that IRBs help promote ethical research. Instead, they assert that there is no fundamental conflict between IRBs and social science researchers: "federal regulations, professional ethics codes, and research practice may have shared goals but tend to speak with different languages—creating frustration and skepticism in a system that could potentially work quite well if transformations are made." (502) Based on that assertion, they suggest four such transformations, ranging from the bold to the timid.
Decentralizing the IRB
The first suggestion is the boldest: establishing IRBs at the level of the individual department or research unit. These departmental IRBs would still meet the requirements of 45 CFR 46, but they would include--and presumably be led by--researchers familiar with the methods under review. As I've written before, for those projects that can benefit from prospective review, I do like the idea of putting the responsibility for review in the hands of scholars who know something about the proposals they encounter.
But Levine and Skedsvold seem naive when they suggest that "the regulations still provide latitude for institutions interested in developing new models for a local human research protection system to do so." (502) Reading the regulations in the absence of OHRP actions and policy statements doesn't tell you much about the real requirements for IRBs. For example, the regulations do not require that a quorum be documented for every IRB action item, but OHRP does. [Norman Fost and Robert J. Levine, "The Dysregulation of Human Subjects Research," JAMA 298 (14 November 2007), 2196.] That kind of picky demand can only be met by expert staff, which tends to move power away from IRB members and toward the administrators who have the time to keep up with all the rules. It also makes it harder to establish multiple IRBs.
Particularly painful is the article's pseudo-historical claim that "in contrast to when IRBs were first established, colleges and universities today are larger and more complex organizations with many more human subjects protocols to review," and therefore decentralization is more needed today than in the past. (503) Nonsense. When the Department of Health, Education, and Welfare first imposed IRB requirements for a wide range of research in the early 1970s, the fields of anthropology, political science, psychology, sociology, and the like were as varied as they are today, with differing ethical codes and methodological practices. To accommodate this diversity, some universities sought just the kind of department-level review that Levine and Skedsvold now propose. But OPRR (OHRP's predecessor) crushed that effort, insisting that all these fields be lumped into a single "behavioral" category. As Donald Chalkley, OPRR's director, put it in 1976,
There were several questions with regard to the use of sub-committees by Institutional Review Boards. We have encouraged it, we have discouraged it. We have discouraged it when the tendency was to put a sub-committee in every department. We beat Ohio State out in that. And we have encouraged it when it was obvious to us that a board that had begun primarily as a medical board, was not capable of dealing with behavioral research and things of this sort. In fact, our current listing in general assurances distinguishes between those institutions which are capable of dealing with medical subjects, and those that are capable of dealing only with the behavioral. [National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, Transcript, Meeting #24, November 1976, 175].
Would today's OHRP act differently? Here's hoping.
Simplifying and Expediting Expedited Review
The second suggestion is modest, but still important. Levine and Skedsvold suggest that IRBs be required to report on their efficiency at handling expedited reviews:
Through the FWA, institutions could describe the expected timeframe for completing expedited review and provide an annual report regarding, for example, the number of applications submitted and approved for expedited review, the number that were ultimately taken to full review, and the time span between filing for expedited review and action by the expedited review official. (503)
While it seems a pity to add another layer of paperwork to the IRB process, quantifying IRB performance might be a start toward IRB accountability. As Atul Gawande has advised, "if you count something interesting to you, I tell you: you will find something interesting."
It's important, however, to find ways to keep IRBs from gaming the statistics. IRBs have killed a lot of projects--by delaying approval until the researcher gives up, or forcing the researcher to modify the proposal beyond recognition--without outright rejecting them. I suggest, then, that the statistics proposed here are only the beginning. Indeed, if we are serious about IRB accountability, we need several measures of effectiveness. To date, we don't have one.
Limiting Review of Public Use Data Files
The third suggestion is that IRBs stop reviewing studies that use "public use data files," which contain "only data collected in anonymous form or that has been stripped of direct and indirect identifiers." The authors suggest a certification scheme so that "local IRBs would no longer need to determine that the use of the file by an investigator meets the criteria for exemption from IRB review." (503)
This sounds sensible enough, though I'd like to get more details about what kinds of files would need such certification. As I reported earlier, UCLA believes that its researchers need IRB permission to read blogs (UCLA's Policy 42 includes specific instructions for how to get permission to read a blog, or a letter to the editor.) I'd like Levine and Skedsvold to elaborate definitions to avoid this kind of hyper-regulation.
Enhancing the Educative Function of IRBs
The final suggestion is the most timid, and the least helpful. Levine and Skedsvold suggest education as a panacea:
Local IRBs could sponsor a monthly open meeting to answer questions about federal regulations, local policies, and/or issues relating to specific protocols. Developing opportunities for education and advisement could assist institutions in creating a positive climate for improving human research protections. While educating investigators about the overall system, IRB members and investigators can discuss ideas about, for example, reducing risk or enhancing confidentiality protections as researchers are developing protocols. (503)
This reminds me of earlier calls for simple cooperation between IRBs and researchers, for example, Robert Cleary's 1987 suggestion:
Fundamentally, [IRBs’] work aims at the heightening of sensitivity among researchers to the need to protect human subjects. Thus the main problems involving the work of IRBs in political science seem to be perceptual and informational, rather than regulatory. Fortunately, these are the kinds of problems that people of good will can solve. And in my opinion the positive results for the protection of human subjects make it worthwhile to try! [Robert E. Cleary, "The Impact of IRBs on Political Science Research," IRB: Ethics and Human Research 9 (May-June 1987), 10.]
But such calls for communication and discussion ignore the profound differences that divide IRBs and social researchers. Pick any horror story you want--Tony Porter's, Tara Star Johnson's, Scott Atran's--and you'll find a researcher who knows what she's doing confronting an IRB that does not. IRBs cannot educate investigators about the overall system because they don't know very much about the overall system--if the "overall system" is taken to include the ethics and methods of various branches of the social sciences and humanities. Nor do they have much incentive to learn anything.
I refer Levine and Skedsvold to the ethnographic studies of IRBs by Laura Stark and Maureen Fitzgerald. Both found that IRBs' public policies had little bearing on their actual decision processes. An IRB that holds a monthly open meeting to expound on fundamental ethical principles, then retreats behind closed doors to reject proposals based on spelling errors, is not going to instill the kind of respect among researchers that Levine and Skedsvold want.
An alternative would be to require that IRBs document their reasons for decisions on each proposal, creating what Jack Katz has called a system of "legality." Katz explains:
Legality changes the interaction environment of decisionmaking by creating a series of processes in which the reviewed become capable of examining and publicly criticizing the review to which they are subjected, both on a retail, case-by-case basis, and on a wholesale, policymaking level. [Jack Katz, "Toward a Natural History of Ethical Censorship," Law & Society Review 41 (December 2007), 805.]
Katz presents legality primarily as a way to protect researchers against IRB censorship, but it would also be an excellent way to provide the kind of education Levine and Skedsvold want. For example, in the early 1970s, the Berkeley IRB assembled a handbook based on 2500 cases it had decided. By showing in concrete, not hypothetical, terms what the IRB considered to require review and what qualities it looked for in a proposal, the handbook both empowered and educated Berkeley researchers. But the handbook was part of a broader effort that excused some researchers from IRB review, and Chalkley's OPRR killed that too.
OHRP's Role
The authors acknowledge that IRB overregulation emanates from Bethesda, and they conclude that the ball is in OHRP's court.
By acknowledging that there are multiple ways to protect human research participants within the parameters of the federal regulations, OHRP would provide needed reassurance for institutions and investigators. Furthermore, to promote new ideas in this area, OHRP could develop a call for reform models and thereby signal to institutions its support for change. (504)
I agree that OHRP is crucial here; one cannot expect local IRBs--vulnerable as they are to OHRP's wrath--to stick their necks out with new initiatives. But how likely is OHRP to lead the cause of reform?
As Levine and Skedsvold concede, OHRP has ignored earlier calls--going back to 2002--for some of the very reforms they advocate. The authors are silent on why that should be, or why we should expect better treatment now. OHRP's record of ignoring social scientists' calls for reform challenges the overall assumption in this article that OHRP is interested in helping social scientists with their research. Here's a case where they could have used some of the more skeptical perspective of Tony Porter.
Let's face it: the regulation of medical research is a lot more important, in lives and dollars, than the kinds of work that Levine, Skedsvold, and I care about. Whatever the regulations say, whatever "flexibility" they offer, we can expect OHRP to keep its eye on medical research, and to issue rules solely with medical research in mind. Serious reform may take much bolder restructuring than Levine and Skedsvold admit.
Friday, July 11, 2008
The Biomedical Ethics Juggernaut
The third contribution to the PS symposium is Tony Porter, "Research Ethics Governance and Political Science in Canada."
Porter laments that "the history of research ethics governance in Canada reveals recurrent concerns expressed by political scientists and other SSH [social sciences and humanities] researchers that indicate the inappropriateness of the [ethics] regime for SSH research, and that also create the impression that the regime is a juggernaut that continues on its trajectory, relatively impervious to criticism." (495)
Porter then offers a helpful capsule history of the debates leading up to Canada's present policy statements. From an American perspective, they look pretty good. In contrast to the Belmont Report, which calls for informed consent and harms-benefit assessment without specifying the types of research to which it applies, Canada's Tri-Council Policy Statement declares:
Unfortunately, Porter finds that in practice, research ethics boards ignore such guidance. For his own article, he was asked to specify questions in advance, destroy data, and write long explanations of his research plans. And he warns of even stricter regulation ahead.
Porter attributes the imposition of biomedical ethics and regulation on non-biomedical research to the clout that biomedical researchers have in government and universities. There are more of them, they have more money, and they care more about ethics--since they face more serious ethical challenges. As a result, "the growth of a biomedically oriented but unified research ethics regime has appeared as a seemingly unstoppable trend in Canada." (498) Rather dismally, Porter suggests that the only thing that will stop that trend is its own ability to alienate researchers until "opposition on the part of SSH researchers will increase and the legitimacy of the arrangements will be damaged, as will the ability of the regime to elicit the degree of voluntarism and acceptance that is needed to sustain it." (498)
Perhaps for lack of space, Porter does not consider another possibility: that the social sciences will internalize the medical ethics implicit in the "unified research ethics regime." The American Anthropological Association took a big step in this direction in 1998, with the adoption of a code of ethics that comes close to rejecting the idea that research "may legitimately have a negative effect on organizations or on public figures." If the ethics regime grows stronger in Canada and elsewhere, and more social scientists follow the AAA's line, it may be that young people interested in "critical research," as Porter puts it (496), will seek careers in journalism, rather than in university scholarship. To use a Canadian example, if Russel Ogden were writing for a newspaper, no one would be blocking his research.
Porter laments that "the history of research ethics governance in Canada reveals recurrent concerns expressed by political scientists and other SSH [social sciences and humanities] researchers that indicate the inappropriateness of the [ethics] regime for SSH research, and that also create the impression that the regime is a juggernaut that continues on its trajectory, relatively impervious to criticism." (495)
Porter then offers a helpful capsule history of the debates leading up to Canada's present policy statements. From an American perspective, they look pretty good. In contrast to the Belmont Report, which calls for informed consent and harms-benefit assessment without specifying the types of research to which it applies, Canada's Tri-Council Policy Statement declares:
certain types of research— particularly biographies, artistic criticism or public policy research—may legitimately have a negative effect on organizations or on public figures in, for example, politics, the arts or business. Such research does not require the consent of the subject, and the research should not be blocked merely on the grounds of harms-benefits analysis because of the potentially negative nature of the findings. (496)
Unfortunately, Porter finds that in practice, research ethics boards ignore such guidance. For his own article, he was asked to specify questions in advance, destroy data, and write long explanations of his research plans. And he warns of even stricter regulation ahead.
Porter attributes the imposition of biomedical ethics and regulation on non-biomedical research to the clout that biomedical researchers have in government and universities. There are more of them, they have more money, and they care more about ethics--since they face more serious ethical challenges. As a result, "the growth of a biomedically oriented but unified research ethics regime has appeared as a seemingly unstoppable trend in Canada." (498) Rather dismally, Porter suggests that the only thing that will stop that trend is its own ability to alienate researchers until "opposition on the part of SSH researchers will increase and the legitimacy of the arrangements will be damaged, as will the ability of the regime to elicit the degree of voluntarism and acceptance that is needed to sustain it." (498)
Perhaps for lack of space, Porter does not consider another possibility: that the social sciences will internalize the medical ethics implicit in the "unified research ethics regime." The American Anthropological Association took a big step in this direction in 1998, with the adoption of a code of ethics that comes close to rejecting the idea that research "may legitimately have a negative effect on organizations or on public figures." If the ethics regime grows stronger in Canada and elsewhere, and more social scientists follow the AAA's line, it may be that young people interested in "critical research," as Porter puts it (496), will seek careers in journalism, rather than in university scholarship. To use a Canadian example, if Russel Ogden were writing for a newspaper, no one would be blocking his research.
Monday, July 7, 2008
Ideas on Fieldwork Are Oldies but Goodies
The second article in the PS symposium on IRBs is Dvora Yanow and Peregrine Schwartz-Shea, "Reforming Institutional Review Board Policy: Issues in Implementation and Field Research."
The authors argue that "the character of its implicit research design model, embedded in its historical development, . . . renders IRB policy problematic for ethnographic and other field researchers." (483) Specifically, they contend that ethnographers are likley to have trouble meeting IRB demands that their protocols spell out procedures for selecting subjects, obtaining informed consent, disgusing the identity of participants, balancing risks and benefits, and protecting the data they collect. (489)
Fieldwork, they argue, is just too unpredictable to be planned out so thoroughly in advance. They note,
And, they find,
All of this sounds perfectly sensible. What suprises me a bit is the authors' belief that they are the first to make these arguments:
In fact, ethnographers have been complaining about the imposition of experimental research ethics on non-experimental research for thirty or forty years. Anthropologist Murray Wax, in particular, eloquently distinguished experimental research from fieldwork in just the way that Yanow and Schwartz-Shea do. See, for example, his essay, "On Fieldworkers and Those Exposed to Fieldwork: Federal Regulations and Moral Issues," Human Organization 36 (Fall 1977): 321-28. Indeed, despite a long bibliography, Yanow and Schwartz-Shea cite none of the many IRB critiques written in 1978-1980, when the IRB regulations were being overhauled.
I don't fault Yanow and Schwartz-Shea too much for not knowing this history. It is one of the tasks of the historian to save others from having to reinvent the wheel, and I hope my book, when finished, will make such a contribution.
Yanow and Schwartz-Shea end their article with "A Call for Action," most of which is fairly vague. IRB critics are split between those who seek to "tinker with the existing system," and those who seek to exclude large categories of research from any IRB jurisdiction. Yet it's not even clear on which side of this divide these authors fall. For example, they want APSA to "Issue a statement calling for reform of IRB policy in a substantive way that protects the interests of APSA members." (492) Lovely, but what should such a statement say? They demand reform without defining it.
More promising is their call for more research. They note,
They mention the possibility of an APSA handbook on ethical issues and current regulations.
This sounds a bit like the effort undertaken by the American Psychological Association in the preparation of its 1973 Ethical Principles in the Conduct of Research with Human Participants. As described in the first chapter of that book, rather than sit together and lay down some rules, the drafting committee surveyed the APA membership and assembled thousands of descriptions of real research projects that had raised ethical issues. The descriptions became the basis for an ethical guide directly relevant to the needs and values of the APA's members.
Around the same time, APSA itself undertook a similar effort, on a smaller scale, by conducting a study of actual cases in which researchers faced problems with confidentiality. Unfortunately, the full study seems not to have been published. A brief summary was published as James D. Carroll and Charles R. Knerr, "The APSA Confidentiality in Social Science Research Project: A Final Report," PS 9 (Autumn 1976): 416-419.
Whether or not a detailed ethical study would help ethnographic political scientists with their IRBs, it would be a great resource for scholars who want to do right by the people they study. I hope APSA--and other scholarly societies--will consider such a project.
The authors argue that "the character of its implicit research design model, embedded in its historical development, . . . renders IRB policy problematic for ethnographic and other field researchers." (483) Specifically, they contend that ethnographers are likley to have trouble meeting IRB demands that their protocols spell out procedures for selecting subjects, obtaining informed consent, disgusing the identity of participants, balancing risks and benefits, and protecting the data they collect. (489)
Fieldwork, they argue, is just too unpredictable to be planned out so thoroughly in advance. They note,
Field researchers must enter others’ worlds, and are expected to do so with care and respect, and these worlds can be complex, unbounded, and in flux. Instead of rigidly delimited, predesigned protocols laying out research steps that are invariable with respect to persons and time, which subjects can be handed as they step into the world of the medical researcher, field research often requires flexing the research design to accommodate unanticipated persons and personalities and unforeseen conditions.
And, they find,
extending [the Belmont] principles to other, non-experimental research settings without making the underlying mode of science and its methodology explicit and without exploring their suitability to non-experimental scientific modes and methodologies has resulted in a hodgepodge of ethical guidance that is confused and confusing. Those guidelines do not give the many serious ethical problems of field research design and methodologies the sustained attention they deserve. (491)
All of this sounds perfectly sensible. What suprises me a bit is the authors' belief that they are the first to make these arguments:
The proposals that we have seen to date for reforming IRB policy (e.g., Carpenter 2007) all tinker with the existing system. None of them, to the best of our knowledge, has yet identified and engaged the underlying methodological frame—experimental research design—shaping that policy and its implementation. Policy reforms that address resource, organizational, and other features of the existing policy leave that framing and its prosecution in place. The impact of these policies on field research is, however, serious, extending IRB policy to these other forms of research in the absence of systematic evidence of their having harmed research participants. If we are to have policies to ensure the protection of human participants in all areas of research, those policies need to be suited to other than just experimental research designs in ways that are commensurate with their own potential for harms. It is vital that recognition of the misfit between existing experimentally based policy and field research design and methodologies also be on the table in discussions of IRB policy reform. (491)
In fact, ethnographers have been complaining about the imposition of experimental research ethics on non-experimental research for thirty or forty years. Anthropologist Murray Wax, in particular, eloquently distinguished experimental research from fieldwork in just the way that Yanow and Schwartz-Shea do. See, for example, his essay, "On Fieldworkers and Those Exposed to Fieldwork: Federal Regulations and Moral Issues," Human Organization 36 (Fall 1977): 321-28. Indeed, despite a long bibliography, Yanow and Schwartz-Shea cite none of the many IRB critiques written in 1978-1980, when the IRB regulations were being overhauled.
I don't fault Yanow and Schwartz-Shea too much for not knowing this history. It is one of the tasks of the historian to save others from having to reinvent the wheel, and I hope my book, when finished, will make such a contribution.
Yanow and Schwartz-Shea end their article with "A Call for Action," most of which is fairly vague. IRB critics are split between those who seek to "tinker with the existing system," and those who seek to exclude large categories of research from any IRB jurisdiction. Yet it's not even clear on which side of this divide these authors fall. For example, they want APSA to "Issue a statement calling for reform of IRB policy in a substantive way that protects the interests of APSA members." (492) Lovely, but what should such a statement say? They demand reform without defining it.
More promising is their call for more research. They note,
There is much that we do not know about the kind(s) of field research political scientists are doing today . . . We need more systematic, policy-oriented research about members’ field research practices, and we call on APSA to take the lead in conducting or facilitating it . . . (491)
They mention the possibility of an APSA handbook on ethical issues and current regulations.
This sounds a bit like the effort undertaken by the American Psychological Association in the preparation of its 1973 Ethical Principles in the Conduct of Research with Human Participants. As described in the first chapter of that book, rather than sit together and lay down some rules, the drafting committee surveyed the APA membership and assembled thousands of descriptions of real research projects that had raised ethical issues. The descriptions became the basis for an ethical guide directly relevant to the needs and values of the APA's members.
Around the same time, APSA itself undertook a similar effort, on a smaller scale, by conducting a study of actual cases in which researchers faced problems with confidentiality. Unfortunately, the full study seems not to have been published. A brief summary was published as James D. Carroll and Charles R. Knerr, "The APSA Confidentiality in Social Science Research Project: A Final Report," PS 9 (Autumn 1976): 416-419.
Whether or not a detailed ethical study would help ethnographic political scientists with their IRBs, it would be a great resource for scholars who want to do right by the people they study. I hope APSA--and other scholarly societies--will consider such a project.
Subscribe to:
Posts (Atom)
