Monday, January 16, 2012

SBS White Paper Calls for Drastic Reform

Weighing in at 72 pages, the "Social and Behavioral Science White Paper" is the most detailed response to the ANPRM from scholars in the social sciences. The paper presents a grim picture of the state of IRB review and is generally supportive of the ANPRM's goal of reform. But it offers detailed, helpful warnings about the potential effects of the proposed "excused" category and the adoption of HIPAA as a model for confidentiality requirements. Though it shies away from the toughest questions about the IRB system, it is a good expression of the frustrations felt by so many researchers in the social sciences and humanities.

[American Educational Research Association et al., "Social and Behavioral Science White Paper on Advanced Notice for Proposed Rulemaking (ANPRM), Federal Register 44512-531 (July 26, 2011); ID Docket HHS-OPHS-2011-0005," 26 October 2011.]


The response bears an impressive pedgriee. It was drafted by Felice J. Levine--who has been involved with IRB issues for over a decade--Richard O. Lempert, and Paula R. Skedsvold, and endorsed by almost two dozen scholarly and professional organizations:

  • American Educational Research Association
  • Federation of Associations in Behavioral and Brain Sciences
  • Consortium of Social Science Associations
  • American Political Science Association
  • American Sociological Association
  • Association of Psychological Science
  • Association of Population Centers
  • Cognitive Science Society
  • Council of Professional Associations on Federal Statistics
  • Human Factors and Ergonomics Society
  • Inter-university Consortium for Political and Social Research
  • Law and Society Association
  • National Academy of Neuropsychology
  • National Communication Association
  • NORC
  • Population Association of America
  • Social Science History Association
  • Society for Behavioral Neuroendocrinology
  • Society for Computers in Psychology
  • Society of Experimental Social Psychology
  • Society for Industrial and Organizational Psychology
  • Society for Judgment and Decision Making

While it's not clear exactly how much input these organizations had in the final document, it may be that some of them pushed for the White Paper's critical stance. For example, a column in the American Sociological Association's newsletter mentions the role of Harry Perlstadt in prodding the ASA to respond to the ANPRM. Some of the key concerns voiced in the White Paper resemble some of Perlstadt's own comments.

In any case, I trust regulators will take seriously such a long response from so large a group of organizations. Certainly I do, and I remark on the following features.

Criticisms of the Status Quo

Spread throughout the White Paper are suggestions that the current Common Rule has serious flaws.
  • The regulations are vague

    The regulations are hard to understand. The White Paper finds that the exemption for studies of educational practices "been administered with ambiguity, uncertainty and inconsistency regarding the activities that are not covered by 45CFR46."

    Similarly, the White Paper blames the regulations for encouraging IRBs to overestimate risk:

    There is considerable ambiguity in the discussion of psychological risk in 45CFR46 creating a danger that IRBs will misjudge the nature of possible psychological harms and overestimate their likely magnitudes and risks. The result will be unneeded reviews and unnecessary regulation of important but low risk SBS research. Among the negative psychological risks labeled “psychological harms” that human subjects may experience are such emotions as boredom, worry, frustration, annoyance, stress, upset, guilt, and loss of self confidence. These may be minor in magnitude or transitory and may even stimulate new levels of personal insight or self awareness.

    Even the definition of human subjects research is flawed:
    We suggest that rather than focus on what does or does not contribute to generalizable knowledge (which in fields of science and scholarship can be a rather insulting task) or making judgments based solely on the disciplinary identification of researchers, it would be wiser to define the categories of research activities that 45CFR46 should cover adhering to the explicit definition of human subjects, as is relied on in exemptions 46.101(b)(2), (b)(3), and (b)(4), for example.
  • IRBs base decisions on unrealistic assessments of risk

    The White Paper repeatedly warns against allowing IRBs to base decisions on improbable risks, suggesting that its authors understand that IRBs now do this routinely. The paper calls for regulations that will judge risk and harm "not in the abstract or by envisioned worst case scenarios" and suggests that "unlikely deviant cases should not drive decision making."
  • Such vivid imagination also infects consent requirements:

    One of the primary factors contributing to the length and complexity of informed consent forms is the perceived need for institutions (through their IRBs) to require researchers to identify and mention every conceivable risk to human subjects, however remote. We assume institutional concerns with federal regulatory compliance and liability issues drive this requirement. However, in doing so, human subjects are presented with mounds of information that aid neither in comprehending their role in the research effort nor in assessing the risks and benefits associated with participation.
  • IRBs impose unnecessary levels of scrutiny

    The White Paper finds that IRBs are overregulating research. Exempt studies have been "beset by delays," and studies that should be expedited suffer "regulatory creep to full review . . . given the all too common migration to full review by IRBs over time."

    It also hints that IRBs are currently overstepping their mission to protect human subjects. In a strongly worded reply to Question 27, the White Paper states,

    Consideration of other factors can limit research that needs to be done and chill the freedom of inquiry that scientists need to advance knowledge. We are concerned that factors other than human subjects protection not creep into the work of IRBs. It is of paramount importance that other considerations do not color the work or judgment of IRBs and that policy considerations do not interfere with academic freedom and the pursuit of knowledge. In addition, we urge emphasizing the core principle, perhaps with examples, to make its intention inescapably clear.
  • The regulations encourage inappropriate requirements for informed consent

    The White Paper finds that

    Current regulations as well as the proposed changes continue to assume a one-dimensional mental model of written consent at one point in time. In general, the informed consent process could benefit from more research (e.g., what do human subjects want to know, what do they comprehend, what kinds of information are most relevant to making a decision about participation) so that any changes to the regulations are evidence-based.
  • Overregulation has a price

    Finally, the White Paper notes that these problems impose a high price:

    the burdens to IRBs and to researchers are major and costly impediments to socially valuable research and that much of the impulse for this long overdue review is a sense of the need to divest IRBs of regulatory obligations that distract them from focusing on research that involves more than minimal risk.
  • Overregulation is a systemic problem

    All told, "SBS researchers . . . often encounter IRBs that look only to guidance developed with biomedical research in mind even when the guidance is poorly suited to the proposed SBS research."

    Rather than dismiss such IRB failures as anomalies, the White Paper sees them as the result of systemic pressures to overregulate.

    It warns of "the drift toward hyper-regulation" and blames the system itself, noting, "A conservative bias is natural in a system that threatens to penalize insufficient scrutiny but that does not sanction unwarranted review." For example, it notes that OHRP's suggestion that researchers not be allowed to determine if their projects are exempt has created the "danger of a slippery slope of hyper-regulation and decisions that do not reflect legislative intent but instead the risk-aversion of IRBs."

    The result is that "Institutional practices surrounding human subjects regulations are increasingly driven by regulatory and legal liability concerns, rather than concerns for the welfare of human subjects."

Concerns about the "Excused" Category

The White Paper warns that moving from "exempt" to "excused" could make things worse, not better.

Our concern is with the functional abandonment of an exempt category for research activities that are best left as uncovered by 45CFR46. To relocate activities as excused that have previously been classified as exempt from 45CFR46 and then to introduce new requirements they must meet is potentially a step backwards. The logic that led to the original creation of the exempt category for activities that fall outside of the province of 45CFR46 is principled and compelling. Whether the current categories and classifications of exempt are all the right choices is not the issue. The point here is that to shift activities that have been determined to be outside of the purview of 45CFR46 so that they now fall within it may, no matter how limited the requirements for excused research, place new burdens on work that as a matter of policy or principle should be outside of the scope of human subjects research.

I don't know if I would go so far as to say the current exemptions are based on "logic," but the 1981 Federal Register announcement did present them as a means to "clarify coverage questions, significantly reduce the work load of IRBs, and thus allows IRBs to concentrate on the review of research which involves a greater degree of risk to subjects."

Concerns about Confidentiality Rules and HIPAA

The White Paper sees in the ANPRM's proposed confidentiality regime "a new set of informed consent requirements about prior human subjects that are not based on empirical study but on potentially misleading if not erroneous, assumption about the likely intentions of human subjects and their wishes in agreeing to participate in research."

It complains that the ANPRM "does not build on the expertise or decades of guidance and experience in considering the relationship between consent, data protection, and appropriate data access and use," and warns that its proposal to require fresh consent for new uses of data "could have a devastating effect on much SBS and other large sample research where contacting subjects for additional consent will often be difficult, never complete, and not practically possible if in the interest of subject privacy identifiers and contact information was not originally collected or, if collected, retained in the data set."

And it is particularly scornful of the ANPRM's proposal to base human subjects requirements on HIPAA, which it dissects in detail, concluding that "Privacy acts like FERPA or HIPPA [sic] have their place for protecting administrative record systems under their aegis, but they fall short as guidance for preparing, storing, protecting data or stipulating conditions for public access or restricted data use. "Indeed, "Absent instances of serious harms associated with survey interviews or other means of data collection, we see no need to require standardized means of data security and information protection at the data gathering phase. We urge strongly against further consideration of this additional burden with no obvious benefit."

Recommendations based on the ANRPM

The White Paper addresses the problems it identifies with the following solutions:

  • To fix hyper-protection, require empirical evidence

    The White Paper responds to Question 4 with the suggestion that
    regulations should be changed to make clear that IRBs should only consider “reasonably foreseeable risks or discomforts.” The proposed language properly emphasizes that although IRBs should evaluate the likelihood that harms might be more than minimal, they should not speculate about every possible harm. If harm is not reasonably foreseeable (likely), then the probability of that harm is minimal. As emphasized earlier in this white paper, the language should speak of “harm” in the same way it speaks of “discomfort” rather than use the word “risk.” It is the possibility of harm and not risk that must be foreseeable. If, for example, the risk of harm is one in 10 million, the risk is reasonably foreseeable, but it would be unreasonable to foresee harm.

    One way to deploy such evidence would be the the collection of "Documents based on materials appropriate to similar studies or previously approved for use in similar expedited research—whether approved at the same institution or not—[which] should qualify as acceptable templates for expedited studies." (cf. Alexander Halavais's recent suggestions.)

    The White Paper also argues that "Risks should not be considered greater than minimal simply because deception is involved" and that "Absent a sufficient research base, the rules should not presume that any particular study feature creates more than minimal non-physical risks." If a procedure does make it onto the list for expedited review, it should not be removed "except where a clear pattern of harm has developed."

    Similarly, "The vague language of 'emotionally charged' is unsatisfactory because topics that elicit no strong emotional reaction from most persons will be emotionally charged for some."

  • To fix informed consent, reduce the regulatory requirements

    The White Paper recommends that non-clinical researchers not be burdened with irrelevant requirements that they tell participants about "appropriate alternative procedures or treatments," that the research is experimental, and that injuries will be compensated. More generally, it calls for "research on comprehension and modes of inviting consent."

    I think the White Paper could have gone a little further here, also dropping the requirement that researchers disclose "reasonably foreseeable risks and benefits." I don't think commercial or political researchers offer such information, which in many social science projects will be "none."

  • To fix regulatory creep, require due process protections

    The White Paper enthuses about the ANPRM's suggestion that IRBs be required to provide written justification every time they impose more scrutiny than is required by the regulations. It suggests that if an IRB wants to deny excused status to a proposal, "it should be incumbent on the reviewer to state specifically in writing what aspect of a registration document leads to doubt a project’s excused status and to refrain from review where it finds none."

    It also
    strongly support[s] the requirement of an appeals process. The consequences of an IRB’s denying or substantially limiting how research can proceed could be severe, affecting those who might reap the benefits of the research, the immediate interests of the researcher, and the interests of institution or society over the longer run. When consequences are great, the fallibility of judges suggests the desirability of an appeals mechanism.

    By contrast, the White Paper calls for regulators to trust researchers: "Researchers can for the most part be trusted to make fair initial assessments so long as they must register their studies and know they risk being audited."

  • Base confidentiality requirements on research

    The White Paper recommends rather than use HIPAA as a starting place, the regulations should encourage institutions to develop standards based on "National Research Council reports, in the ethics codes of professional associations, and in the practices employed by entities like the ICPSR which are responsible for storing and disseminating SBS data varying in sensitivity." And it suggests that "in almost all cases pre-existing data, whether collected for research or non-research purposes, should be reusable without consent as long as risks to human subjects are minimal and sufficient confidentiality protections are in place."

Recommendations Beyond the ANPRM

Some of the White Paper's most intriguing suggestions concern matters not raised directly by the ANPRM.

Some suggest that the current exemptions are too narrow. For example, the White Paper suggests that the current 45 CFR 46.101(b)(3)(i) exemption for research on "elected or appointed public officials or candidates for public office" be not only preserved but expanded to cover "any person who has public figure status." And it calls for projects to be excused from review

where access to a location or to subjects is given by a person entitled to give access to a research setting and where the data will be collected by observation or by other methods that themselves qualify as excused so long as the data collected will not be linked to named persons. A business might, for example, provide a researcher access to observe and analyze how plant architecture affects worker interactions, or a police department may allow researchers to ride in police cars to observe exchanges between police and citizens.

The White Paper also envisions an ongoing, inclusive process of policy formation more like Canada's. The paper supports the creation of a panel to review the expedited review list every two years, and urges "strongly that SBS sciences be visibly present on that panel because of the preponderance of research in SBS fields that are appropriate for expedited review."

More generally, it calls for SBS researchers to be represented in policy-making bodies. It hopes that "An interagency committee under OSTP would facilitate communication and planning across agencies as new issues arise and ensure that the focus is not predominately biomedical." It calls for a National Research Council committee to "examine what types of research activities are appropriately exempt and how these determinations should be made going forward," and asks that if an adverse-event reporting system is developed, the "creators of such a system to include non-DHHS agency representatives and SBS researchers in the development so that concerns may be addresses at the outset. Similarly, attempts to standardize forms should involve agencies who have signed on to the Common Rule and a wide range of research approaches, including SBS research."

Some Silences and a Departure

Taken together, the White Paper offers some stinging critiques of the current regulations and the ANPRM's proposals, while applauding some of the ANPRM's bolder proposals. It is worth noting, however, the degree to which the White Paper fails to examine some basic assumptions underlying today's IRB system.

First, the White Paper pays homage to the Belmont Report, noting that "our aim is to foster revisions to the Common Rule that are consistent with the principles set forth in the Belmont Report in 1979." Yet it calls for the abandonment of two of the three Belmont "applications" for minimal risk research, stating that a weighing of risks against benefits and a determination that subjects be equitably selected are unnecessary for such research. If major chunks of the Belmont Report are inapplicable to so much research, why keep that report as the basis of ethics governance? Why not start fresh?

Second, the White Paper suggests that the regulations be revised to abandon the distinction between research that is designed to produce generalizable knowledge, but it does not propose a clear alternative definition of human subjects research. This runs the risk of expanding the reach of the Common Rule, which the White Paper clearly does not want.

Third, the White Paper fails to note just how drastically the current regulations fall short of its standards for sound policy. The White Paper wants rules that are based on "a clear pattern of harm," "are evidence-based," and "reflect legislative intent." It concludes that "any decision to expand the Common Rule be evidence-based, and that research be funded to provide needed evidence."

Yet it ignores the fact that the requirements for IRB review of research in the humanities and social sciences have never met any of these standards. If the authors of the White Paper believe that "absent instances of serious harms" research should not be restricted, why not call for the wholesale liberation of the social sciences? If they believe that the Common Rule should only be expanded based on evidence proceeding from funded research, why not apply that criterion retroactively to the existing regulations?

Finally, buried in the paper is the claim that "IRBs are experts in research ethics, but seldom are experts in data security and data protection." While the emphasis in that sentence is clearly on the second half, it is a pity that the authors did not scrutinize the first. The White Paper tells us that IRBs impose inappropriate consent requirements, base decisions in improbable estimates of risk, get distracted by concerns other than the protection of research subjects, and "chill the freedom of inquiry." How, then, do they conclude that "IRBs are experts in research ethics"? Given that IRBs have shown themselves so inept, why does the White Paper believe that "research that poses greater than minimal risk should be reviewed by a fully convened IRB and that 45CFR46 should not be changed in that regard"?

Though it shies away from asking if there is any reason to think the IRB system is worth preserving, the White Paper represents a substantial departure from more accommodationist writings by some of the individuals and organizations it represents. In 2008, for example, two of the White Paper's lead authors--Felice J. Levine and Paula R. Skedsvold-- believed that "significant regulatory reform [was] not likely." Accordingly, they offered what I considered fairly tame suggestions for reform based on the premise that "creative use of the flexibility within the current system might resolve some of the pressing concerns of social and behavioral science investigators while ensuring adequate oversight of research involving human participants." Now, with regulatory reform at least on the table, it is delightful to learn that they have so many proposals for dramatic changes in the regulations.

No comments: