Friday, July 11, 2014

Microsoft Seeks Ethics Review Without Bottlenecks and Frustration

Duncan Watts of Microsoft Research announces Microsoft will soon launch "an ethics-review process for human-subject research designed explicitly for web-based research." Could such a process avoid the pitfalls of the IRB?

[Watts, Duncan J. “Lessons Learned From the Facebook Study.” Chronicle of Higher Education Blogs: The Conversation, July 9, 2014. h/t Rebecca Tushnet]

Here is the proposal:

What we need is an ethics-review process for human-subject research designed explicitly for web-based research, in a way that works across the regulatory and institutional boundaries separating universities and companies. For the past two years, my colleagues at Microsoft Research have been designing precisely such a system, which is to be rolled out shortly.

It is still a work in process, and many details are liable to change as we learn what works and what doesn’t, but the core principle is one of peer review. Although we have an ethics board composed of experienced researchers (including me), the idea is not to have every proposal submitted to the board for review—a recipe for bottlenecks and frustration. Rather, it is to force researchers to engage in structured, critical discussions with educated peers, where everyone involved will be accountable for the outcome and hence will have strong incentives to take the review seriously. Unproblematic designs will be approved via an expedited process, while red flags will provoke a full review—a two-tier system modeled on existing IRBs.

Aside from its inherent scalability, the peer-review approach also has the benefit of involving the entire research community in discussions about ethics. Rather than placing the burden of review on a small committee of experts, everyone will have to undergo some basic training and consider the ethical implications of their research. The goal is to create an educated community that, in subjecting all cases to diverse viewpoints, lets fewer errors slip through. And because the process is designed to run continuously, insights arising from novel cases will diffuse quickly.

So this would be like current university IRBs in having a multi-tier level of review, but unlike those IRBs in that it would involve "the entire research community in discussions about ethics," rather than "a small committee of experts."

Watts claims "experience with review procedures both at Columbia University as well as in corporate research labs (first at Yahoo! Research, where we implemented an IRB-like process, and now at Microsoft)," and he knows their limits: "Although progress has been made over that time, many university IRBs still have little experience with the mechanics of web-based research." So I think it would be better to characterize those IRBs not as small committees of experts, but small committees of non-experts, or, worse, pseudo-experts. [That last clause added 1:35pm]

And Watt's proposal to diffuse insights sounds encouragingly close to the National Research Council's suggestion to bring more empirical evidence into ethics deliberations.

But I wonder if Watts is sufficiently aware of the problems with existing IRBs. He makes the odd claim, "we have pretty good procedures for reviewing and approving psychology experiments done in university labs. We also have a pretty clear idea of how to handle survey research or ethnographic studies."

No, we don't.

No comments: