Ethics of big data research are unsettled, and so are mechanisms

Sarah Zhang of WIRED uses the OKCupid data dump to explore the unsettled state of big data research ethics and mechanisms to promote them.

[Sarah Zhang, “Scientists Are Just as Confused About the Ethics of Big-Data Research as You,” WIRED, May 20, 2016.]

Exemption by the Numbers?

Two computer researchers describe a system at Microsoft Research designed to provide automatic approval for low-risk studies. Rather than follow the Common Rule’s exemption model of requiring IRB review if any of a series of conditions is met, the Microsoft system assigns numerical values to aspects of a proposal that bear some risk to participants. Proposals with a low total get immediate approval from an Excel spreadsheet.

[Bowser, Anne, and Janice Y. Tsai. “Supporting Ethical Web Research: A New Research Ethics Review.” In Proceedings of the 24th International Conference on World Wide Web, 151–61. WWW ’15. Republic and Canton of Geneva, Switzerland: International World Wide Web Conferences Steering Committee, 2015. doi:10.1145/2736277.2741654.]

Microsoft Seeks Ethics Review Without Bottlenecks and Frustration

Duncan Watts of Microsoft Research announces Microsoft will soon launch "an ethics-review process for human-subject research designed explicitly for web-based research." Could such a process avoid the pitfalls of the IRB?

[Watts, Duncan J. “Lessons Learned From the Facebook Study.” Chronicle of Higher Education Blogs: The Conversation, July 9, 2014. h/t Rebecca Tushnet]

Computer Scientist: Informed Consent is the Wrong Metaphor

Michael Bernstein, assistant professor of computer science at Stanford and a former postdoctoral scholar on Facebook’s Data Science team, argues that "Hammering ethical protocols designed for laboratory studies onto internet experimentation is fundamentally misguided."

[Bernstein, Michael. “The Destructive Silence of Social Computing Researchers.” Medium, July 7, 2014. https://medium.com/@msbernst/9155cdff659.]

A Bit of Historical Perspective on the Facebook Flap

IRBs and behavioral research are all over the news, as a result of a paper that manipulated the news feeds of 689,003 Facebook users.

[Kramer, Adam D. I., Jamie E. Guillory, and Jeffrey T. Hancock. “Experimental Evidence of Massive-Scale Emotional Contagion through Social Networks.” Proceedings of the National Academy of Sciences 111, no. 24 (June 17, 2014): 8788–90. doi:10.1073/pnas.1320040111.]

Michelle Meyer has posted a detailed analysis of the regulatory context, explaining multiple ways a project like this could have been approved. She concludes that "so long as we allow private entities freely to engage in these practices, we ought not unduly restrain academics trying to determine their effects."

[Meyer, Michelle N. “How an IRB Could Have Legitimately Approved the Facebook Experiment—and Why That May Be a Good Thing.” The Faculty Lounge, June 29, 2014. http://www.thefacultylounge.org/2014/06/how-an-irb-could-have-legitimately-approved-the-facebook-experimentand-why-that-may-be-a-good-thing.html.]

I have little to add to Meyer's excellent post, except a bit of historical perspective. Psychological experiments—whether in the lab, in the field, or online—fall outside my main area of concern, but perhaps I can offer a few relevant points.

SACHRP: Exempt Research May "Be Subject to IRB Review"

As reported by Erica Check Hayden in Nature, at its March meeting, SACHRP endorsed "Considerations and Recommendations Concerning Internet Research and Human Subjects Research Regulations, with Revisions," prepared by Elizabeth Buchanan and Dean Gallant. The guidance offers some common sense, but it struggles with the legacy of the poorly drafted Common Rule. And it threatens to make matters worse by suggesting that some exempt research may "be subject to IRB review."

College Freshmen Don't Mind Research Use of Public Facebook Profiles

An interdisciplinary team of researchers "investigated publicly available Facebook profiles of freshmen undergraduate students within one large state university Facebook network" and invited 188 of those freshmen to be interviewed for a health study. At the end of the interview, they told the freshmen that they had been selected using their public Facebook profiles. "Participant responses included endorsement (19.7%), fine (36.4%), neutral (28.8%), uneasy (9.1%), and concerned (6.1%)." The researchers acknowledge that 6 percent minority but conclude that "publicly available Facebook profiles of older adolescents are viewed as public spaces by both the adolescents themselves as well as the legal system."

[Moreno, Megan A., Alison Grant, Lauren Kacvinsky, Peter Moreno, and Michael Fleming. “Older Adolescents’ Views Regarding Participation in Facebook Research.” Journal of Adolescent Health 51, no. 5 (November 2012): 439–444. doi:10.1016/j.jadohealth.2012.02.001]

IRB Demands and Forbids that Survey Researchers Collect E-Mail Addresses

Andrew Gelman revisits IRB nightmares at Statistical Modeling, Causal Inference, and Social Science.

Halavais Calls for "Open Publication of IRB Protocols or Ethics Reflections"

In an essay in Nature and on his blog, Alexander Halavais, president of the Association of Internet Researchers, calls for funders to require "the open publication of IRB protocols or ethics reflections."

[Alexander Halavais, "Social Science: Open Up Online Research," Nature 480 (8 December 2011): 174–175, doi:10.1038/480174a; Alexander Halavais, "IRBs and Clean Secrets, A Thaumaturgical Compendium, 8 December 2011.]

Alarmist Views on Harvard Facebook Study

The Chronicle of Higher Ed reports a debate over a study of Facebook profiles, started in 2006 by Jason Kaufman of Harvard's Berkman Center for Internet & Society. The debate suggests that researchers may not be aware of how easy it can be to identify allegedly anonymous institutions and individuals, but that neither IRBs nor outside critics may understand all the implications of a study either.

[Marc Parry, "Harvard's Privacy Meltdown," Chronicle of Higher Education, 10 July 2011.]

Law Professor Sees Broad Role for IRBs in Virtual Worlds

Professor Joshua A. T. Fairfield of the Washington & Lee University School of Law explores the law of research in virtual worlds.

[Joshua Fairfield, "Avatar Experimentation: Human Subjects Research in Virtual Worlds." (November 29, 2010). U.C. Irvine Law Review, Symposium Issue, 2011; Washington & Lee Legal Studies Paper No. 2010-14. Available at SSRN: http://ssrn.com/abstract=1717057.]

Fairfield makes some important points about how researchers interested in these environments might apply ethical guidelines developed for older forms of research. But the article pays insufficient attention to the limits of ethical guidance developed to govern medical experimentation, misstates some of the provisions of current regulations, and downplays the troubles Internet researchers have faced with IRBs.

Anxious Pessimism on UK's New Framework for Research Ethics

In March 2010, the United Kingdom's Economic and Social Research Council (ESRC) released its Framework for Research Ethics (FRE) as a successor to its 2005 Research Ethics Framework (REF).

David Erdos kindly alerted me to the November 2010 (Volume 15, Issue 4) issue of Sociological Research Online, which devotes a special section to essays about the new framework.

The six essays in the section suggest that British sociologists are wary of their research ethics committees and the expanded authority granted to them by the new framework.

Is Facebook Data Mining Human Subjects Research?

Recent law-school graduate Lauren Solberg finds that "data mining on Facebook likely does not constitute research with human subjects, and therefore does not require IRB review, because a researcher who collects data from Facebook pages does not 'interact' with the individual users, and the information on Facebook that researchers mine from individual users' pages is not 'private information.'"

[Lauren Solberg, "Data Mining on Facebook: A Free Space for Researchers or an IRB Nightmare?" article under review, University of Illinois Journal of Law, Technology & Policy 2010 (2). The article has been accepted for publication, but the journal is still soliciting comments.]

Solberg challenges policies now in place at Indiana University and the University of Massachusetts Boston, where researchers must get Facebook's written permission or the written permission of every individual who is studied. These policies, she argues, impose unnecessary burdens on researchers and IRBs alike. (The two policies are identical, but it's not clear which university borrowed from the other.)

She argues that most data mining projects do not meet the regulatory definition of human subjects research. Reading existing profiles is not interaction with an individual. Nor is a Facebook profile that is open to strangers private information, i.e., "information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example, a medical record)." If a college admissions officer or a potential employer can read your profile, you've lost little by having an anthropologist read it as well.

This analysis seems sound, but it's not clear to me that anyone disagrees. In particular, the third university Solberg mentions, Washington University in St. Louis, applies its policy only to "
Any activity meeting the definition of 'human subject research' which is designed to recruit participants or collect data via the Internet." It then lists several examples, most of which involve interaction with living individuals. Thus, I doubt Solberg's claim that "researchers at Washington University need only inform Facebook users that they are recording information that is posted on their pages." Rather, if the project does not meet the definition of human subject research, then Wash U. researchers need not do even that much.

Solberg's article skirts some interesting questions. One concerns the boundaries of a reasonable expectation of privacy. Thus, Michael Zimmer gives the example of a study by Harvard graduate students of the Facebook profiles of Harvard undergraduates. If an undergraduate had made some information visible only to other Harvard students (a choice Facebook's software allows), and a Harvard student-researcher sees it, does that change Solberg's analysis?

A second question concerns the authority of university research offices and IRBs to insist that researchers abide by website terms of service. Notably, the Indiana and UMASS policies do not cite federal human subjects regulations as their authority. Rather, they claim that Facebook and Myspace "explicitly state that their sites are not intended for research but for social networking only."

Solberg writes that evaluating such claims is "outside the scope of this article," but they are interesting in three ways. First, they may be factually false; I could find no such explicit statements in the Facebook or Myspace terms of service. Second, they are divorced from federal regulation. For example, the Facebook terms of service do not distinguish between living and dead Facebook members, whereas federal human subjects protections apply only to the living. Finally, they are internally inconsistent. If Facebook and Myspace did prohibit the use of their sites for research, would not researchers still be violating the terms of service even if they got signed consent from individual members, as allowed by the policies? Just who are these two universities trying to protect?

Solberg concludes that "Unfortunately, and somewhat surprisingly, the OHRP has issued no guidance pertaining to Internet research in general, let alone guidance specifically relating to the issue of data mining on the Internet." To give the feds some credit, in summer 2010 (after Solberg wrote her article), SACHRP did sponsor a panel on the Internet in Human Subjects Research. It can take a long time from a SACHRP presentation to OHRP guidance, but the wheels may be moving on this one.

---

Note, 19 November 2010: The original version of this post identified Ms. Solberg as a law student. She has in fact graduated. I have also changed the link about Michael Zimmer's work from his SACHRP presentation to his article, "'But the data is already public': on the ethics of research in Facebook," Ethics and Information Technology 12 (2010): 313-325.

SACHRP to Discuss Internet Research

The July 21 meeting of the Secretary's Advisory Committee on Human Research Protections will sponsort a panel entitled "The Internet in Human Subjects Research," featuring Elizabeth Buchanan, Montana Miller, Michael Zimmer, and John Palfrey. It should be an interesting session. Unfortunately (and ironically), SACHRP has stopped posting transcripts of its meetings, so it's not clear how much of the content will be available to Internet researchers. I am told the meeting minutes will be posted at some point.