Exemption by the Numbers?

Two computer researchers describe a system at Microsoft Research designed to provide automatic approval for low-risk studies. Rather than follow the Common Rule’s exemption model of requiring IRB review if any of a series of conditions is met, the Microsoft system assigns numerical values to aspects of a proposal that bear some risk to participants. Proposals with a low total get immediate approval from an Excel spreadsheet.

[Bowser, Anne, and Janice Y. Tsai. “Supporting Ethical Web Research: A New Research Ethics Review.” In Proceedings of the 24th International Conference on World Wide Web, 151–61. WWW ’15. Republic and Canton of Geneva, Switzerland: International World Wide Web Conferences Steering Committee, 2015. doi:10.1145/2736277.2741654.]

A/B ≠ A&B

A/B testing is the comparison of two products by providing each, simultaneously, to two randomly selected groups. As Michelle Meyer notes, the practice is less ethically suspect than a common alternative: imposing a new product on all of one’s customers without first testing it.

A&B is an abbreviation for assault and battery. Depending on the jurisdiction, the two may be a single crime or distinct crimes, but everywhere A&B is unlawful.

Law professor James Grimmelmann has confused the two:

Suppose that Professor Cranium at Stonewall University wants to find out whether people bleed when hit in the head with bricks, but doesn’t want to bother with the pesky IRB and its concern for “safety” and “ethics.” So Cranium calls up a friend at Brickbook, which actually throws the bricks at people, and the two of them write a paper together describing the results. Professor Cranium has successfully laundered his research through Brickbook, cutting his own IRB out of the loop. This, I submit, is Not Good.

IRBs for everyone, or you get hit with a brick. I suggest that this parade of horribles is missing its elephant. Blame the IACUC?

Anyway, thanks for the link.

ETA(2:18PM): I originally linked to the wrong Meyer paper. Of course, that one's worth reading too.

Microsoft Seeks Ethics Review Without Bottlenecks and Frustration

Duncan Watts of Microsoft Research announces Microsoft will soon launch "an ethics-review process for human-subject research designed explicitly for web-based research." Could such a process avoid the pitfalls of the IRB?

[Watts, Duncan J. “Lessons Learned From the Facebook Study.” Chronicle of Higher Education Blogs: The Conversation, July 9, 2014. h/t Rebecca Tushnet]

Computer Scientist: Informed Consent is the Wrong Metaphor

Michael Bernstein, assistant professor of computer science at Stanford and a former postdoctoral scholar on Facebook’s Data Science team, argues that "Hammering ethical protocols designed for laboratory studies onto internet experimentation is fundamentally misguided."

[Bernstein, Michael. “The Destructive Silence of Social Computing Researchers.” Medium, July 7, 2014. https://medium.com/@msbernst/9155cdff659.]

A Bit of Historical Perspective on the Facebook Flap

IRBs and behavioral research are all over the news, as a result of a paper that manipulated the news feeds of 689,003 Facebook users.

[Kramer, Adam D. I., Jamie E. Guillory, and Jeffrey T. Hancock. “Experimental Evidence of Massive-Scale Emotional Contagion through Social Networks.” Proceedings of the National Academy of Sciences 111, no. 24 (June 17, 2014): 8788–90. doi:10.1073/pnas.1320040111.]

Michelle Meyer has posted a detailed analysis of the regulatory context, explaining multiple ways a project like this could have been approved. She concludes that "so long as we allow private entities freely to engage in these practices, we ought not unduly restrain academics trying to determine their effects."

[Meyer, Michelle N. “How an IRB Could Have Legitimately Approved the Facebook Experiment—and Why That May Be a Good Thing.” The Faculty Lounge, June 29, 2014. http://www.thefacultylounge.org/2014/06/how-an-irb-could-have-legitimately-approved-the-facebook-experimentand-why-that-may-be-a-good-thing.html.]

I have little to add to Meyer's excellent post, except a bit of historical perspective. Psychological experiments—whether in the lab, in the field, or online—fall outside my main area of concern, but perhaps I can offer a few relevant points.

College Freshmen Don't Mind Research Use of Public Facebook Profiles

An interdisciplinary team of researchers "investigated publicly available Facebook profiles of freshmen undergraduate students within one large state university Facebook network" and invited 188 of those freshmen to be interviewed for a health study. At the end of the interview, they told the freshmen that they had been selected using their public Facebook profiles. "Participant responses included endorsement (19.7%), fine (36.4%), neutral (28.8%), uneasy (9.1%), and concerned (6.1%)." The researchers acknowledge that 6 percent minority but conclude that "publicly available Facebook profiles of older adolescents are viewed as public spaces by both the adolescents themselves as well as the legal system."

[Moreno, Megan A., Alison Grant, Lauren Kacvinsky, Peter Moreno, and Michael Fleming. “Older Adolescents’ Views Regarding Participation in Facebook Research.” Journal of Adolescent Health 51, no. 5 (November 2012): 439–444. doi:10.1016/j.jadohealth.2012.02.001]

Halavais Calls for "Open Publication of IRB Protocols or Ethics Reflections"

In an essay in Nature and on his blog, Alexander Halavais, president of the Association of Internet Researchers, calls for funders to require "the open publication of IRB protocols or ethics reflections."

[Alexander Halavais, "Social Science: Open Up Online Research," Nature 480 (8 December 2011): 174–175, doi:10.1038/480174a; Alexander Halavais, "IRBs and Clean Secrets, A Thaumaturgical Compendium, 8 December 2011.]

Alarmist Views on Harvard Facebook Study

The Chronicle of Higher Ed reports a debate over a study of Facebook profiles, started in 2006 by Jason Kaufman of Harvard's Berkman Center for Internet & Society. The debate suggests that researchers may not be aware of how easy it can be to identify allegedly anonymous institutions and individuals, but that neither IRBs nor outside critics may understand all the implications of a study either.

[Marc Parry, "Harvard's Privacy Meltdown," Chronicle of Higher Education, 10 July 2011.]

Is Facebook Data Mining Human Subjects Research?

Recent law-school graduate Lauren Solberg finds that "data mining on Facebook likely does not constitute research with human subjects, and therefore does not require IRB review, because a researcher who collects data from Facebook pages does not 'interact' with the individual users, and the information on Facebook that researchers mine from individual users' pages is not 'private information.'"

[Lauren Solberg, "Data Mining on Facebook: A Free Space for Researchers or an IRB Nightmare?" article under review, University of Illinois Journal of Law, Technology & Policy 2010 (2). The article has been accepted for publication, but the journal is still soliciting comments.]

Solberg challenges policies now in place at Indiana University and the University of Massachusetts Boston, where researchers must get Facebook's written permission or the written permission of every individual who is studied. These policies, she argues, impose unnecessary burdens on researchers and IRBs alike. (The two policies are identical, but it's not clear which university borrowed from the other.)

She argues that most data mining projects do not meet the regulatory definition of human subjects research. Reading existing profiles is not interaction with an individual. Nor is a Facebook profile that is open to strangers private information, i.e., "information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example, a medical record)." If a college admissions officer or a potential employer can read your profile, you've lost little by having an anthropologist read it as well.

This analysis seems sound, but it's not clear to me that anyone disagrees. In particular, the third university Solberg mentions, Washington University in St. Louis, applies its policy only to "
Any activity meeting the definition of 'human subject research' which is designed to recruit participants or collect data via the Internet." It then lists several examples, most of which involve interaction with living individuals. Thus, I doubt Solberg's claim that "researchers at Washington University need only inform Facebook users that they are recording information that is posted on their pages." Rather, if the project does not meet the definition of human subject research, then Wash U. researchers need not do even that much.

Solberg's article skirts some interesting questions. One concerns the boundaries of a reasonable expectation of privacy. Thus, Michael Zimmer gives the example of a study by Harvard graduate students of the Facebook profiles of Harvard undergraduates. If an undergraduate had made some information visible only to other Harvard students (a choice Facebook's software allows), and a Harvard student-researcher sees it, does that change Solberg's analysis?

A second question concerns the authority of university research offices and IRBs to insist that researchers abide by website terms of service. Notably, the Indiana and UMASS policies do not cite federal human subjects regulations as their authority. Rather, they claim that Facebook and Myspace "explicitly state that their sites are not intended for research but for social networking only."

Solberg writes that evaluating such claims is "outside the scope of this article," but they are interesting in three ways. First, they may be factually false; I could find no such explicit statements in the Facebook or Myspace terms of service. Second, they are divorced from federal regulation. For example, the Facebook terms of service do not distinguish between living and dead Facebook members, whereas federal human subjects protections apply only to the living. Finally, they are internally inconsistent. If Facebook and Myspace did prohibit the use of their sites for research, would not researchers still be violating the terms of service even if they got signed consent from individual members, as allowed by the policies? Just who are these two universities trying to protect?

Solberg concludes that "Unfortunately, and somewhat surprisingly, the OHRP has issued no guidance pertaining to Internet research in general, let alone guidance specifically relating to the issue of data mining on the Internet." To give the feds some credit, in summer 2010 (after Solberg wrote her article), SACHRP did sponsor a panel on the Internet in Human Subjects Research. It can take a long time from a SACHRP presentation to OHRP guidance, but the wheels may be moving on this one.

---

Note, 19 November 2010: The original version of this post identified Ms. Solberg as a law student. She has in fact graduated. I have also changed the link about Michael Zimmer's work from his SACHRP presentation to his article, "'But the data is already public': on the ethics of research in Facebook," Ethics and Information Technology 12 (2010): 313-325.