Saturday, May 8, 2010

Researchers Deceive Thousands of Professors

Professor Andrew Gelman reports that he was sent a deceptive e-mail as part of a research project by two business professors, Katherine Milkman and Modupe Akinola. Milkman and Akinola wished to see if "students from underrepresented groups" (presumably racial and ethnic minorities, and perhaps women) would be less likely to gain the interest of doctoral faculty than "other students" (i.e., white guys). So they sent e-mails to Gelman and about 6300 other professors in PhD-granting departments at American universities. The messages, purportedly from a student planning to apply to PhD programs and wishing for a brief meeting, varied by the name of the student and by the time of the proposed meeting. When a professor answered, they replied to cancel the meeting. The idea was to see if students with white-guy names received more or fewer invitations to meet than others.

After receiving a debriefing e-mail explaining the sham, Gelman replied to the authors that "My helpful impulses toward inquiring students are being abused by this sort of study, which I think belongs in the trash heap of ill-advised research projects along with Frank Flynn's notorious survey from a few years ago when he tried to get free meals out of NYC restaurants by falsely claiming food poisoning." He later elaborated on his blog, "What bothers me is that we were involuntary participants in the study. The researchers took advantage of our time and our good nature (that we were willing to meet with an unfamiliar student). Not cool."

Gelman's post has spurred several comments. Some feel the study should not have been conducted at all: "I really want . . . a formal apology from both institutions, and an acknowledgment that this project should never, ever have gotten IRB approval." And "Since the researchers took the subjects' time without asking, I think they're guilty of stealing something of this magnitude. People often get put in jail for less."

Others think Gelman is being too sensitive: "Everyone is so precious about this. . . . The fact everyone here wrote, read or commented on this blog post suggests that a few minutes here and there don't cost anyone much." Another cites the work of Peter Riach and Judith Rich, who found that the "minimal inconvenience" imposed on the unwitting subjects of such field experiments can be justified by the "degree of accuracy and transparency which is not available from any other procedure." [Peter A. Riach and Judith Rich, "Deceptive Field Experiments of Discrimination: Are They Ethical?" Kyklos 57 (2004): 457-70.]

Gelman himself is in the middle; he thinks that deceptive surveys are OK, he just wants the researchers to send $10 in compensation to everyone who received the e-mail. "Then, after they send us the study results, if we think the findings are interesting, we can each individually decide whether to send the $10 back to them."

The lesson here is that just because a study is approved by an IRB (two, in this case), doesn't mean it won't leave some participants feeling abused. Nor should IRBs strive for such innocuity; they need to weigh the values of free choice and honesty against the values of knowledge and freedom.

I would be curious to see the original protocols and to learn what reactions Milkman, Akinola, and the two IRBs expected. If they did in fact anticipate that a number of recipients would feel as Gelman does--misused, but only to the extent of $10 in damage--the idea of a compensation fund for the outraged isn't bad. (Though perhaps it could require participants to ask for the money rather than having payment as the default.) If the researchers and the IRBs didn't expect this reaction, there's an opportunity here for some good empirical research. Or, as one commenter put it, "Maybe it's a double dummy study design where they wanted to see how easy it is to annoy professors with email."


PCM said...

Too precious indeed. It would be unethical only if they didn't cancel the appointment. It would also be unethical if they named or accused individual professors of being something (like being racist) based on their response to an email.

But I for one am curious about the aggregate results. And while I do not wish to receive any more emails than I do, nor would I welcome deceitful email, there is benefit in this study. And that benefit should be weighed with the (minimal) level of inconvience to the subject. I assume that's what the IRB did.

Besides, this study is very similar to many that have asked for job-interview call backs. I didn't hear any objection to that study. Is the time of a corporate PR people so less important than overworked professors?

Zachary M. Schrag said...

I see your point. This evening a door-to-door solicitor awoke me from a much-needed nap. I thought about demanding $10 in compensation, but then I remembered that I live on Planet Earth.

Michael H. Court said...

I was actually one of the subjects of the study - and based on the "debriefing" message I was sent the actual purpose of the study was to see if there was a difference between being asked to meet the prospective student "now" (today) versus "later" (next week). Based on reading the PIs published worked - mostly popular press articles - it seems that this has something to do with "impulse buying" - which I guess makes sense for a business school researcher.

The effect of minority/gender was stated as an afterthought (secondary hypothesis) - perhaps in response to initial IRB review.

I don't mind being deceived in this fashion provided it is a well-designed study that is intended to benefit society as a whole - presumably criteria that the IRB should have considered. However, I don't see that either of these criteria were fulfilled.

Zachary M. Schrag said...

Thank you for this comment.

I rather doubt this is a study of impulse buying. Professor Akinola's website explains that she "explores biases that affect the recruitment and retention of minorities in organizations," and she has published in that area. Professor Milkman recently co-authored a paper entitled, "Will I Stay or Will I Go? Cooperative and Competitive Effects of Workgroup Sex and Race Composition on Turnover." So I am inclined to assume, as do the commentators on Professor Gelman's blog, that the researchers' interest in "various backgrounds" is far more than an afterthought.

Your comment raises the question of whether subjects of an IRB-approved study should have the right to review the protocol. Currently, the IRB system is not well set up to consume or produce empirical research, and making protocols public would be a step in the right direction.

Anonymous said...

Even if each of the 6300 recipients averaged only five minutes to read the email, check their calendar, make a decision, and respond (and I spent considerably more time, as did manyothers , lining up resources from administrators and lab operators), then the total time wasted comes out to more than one dozen full work weeks. All in service to the selfish interests of these researchers.

Anonymous said...

I too was subject to this email scam, and am amazed that an IRB could approve direct lies to participants - it certainly would not get by a review board in communication disorders where I do my research. But I guess telling lies is par for the course in business.

I emailed to complain and of course was ignored by these scam artists.

Anonymous said...

Sounds like quite an interesting study. I'd be a little bit annoyed by the fake email, but I'll be interested to see the results.

I am a little surprised that this passed the ethics review, but it should be interesting to read!