Tuesday, July 28, 2009

A Defense of RECs

Professor Adam Hedgecoe of Cardiff University kindly pointed me to his article, "Research Ethics Review and the Sociological Research Relationship," Sociology 42 (2008): 873-886.

The article is a response to longstanding criticisms of British research ethics committees (RECs), especially those affiliated with the National Health Service (NHS). For example, Sue Richardson and Miriam McMullan surveyed "UK academic social researchers working in Health, or health services researchers, who had experience of using the NHS research ethics process prior to March 2004," in "Research Ethics in the UK: What Can Sociology Learn from Health?," Sociology 41 (2007): 1115-1132. Fifty-one percent of their respondents reported degrading their research design as a result of the committee approval process, while only 32 percent reported making changes for the better. Overall, 59 percent offered negative comments, while only 15 percent offered positive comments. And Richardson and McMullan set a pretty low bar for a positive comment, counting this: "It’s a lot of paperwork but once you know what is required, it’s acceptable." Overall, it seems, NHS RECs are inhibiting the sociological study of health care in the United Kingdom.

Hedgecoe seeks to rebut this impression, based on his observation of three NHS RECs in 2005 and 2006, and some follow-up interviews. He argues that "NHS RECs are not inherently hostile to social science research, especially qualitative research." (882) The double-negative construction of that thesis suggests Hedgecoe's problem: he's trying to prove that something doesn't happen, or at least not as often as ethics-committee critics believe. That's not an easy task, and I congratulate him for trying. But I find the article unpersuasive.

The article offers mostly generalities, rather than the detailed stories that are the heart of a good ethnography. For example, though telling us he "took extensive notes (including verbatim quotes)" (876), Hedgecoe offers only two vertbatim quotes from committee meetings in the whole piece. It's also short on quantitative data. Hedgecoe notes that he observed 33 committee meetings, but he doesn't say how many ethnographic projects came up for review. He describes only four, one of which, his own, wasn't technically part of his study.

Here are the four:

1. Hedgecoe's own research on RECs.

After begging to have his project reviewed by an REC, Hedgecoe received approval, including permission to sit in on committee meetings without the consent of the researchers applying for approval for their own projects. Instead, he boasts, he needed only to anonymize the people and institutions he observed. (881) Of course, Hedgecoe's determination to anonymize "people, places, and events" may have resulted in the vagueness of his findings, so I'm not sure I count this as a win. Rather, it may be an example of the blandness that critics see as one of the results of ethics review.

2. A cancer ward study:

When reviewing an application proposing qualitative interviewing of nursing staff working on a cancer ward about the impact of their work on their lives, Coastal MREC presented the applicant with a technical bureaucratic problem caused by the researcher’s independent status. The chair’s reaction to the researcher’s concern over this point was to say, 'This is not a problem we are throwing at you, but a problem we are trying think round', and indeed the solution suggested by one member (which was to affiliate with an academic institution the researcher had previously collaborated with) solved this particular problem. (879

Here the committee offers not ethical judgment, but advice on how to deal with an ethically empty bureaucratic rule. Who is forbidding independent research in the UK?

3. "A proposal to study specialist paramedics with a view to evaluating the role, seeing whether it reduced the number of people who were repeatedly admitted to A&E (so called ‘frequent flyers’)." (881)

Did you notice the word "number" in that description? Why is Hedgecoe using a quantitative project to show that RECs are friendly to qualitative research? It sounds to me as though a medical REC would have no trouble fitting this study into the clinical model.

4. Finally, the most interesting case, the "nurses study":

a senior nurse applied to do research as part of an MSc, looking at nurses' attitude towards performance-related pay. Although the study was going to be on the nurse's own team, the committee was generally inclined to approve the application since the results would be restricted to a dissertation. But when the applicant came before the committee it became clear that she had wider goals for the results of this work, including feeding into policy decisions. She also made statements that worried the REC, about wanting to work with 'people I trust and who trust me'. The REC suggested that the applicant study a team at another hospital, to avoid the issues of conflict of interest, but the applicant was not happy, claiming the REC was 'stifling research'. The committee directed her to the head of nursing research and apparently the applicant went so far as to complain to COREC, NRES’s predecessor. (879)

Hedgecoe applauds the REC, describing its decision as "deft" and writing that "researchers are often oblivious to the potentially coercive nature of supervisors asking those they manage to take part in research, especially research which may require them to reflect on their practice." (880)

I'm not persuaded that the REC made the right call; it seems like the nurses were denied a chance to collaborate in research and to shape policy in their workplace. But the bigger question, which Hedgecoe does not explore, is why the researcher felt stifled, rather than enlightened, by the guidance she received. Wouldn't an effective committee be able to persuade her that her project was flawed, rather than leading her to file a formal complaint?

Hedgecoe's silence on this question may result from his research design. While he interviewed "policy-makers, pharmaceutical industry researchers and executives, and, most pertinently for this article, members of research ethics committees themselves," it is not clear that he spoke with any of the social science researchers who sought approval from the committees he observed. They might have a thing or two to say about the way they were treated. (875) But Hedgecoe's comments about getting informed consent from everyone except researchers suggests he avoided them.

That's a pity. Hedgecoe cites with approval the "rigorous empirical data gathering" of Bradford Gray (874), without absorbing one of Gray's key arguments: it's not enough to observe an ethics committee; one must also talk to those affected by its work. For Gray, that primarily meant research participants, but it also meant researchers, at least in his work for the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. (I'm afraid I don't have handy a copy of Gray's 1975 book, Human Subjects in Medical Experimentation, so I can't recall if he spoke to researchers for that, or only committee members and research participants.) Instead of talking to researchers, Hedgecoe imposed his own judgment that they were being treated fairly.

I am glad to learn that some NHS RECs allowed two or three projects to proceed without much difficulty. But given the much more extensive data--qualitative and quantitative--in the Richardson and McMullan study, I am little reassured by Hedgecoe's assertions that RECs are doing just fine.


Adam said...

Response part i)
Hi Zach
I must confess to being a bit bemused by your dyspeptic response to my article on NHS RECs’ evaluation of qualitative research. It’s a shame you have adopted such a snide tone in your commentary on my work (I was ‘boasting’ when I pointed out that my own work got through REC review. Really?); I was always taught that the best academic arguments are those that give the most generous reading of their opponents’ position. You clearly abide by another set of principles.

Be that as it may, I wanted to clarify a few points in my article that you seem to be a bit confused about. Firstly, you make rather a banal point about my article lacking “than the detailed stories that are the heart of a good ethnography”. Well quite, but then special issues of _Sociology_ have a 6,000 word limit, which rather cramps one’s style on the ethnographic detail front. The issue of word length is a common problem faced by those of us writing ethnographic articles for journals, and is, perhaps, why ethnographers still place so much value on book length presentations of their work, given the space these give us.

Secondly, you rather miss the point about the cancer ward study. You correctly note that “Here the committee offers not ethical judgment, but advice on how to deal with an ethically empty bureaucratic rule”, but fail to accept that this emphasises the generally supportive attitude RECs have towards (all) research. One problem with your wide critique of research ethics review, certainly as it pertains to the UK, is a failure to distinguish between committees, and the system of rules within which they operate (which are drawn up and revised by the Department of Health). To some extent, they are in tension (as is often the case in bureaucracies). A more generous reading than yours would accept that this description of REC behaviour (helpful, supportive of research) is at odds with the general picture painted of RECs by social scientists (and other researchers) who have applied to such committees, and thus worthy of interest....(tbc)

Adam said...

Response Part II
Thirdly, in your keenness to suggest bad faith on my part (“Why is Hedgecoe using a quantitative project to show that RECs are friendly to qualitative research?”) you simply parade your own ignorance about social science methodologies or, alternatively, your failure to read my article properly. You seem to assume that because the overall goal of this study was to reduce the number of a particular event, that ONLY quantitative methods can be used. This is plainly daft: as I say in my paper, the application assessed by the REC was a QUALITATIVE observational study of paramedic behaviour, within the context of a broader piece of work that sought to reduce the incidence of A&E ‘frequent flyers’. Methodologically it makes no sense to claim, as you seem to do, that because the overall study has something to do with numbers, the individual study assessed by the REC must therefore be a “quantitative project”. There are more methodological problems with your assumption that Richardson and McMullan’s questionnaire survey of researchers’ opinions of REC review somehow tells us something about the INTERNAL processes that RECs use to make their decisions. While such surveys of researchers’ opinions are interesting and useful, they do not investigate the way in which RECS make decisions, only researchers’ views on REC decisions and the public reasons given for them. My research was interested in the INTERNAL processes by which RECs reach their positions (in the case of this article, with regard to qualitative research), and thus Richardson and McMullan’s work has little to say about the content of my research . It certainly doesn’t ‘trump’ it, as you seem to think.

You cite Gray's work without noting that he didn't _observe_ IRB decision-making in his study, but rather drew on external data about IRB performance. The methodological question is this: which is the best source of data on the way in which RECs actually make decisions. Is it: a) to draw on the views of people who were not present at the REC meeting and only have access to the formal justifications for the decision or b) attending REC meetings to see how they make decisions and interviewing members of those committees afterwards? I think the answer is b) and that the data gathered by a), while interesting, does not tell us much about the internal mechanics of REC decision-making.

Unlike you, I do not have an axe to grind with regard to research ethics review bodies (IRBs, RECs, whatever). I think they are interesting and I carry out empirical research on them to find out how they work, and how they have developed over time. I don’t have an ideological stance with regard to these bodies. I try and follow the evidence.

adam hedgecoe.

Zachary M. Schrag said...

Thank you for these comments.

I am surprised by your statement that "my research was interested in the INTERNAL processes by which RECs reach their positions," since the article has so little to say about those processes. It presents the RECs' positions on four studies without telling us how the RECs reached those positions: who initially proposed the recommendation, who opposed it, what alternatives were considered, and so on. A contrast is the work of Laura Stark, who explains in detail the sorts of interactions that take place among committee members.

Perhaps 6000 words is insufficient to present both your argument and supporting evidence. If this is the case, you can hardly expect a 6000-word presentation of your views to persuade someone not already sympathetic to them.