Qualitative Transparency Deliberations

[Alan Jacobs University of British Columbia]

Many thanks to all those who have been contributing to this agenda-setting discussion, and to those who will do so in the days to come.

At the beginning of next week, the QTD process begins the transition to Stage 2, in which specialized Working Groups will consult and deliberate about the meaning and practice of transparency for specific kinds or aspects of qualitative research. Each Working Group will, at the end of this process, be tasked with drafting a Community Transparency Statement that articulates the understandings of research transparency that are current among scholars undertaking a particular form of qualitative research. See here for more on the next phases of the QTD.

The Steering Committee would like to get your thoughts on how the foci of these (roughly 12 to 15) Working Groups should be defined. One way to define the Groups might be according to different forms of research that raise distinctive sets of transparency issues and involve different kinds of openness practices. So, for instance, we might have evidence-focused Working Groups on topics such as interview evidence, documentary evidence, ethnographic observation, and so on; and analysis-focused Working Groups on process tracing, interpretation, comparative analysis, and so on; and context-focused Groups on, e.g., research in contexts of political violence or repression.

Creating groups with particular foci will allow for differentiated discussions that take proper account of the particular purposes and logics of inquiry animating different forms of qualitative research and of the particular challenges and constraints facing scholars engaged in different kinds of research. At the same time, we don't want to preclude the possibility of developing or identifying some shared understandings across approaches (see Elliot Posner's comment on this), and we may want to hone in on important issues that cut across methods. So, for instance, we might want to create Working Groups devoted to cross-cutting issues, such as human subjects protections or the role of editorial standards and practices in relation to transparency.

These are just a few possible ways of thinking about how to structure Stage 2 of the deliberations. What do you see as the approaches or issues that merit separate Working Groups?

[Pepper Culpepper European University Institute]

Thanks to the organizers and all the contributors to this conversation, from which I have learned a great deal. On the practical points of where to go next with concrete groups, I have two propositions, which reflect the points that have resonated most forcefully with me. I want to start out by saying, as others may have better than I, that under the rubric of transparency of we have been having a discussion of what it means to do good political science, particularly with respect to the use of the various qualitative methods employed in the discipline. One of those discussions has been about how we should perform and report and archive interview data, which is central to my own research. These discussions have changed the way I think about doing my own work and how I will teach interview methods to my students in the future. I have much more to say on this subject, but others have spoken eloquently already on it, and I think it is clear that one of our groups should be devoted to the problem of interview methods and evidence. That is probably not controversial.

Equally importantly, though, I am a scholar of power. And this subject too has been very much a theme of the deliberations -- Kurt Weyland, Jane Mansbridge, and Gary Marks all made important points on this issue that I endorse fully. We are having this useful discussion of how to do good political science under the transparency rubric, but I would not want the establishment of 'technical' working groups to devolve into 12 working groups about 'how to be more transparent in research method x' and then have no working group devoted to the subject of the likely effects of DA-RT, JETS and the transparency discussion on the representation of various sorts of research traditions in our leading journals, and thus the discipline. To paraphrase Heclo, political scientists not only puzzle, they also power. We are getting better as a discipline by having these discussions about how to make our research more transparent; but we are also having a sustained argument about whose articles will get published in which outlets, and in which proportion, and thus who will wind up getting tenure where and then training the next generation of scholars. The consideration of the likely effects of formal changes in transparency norms -- which is to say, in how power will be (re)allocated in the discipline as a result of these discussions and guidelines -- is worthy of its own working group.

[Guest]

I think the QTD Working Groups, however defined, ought to move beyond transparency and tackle the issue of reproducibility.

That is:

1. If reproducibility means different things to different people, what does reproducibility mean to qualitative researchers?
2. Does reproducibility matter to qualitative researchers?
3. How can qualitative research be reproducible more than it currently stands?

[Nikhar Gaikwad Yale University]

Thank you for what has proved to be a very thought-provoking set of discussions thus far. This thread asks us to think about the aspects of research transparency that could benefit from differentiated deliberations. My preferred approach is a hybrid approach, in which some working groups will focus on issues that cut across methods, while others will focus on distinctive transparency issues that arise in specific research traditions.

In the first category, the role of human subjects protections, guidelines on what constitutes qualitative data access, whether and how transparency standards should be sensitive to the costs of data sharing and replication, the impact of data sharing on the ability of researchers to conduct different kinds of research, and the role of editorial standards in journals seem to be issues that apply broadly to different research communities. It might be useful to obtain focused deliberation on these topics, with a goal of generating a common set of principles that can guide scholars employing different types of qualitative research methodologies.

In the second category, ethnography, interview-based research, interpretive political science, research in contexts of political violence, research in authoritarian settings, research based on documentary evidence, qualitative comparative analysis, and APD appear to be areas that have already generated vigorous deliberations on this forum. Working groups that consider transparency issues unique to these research communities can help propose guidelines that supplement (or even modify) the prescriptions offered by the previous set of broad-based working groups.

Of course, these categorizations are by no means exhaustive. I have proposed thirteen working groups based on the ongoing discussions on this forum, but additional groups can easily be accommodated within this framework. My basic point is that differentiating deliberations according to a mix of both thematic and research-specific considerations is a good idea. This will allow researchers from different qualitative research communities to benefit from a set of shared principles, while being guaranteed focused deliberations on transparency issues in their own research communities. As an example, the working group on human subjects might propose a common set of transparency norms for all researchers working with human subjects, while the working groups on research in authoritarian settings or research in contexts of political violence can further delineate guidelines on human subjects protections (along with guidelines on other issues) for researchers working in these specific research traditions.

An important benefit of this approach is that researchers can easily evaluate how transparency norms in their own research communities relate to broader norms in qualitative political science research, as well as to norms in comparable research communities. Additionally, scholars unfamiliar with qualitative research methodologies, policymakers, and the broader public can easily evaluate points of convergence and divergence in transparency norms across different qualitative research communities without having to master the nuances of each community.

One potential concern is the possibility of competing streams of advice emerging from across the two sets of working groups. Perhaps the Steering Committee can facilitate the sharing of information between different working groups in order to mitigate this concern. For example, the second category of working groups (i.e., those organized by research communities) can be asked to review the guidelines proposed by the first set of broad-based working groups before finalizing their own guidelines.

I have learned a lot from the discussions on this forum, and I look forward to following closely the second stage of deliberations.

[Alan Jacobs University of British Columbia]

Thank you, Pepper, Nikhar, and Guest for these excellent suggestions. We've also received comments on the Working Group structure in a couple of other threads:

a working group with journal editors

Transparency in APD Research