Substantive Dimensions of the Deliberations

Forum rules

We encourage contributors to the Discussion Board to publicly identify by registering and logging in prior to posting. However, if you prefer, you may post anonymously (i.e. without having your post be attributed to you) by posting without logging in. Anonymous posts will display only after a delay to allow for administrator review. Contributors agree to the QTD Terms of Use.

Instructions
To participate, you may either post a contribution to an existing discussion by selecting the thread for that topic (and then click on "Post Reply") or start a new thread by clicking on "New Topic" below.

The transition to Stage 2 of the deliberations is currently underway but will take some time to complete. In the meantime, we very much welcome additional contributions to the existing threads in this forum.

For instructions on how to follow a discussion thread by email, click here.

Evan Lieberman
MIT/Political Science
Posts: 1
Joined: Sun Apr 24, 2016 8:10 am

A Brighter Side to Data Dissemination

PostMon May 02, 2016 5:14 pm

I have read with interest the comments posted on this site (and kudos to Alan Jacobs and Tim Buthe for doing such an excellent job promoting thoughtful online engagement.) Within this forum, in the CP newsletter, and in private conversations, a range of colleagues have raised important concerns that point to quite serious limitations to the DART and JETS initiatives and the potential expectations and demands that those initiatives imply.

Virtually by definition, qualitative research is eclectic, and observations are not easily ordered into rows and columns, which makes dissemination — an operational implication of “research transparency” — potentially challenging. Most observations are recorded in the form of transcripts, which in turn generate privacy concerns that are not as easily protected as in the case of sensitive, individual-level quantitative data, based on closed-ended questionnaires. When people talk, they reveal personal and self-identifying information. As a result, many contributors to this forum highlight that there will be no easy qualitative analog to the archiving of a quantitative dataset + codebook + code. And as a scholarly community, we clearly need more time to test the waters with best practices for dissemination.

That said, I fear that the discussion has been disproportionately negative with respect to the likely consequences of specific initiatives towards greater “transparency” or “explicitness” (hat tip to Craig Parsons for the latter) that involve dissemination of data, protocols, transcripts, archives, or other materials that provide a view of how research was conducted and the data that were generated in a form that would not be appropriate for publication within the context of a journal article.

I think a more balanced discussion needs to consider not just the downsides, but also the possible scholarly and professional gains of incentivizing greater dissemination of such materials for individual qualitative/mixed-method scholars: Organization of data and research in a manner that is amenable for sharing is likely to force scholars to be more self-conscious about the quality of those data and how they are analyzed; Such data are more likely to become used and cited public goods, providing attention to individual scholars and their scholarship; such dissemination may, in some cases, increase readers’ interest in and confidence in the soundness of the findings. Particularly for graduate students and junior scholars, the dissemination of additional materials could lead to much greater attention to their work, engagements in interesting collaborations, etc., as is frequently the case among quantitatively-oriented scholars when they share their data. Is this potentially “extra” work, and might it be difficult to do? Yes. But if done well, with careful ethical foresight and oversight, I think it could be rewarded in terms of recognized quality of research, citations, etc.

What should those practices be? That is a discussion I wish we were having. At the very least, greater attention ought to be paid to the efforts of contributions such as Elman, Kapiszewski, and Vinuela (2010) and Moravcsik (2010, 2014), which seem to me to be positive efforts to make dissemination relatively straightforward. As has been well noted, many types of scholarship may not fit well with these strategies. On the other hand, much qualitative research does not involve highly sensitive individual-level materials and might benefit from the use of such tools. The objection that these strategies will not work for everyone does not mean that they won’t work for anyone. Just as high-quality qualitative research demands creativity and recognition of context, so too should the development of strategies for sharing such work.

To be clear, there are clearly costs and risks associated with increasing the norms around “research transparency.” But to view dissemination as merely a “response” to suspicious reviewers I think loses sight of the rich scholarly contribution that such practice might entail. As a policy, I would rather see data dissemination be an encouraged “opt in” option (thanks to Regina Bateson and Kathy Thelen for sharing important views on this), rather than a mandatory practice that would force some scholars to explicitly “opt out.” We should reward scholars who provide clear and well organized public goods in the form of supplementary information. But we should not punish those who feel that such dissemination is impossible or unnecessary beyond what they report in their main published work. Norms and standards should develop more organically within research communities — and readers and reviewers will either come to demand a certain degree of dissemination or they will not.

There is a potentially brighter side to the practices associated with increased and explicit attention to “research transparency,” and while I am against mandating any particular practices at this moment, I am certainly in favor of encouraging efforts and experimentation.

Post Reply

Marcus Kreuzer
Villanova University
Posts: 26
Joined: Sat Apr 16, 2016 9:48 am

Re: A Brighter Side to Data Dissemination

PostTue May 03, 2016 9:29 pm

I like Evan's emphasis that qualitative transparency standards are still emerging and thus will require experimentation. I think experimentation could easily be extended beyond data dissemination. It would be particularly encouraging if journals themselves would lead the way with several possible experiments:
1. Facilitate post publication online discussion as is the norm in many of the natural sciences.
2. Be more welcoming to submissions from different stages of the research cycle, that is, not to privilege the standard hypothesis testing article. A point Evan emphasized in a recent conference paper.
3. Provide technology for active citations.
4. Maybe even bring back the footnote rather than hiding transparency comments bound for footnotes in endnotes.
5. Don't count footnotes towards the overall word count. This would remove a powerful disincentive to include transparency commentary.
6. Place greater value on literature reviews, especially the ones where the author carefully and thoughtfully cross-examines competing explanations. This strikes me as a key element of analytical transparency.
Such experiments might create incentives for scholars to experiment to see what works and ultimately transform transparency criteria through practice rather than by editorial fiat.

Post Reply