Qualitative Transparency Deliberations

[hillelsoifer]

I wonder whether it might be useful to discuss the actual policies put in place by journals in response to DA-RT? I personally found the statements of the editors of the APSR and CPS in the CP newsletter to diverge sharply from the fears raised by many on this board, and think that they might provide groundwork for scholars to respond to knee-jerk criticisms of qualitative work without falling prey to many of the concerns raised in this conversation. I should note, however, that my work is largely archival, and mixes qualitative and quantitative work, and thus may not have to confront the same issues raised by other scholars posting here who work with human subjects or in more immersive modes of research. I'd be interested to hear, therefore, what others might think about the standards in practice.

Hillel Soifer

[Alan Jacobs University of British Columbia]

Thanks very much for this comment, Hillel. It would indeed be helpful to hear what people think of those journal policies that are currently in place. Judging by their online author guidelines, many policies of JETS journals remain rather general or appear to still be under development. Some indicate specific rules for quantitative research, but not for qualitative. But what do people think of the policies for qualitative research that are have been posted so far?

Here are links to the policies of a few journals that say something in particular about transparency for qualitative research:

- APSR: The APSR's submission guidelines are here. And a set of FAQs about the APSR's transparency policy is here.

- The Italian Political Science Review provides one of the more specific statements about transparency for qualitative research. Scroll down to "Data access and research transparency in qualitative research."

- Conflict and Cooperation's policy, which discusses both qualitative and quantitative forms of evidence, is here (click on "Submission Guidelines" tab, then "Data" section).

- Research and Politics's policy is here

- Perspectives on Politics, while not a DA-RT signatory, has issued "Supplementary Materials and Data Guidelines" that separately discuss quantitative and qualitative research.

- European Political Science Review, also not a DA-RT signatory, states a transparency policy here under point 3.

- The editors of Comparative Political Studies have indicated that they want to hear from the QTD process before putting their transparency policy for qualitative research into place.

The Comparative Politics section newsletter issue that Hillel refers to is here.

[Guest]

A number of posts extended invitations to either make specific transparency recommendations or to comment on existing ones. So, I looked at the guidelines of the journals that Alan kindly listed above.
Here is what I found.
1) The APSR was the only journal trying to spell out in some detail transparency guidelines. The other journals either explicitly reject them or show little dialogue with DA-RT related concerns.
2) The APSR guidelines for qualitative and historical/archival research are little more than re-articulations of current practices. Scholars working with documents are reminded to be careful with their footnotes and, if adventurous, maybe consider active citations. Quantitative scholars, in turn, are required to make their data more readily available. Pretty déja vue stuff.
3) The APSR guidelines were the vaguest for ethnographic or interview-based research. This combine with the greater explicitness for quantitative and archival research might contribute to the impression that research involving direct human subjects might be expected to adhere to practices of other research traditions.
4) The APSR guidelines were surprisingly vague and tentative in spelling out analytical transparency. If you think of analytical transparency (e.g. "clearly mapping the path from evidence to claims.") in terms of valid causal inference, then the question becomes how specific/strict do you want to get about analytical transparency? Is active citations the new gold standard? Or do you want to include Bayesian consideration of test strength? Or should scholars be expected to spell out the underlying ontological assumptions about their data (i.e. conditional independence, uniformity)? Analytical transparency could be a real Pandora's box where it is not clear how it differs from the key issues we commonly discuss under the labels methodology or epistemology.

[Ian Lustick University of Pennsylvania]

I'd like to raise an infinite regress question. If editors are unwilling to assign credibility to judgments offered by the scholars who submit manuscripts for publication unless those scholars meet requirements for transparency and replicability, then how can those editors assign credibility to the scholars who offer positive or negative judgments about those manuscripts in the peer review process unless each reviewer's review is also subject to those same standards of transparency and replicability? How many of us, as manuscript reviewers, have seethed at the validation and verification errors of reviewers whose errant and discreditable judgments often drive editorial decisions? Why isn't it true, from the DA-RT perspective, that what is good for the manuscript submitter is good for the manuscript reviewer?

Presumably the answer is that it is difficult enough to recruit reviewers even if they are not required to make transparently available the inferential and data bases for their recommendations. From a practical point of view, in other words, editors might say they simply cannot function unless they excuse reviewers from the process that they nevertheless claim is necessary for the credibility of the scientific progress. OK, but if there is room for practical compromise so that scientific work can proceed, then there is no room for an absolutist imposition of DA-RT standards since the editors are already conceding that scientific judgments can be made, and can be acted upon as credible, without honoring those standards.

Ian Lustick

[Mala Htun Univ of New Mexico]

[quote="ilustick"]If editors are unwilling to assign credibility to judgments offered by the scholars who submit manuscripts for publication unless those scholars meet requirements for transparency and replicability, then how can those editors assign credibility to the scholars who offer positive or negative judgments about those manuscripts in the peer review process unless each reviewer's review is also subject to those same standards of transparency and replicability?" "if there is room for practical compromise so that scientific work can proceed, then there is no room for an absolutist imposition of DA-RT standards since the editors are already conceding that scientific judgments can be made, and can be acted upon as credible, without honoring those standards."

EXCELLENT POINT!!!!

[Guest]

Very much agree with Ian Lustick's point.

[Amel Ahmed University of Massachusetts, Amherst]

I think part of the problem is that there are a lot of conflicting statements. I've been following this debate from the beginning (though probably not as closely as I should have) and I still don't really have a handle on what is proposed. Figuring out the requirements for archival work for example – yes, most of the journals simply say that we need clear citations identifying the archive, location, etc., but then the original APSA statement seems to suggest that we would need to make the original source material available so as not to place too great a burden on those who would want to track it down. So is that still on the table? If so, that would be ludicrous for most archival research. How would I make my microfilm and rare documents available? Then there are the AJPS badges, which make no sense for most qualitative work, esp. archival. The point it, there are a lot of different statements out there, many of them conflicting. I think you're right Hillel that so far the statements seem rather benign, but I suspect the journals have been somewhat tentative about their policy statements, in part due to the push back. But those policies are likely to evolve and it is in our interest to make sure they evolve in a way that makes sense.

Also, I very much appreciate Ian's points. Even outside of the DART debate, I've always been surprised by the weak evidentiary standards for reviewers.

[Ingo Rohlfing]

Ingo Rohlfing, University of Bremen:
I am not sure I get Ian Lusticks's point right, which is a valuable point (if I am getting it right): I guess we all agree peer review does not work perfectly. But I am not sure what the reviewers should be transparent about. In my reviews, I rarely make claims that needed to be verified by DA-RT standards. If a theoretical argument is bad, then it is a theoretical point. If the choice of cases is not discussed, then I criticize this and I usually give the part in the manuscript to which I refer. If I review a QCA study, it might happen that I try to reproduce the result if I have access to the data. If I find something in the analysis I consider weak, I submit my analysis to the editors and do not simply make a claim in my review.
In any case, the shortcomings of the peer review system should not be turned against DA-RT. If at all, we might use DA-RT to make peer review better. Transparency of the peer review process is an important issue and making the reviews publicly available is one of the issues that is regularly raised in this debate.