Beyond the qual/quant divide
Posted: Sat Apr 09, 2016 9:33 pm
Having read some of the previous posts - and paid some attention to the debate carried out in other venues over DA*RT - i have lots of thoughts, but let me try to boil things down so i don't take up too much of everyone's time...
Insofar as ours is an empirical endeavor the goal of transparency would seem to matter greatly. If an argument is based on some sort of evidence (be it interviews, observations, texts, pictures, shards, whatever) then it is a problem if the author cannot produce the evidence, as well as the form of analysis the researcher used to reach his/her conclusions. I grant that it is not always possible to do this, and i'll discuss some of these problems below. But my point is (I think) a simple one: transparency is good, ceteris paribus.
The debate over DA*RT rests on the ceteris paribus conditions. What sacrifices are entailed, and are they worth the costs (or in what situations might they be worth the cost)?
In answering this question there seems to be a common view that the obstacles are greater for quals than for quants. I agree, but only to a point. Let me give some examples...
1. Protecting sources. This is a persistent - and very real - worry about transparency. But it is certainly not unique to qual work; indeed, there are elaborate procedures for protecting the anonymity of survey respondents. Nor does it apply to all qual work, e.g., to research based on archives or published work (as Lisa Wedeen notes). Protecting sources, in my view, runs orthogonal to the qual/quant divide.
2. Costs for the researcher (time and money, if RA's are involved). Again, this is a persistent and very real concern. But it is certainly not unique to qual research. To be sure, it may be somewhat more time-consuming for qual researchers to prepare their notes (or other material) for archiving, including anonymizing sources, where necessary. But this is a matter of degrees, i would think. And the process is eased (i would hope) by the Qualitative Data Repository hosted at IQMR.
3. Proprietarial material. I can think of at least one obstacle to transparency that is found more commonly on the quant side than the qual side. Sometimes data used for research is owned by someone else. While it may be used, perhaps in an aggregate format, the owner of the data does not consent to make the data public. (I imagine there are examples of this in qual work but i can't think of any off the top of my head.) An example in the quant world is the influential World Bank Governance Indicators, developed by Daniel Kaufmann and collaborators. The data used to construct these indices is mostly proprietarial and therefore is not released with each version of the dataset. People are troubled by this - as they should be - but that doesn't seem to prevent publications in (top) scholarly journals.
Other examples might be discussed but i think the point is clear. There are costs to achieving greater research transparency, and they fall on all researchers. Perhaps the burden is somewhat heavier on the qual side, but we ought not imagine that the quantoids are getting off lightly.
By way of conclusion, i would suggest that we focus on the specific methodological issues that are raised by DA*RT rather than whether the work under review is quant or qual. It is perfectly reasonable to insist that no transparency initiative should compromise the confidentiality of sources, for example. But it does not seem reasonable to say that qualitative work should be exempt from the DA*RT, as some contributors to this symposium seem to suggest.
For those who are unhappy with standards adopted by the APSR or other journals, or by APSA, i would issue a challenge: let's work to improve them.
Insofar as ours is an empirical endeavor the goal of transparency would seem to matter greatly. If an argument is based on some sort of evidence (be it interviews, observations, texts, pictures, shards, whatever) then it is a problem if the author cannot produce the evidence, as well as the form of analysis the researcher used to reach his/her conclusions. I grant that it is not always possible to do this, and i'll discuss some of these problems below. But my point is (I think) a simple one: transparency is good, ceteris paribus.
The debate over DA*RT rests on the ceteris paribus conditions. What sacrifices are entailed, and are they worth the costs (or in what situations might they be worth the cost)?
In answering this question there seems to be a common view that the obstacles are greater for quals than for quants. I agree, but only to a point. Let me give some examples...
1. Protecting sources. This is a persistent - and very real - worry about transparency. But it is certainly not unique to qual work; indeed, there are elaborate procedures for protecting the anonymity of survey respondents. Nor does it apply to all qual work, e.g., to research based on archives or published work (as Lisa Wedeen notes). Protecting sources, in my view, runs orthogonal to the qual/quant divide.
2. Costs for the researcher (time and money, if RA's are involved). Again, this is a persistent and very real concern. But it is certainly not unique to qual research. To be sure, it may be somewhat more time-consuming for qual researchers to prepare their notes (or other material) for archiving, including anonymizing sources, where necessary. But this is a matter of degrees, i would think. And the process is eased (i would hope) by the Qualitative Data Repository hosted at IQMR.
3. Proprietarial material. I can think of at least one obstacle to transparency that is found more commonly on the quant side than the qual side. Sometimes data used for research is owned by someone else. While it may be used, perhaps in an aggregate format, the owner of the data does not consent to make the data public. (I imagine there are examples of this in qual work but i can't think of any off the top of my head.) An example in the quant world is the influential World Bank Governance Indicators, developed by Daniel Kaufmann and collaborators. The data used to construct these indices is mostly proprietarial and therefore is not released with each version of the dataset. People are troubled by this - as they should be - but that doesn't seem to prevent publications in (top) scholarly journals.
Other examples might be discussed but i think the point is clear. There are costs to achieving greater research transparency, and they fall on all researchers. Perhaps the burden is somewhat heavier on the qual side, but we ought not imagine that the quantoids are getting off lightly.
By way of conclusion, i would suggest that we focus on the specific methodological issues that are raised by DA*RT rather than whether the work under review is quant or qual. It is perfectly reasonable to insist that no transparency initiative should compromise the confidentiality of sources, for example. But it does not seem reasonable to say that qualitative work should be exempt from the DA*RT, as some contributors to this symposium seem to suggest.
For those who are unhappy with standards adopted by the APSR or other journals, or by APSA, i would issue a challenge: let's work to improve them.