TimButhe wrote:For the kind of content analytic-work you "produce" or "consume," what is the information about research methods and decisions, which you find most helpful to allow you to understand and assess the research? In other words, what kind of research transparency about content-analytic work do you most appreciate when it is provided? What do you consider indispensible but at times don't find included in content-analytic work you read? And how is such information best provided?
Regarding the conversation about transparency standards, it’s easiest (obviously) for me to begin by talking about my own experience. Because I have coded (or managed) the coding of hundreds of variables, I want to start by talking about the set of variables that posed the biggest challenge – i.e., those abstract concepts that many doubt can even be coded systematically: the problems of cooperation that we in international relations have been talking for decades.
The justifications for how I coded the underlying cooperation problems of the agreements in the Continent of International Law (COIL) research program vary a lot with respect to both the amount of research that went into the coding and the degree of documentation. As background, each particular agreement is characterized by the presence (coded as high or 1) or absence (coded as low or 0) of each possible cooperation problem. In reality, all situations are characterized by almost all cooperation problems to some degree, but COIL codes them as existing if they are present in high as opposed to low levels. For example, Uncertainty about Preferences always exists to some degree in any interaction, but a situation has to be characterized by high Uncertainty about Preferences (e.g., Soviet Union and United States during the Cold War for some issues, as opposed to United States and Canada during the same period) for it to be considered present. (Why I decided on such a coding instead of a numerical scale, for instance, can be the subject of another posting if anyone is interested.)
The empirical work that went into the coding of each underlying cooperation problem often drew on materials like memoirs, historical analyses, and the parties’ institutional history, but not always. Why? Let me discuss two endpoints: one in which there was relatively little discussion and hence documentation and one that required a lot of investigation.
Regarding the former, my sample features the Interim Agreement between the US and USSR from 1972 (SALT I). This is a textbook example (literally) of a Prisoners’ Dilemma (PD) game structure, wherein each side ideally would like to defect while the other cooperates. The justification written by the coders for an underlying Enforcement problem (PD) is short without any citations: “Strategic offensive arms are important military tools, and hence there are strong incentives to renege on the agreement while the other side remains cooperating. This attempt at cooperation is subject to the typical arms-race dynamic – a well-known example of a prisoners’ dilemma.”
At the opposite end of the spectrum, my team was extremely puzzled by The Agreement for Environmental Cooperation between Denmark and Oman signed in 1993. The agreement substantively covers solid and hazardous waste disposal and noise pollution. Given the distance between Denmark and Oman, the common explanation for many environmental agreements, which is that they solve the enforcement or free-riding problem that arises in part from the generation of negative externalities in this issue area, was not satisfying. Neither state will benefit in any material way from less waste or noise pollution in the other state. Moreover, if the benefits of minimizing this kind of pollution in Oman were so advantageous to Denmark, why had Denmark not agreed to similar agreements with other, geographically closer states? (This is something that we researched.) Oman itself had not signed a single environmental agreement in the decade before with any state. (Again, something we researched in our investigation.) Why then did these two states decide to cooperate in this issue area?
Research on the issue area and on these two states led to a couple of interesting discoveries. First, Denmark signed five substantively similar bilateral agreements with Slovakia, Belarus, Ukraine, Bulgaria, and Poland in 1994. As in the case of Oman, these five states were at the time far less wealthy and developed than Denmark, not contiguous to Denmark, and, with the exception of Poland, without any history of making environmental agreements. Second, these bilateral agreements closely followed the 1992 Earth Summit in Rio de Janeiro at which environmental development was discussed at length. Given the importance of international environmental protection to the Danish, which was also documented, coders concluded that one of the underlying cooperation problems was Norm Exportation.
I made the decision regarding how much justification and documentation was needed. “Textbook examples” required little (although I did make sure I agreed with the logic) while others required extensive research. At the time when I started this in the early 2000s, there were no guidelines.
Do others have similar experiences with varying degrees of research and documentation? Is that acceptable?