Substantive Dimensions of the Deliberations

Forum rules

We encourage contributors to the Discussion Board to publicly identify by registering and logging in prior to posting. However, if you prefer, you may post anonymously (i.e. without having your post be attributed to you) by posting without logging in. Anonymous posts will display only after a delay to allow for administrator review. Contributors agree to the QTD Terms of Use.

Instructions
To participate, you may either post a contribution to an existing discussion by selecting the thread for that topic (and then click on "Post Reply") or start a new thread by clicking on "New Topic" below.

The transition to Stage 2 of the deliberations is currently underway but will take some time to complete. In the meantime, we very much welcome additional contributions to the existing threads in this forum.

For instructions on how to follow a discussion thread by email, click here.

Nancy Hirschmann
The University of Pennsylvania
Posts: 6
Joined: Wed Apr 06, 2016 1:59 pm

CP newsletter: editorial or ideology?

PostFri Apr 08, 2016 7:44 pm

Having just read the introduction to the Comparative Politics newsletter, I found the lead editorial outlining the "myths" of DA-RT to be (presumably unintentionally) quite ideological. These are not “myths”: they are claims of how the DA-RT principles may be--and in some individual cases already have been--interpreted by reviewers and editors, thus resulting in unfair and inappropriate assessments of academic work deploying non-quantitative methods. There are many fine articles in this newsletter, and we can be grateful to the editors for providing the forum in which these issues can be aired. But the implication of the editorial—again, perhaps unintended--is to skew the conversation in a particular pro-DA-RT way. Isn't this exactly the kind of "bias" that DA-RT is supposed to guard against? Certainly, editors may have particular views—that’s what makes it an editorial, after all. But these should be based on an accurate reflection, and serious consideration, of the wide variety of views expressed on this issue—including some of the articles in that very same newsletter--rather than dismissing them as “myths.” That gesture moves the editorial from “considered judgment” to “ideology.”

Post Reply

Paolo Spada
Southampton
Posts: 11
Joined: Wed Apr 06, 2016 8:24 pm

Re: CP newsletter: editorial or ideology?

PostSat Apr 09, 2016 4:03 am


Post Reply

Mala Htun
Univ of New Mexico
Posts: 16
Joined: Wed Apr 06, 2016 9:20 am

Re: CP newsletter: editorial or ideology?

PostSun Apr 10, 2016 8:28 pm

please post these comments on the website of the CP newsletter, set up to stimulate discussion and responses about the newsletter:
http://comparativenewsletter.com/da-rt

Post Reply

Rudra Sil
University of Pennsylvania
Posts: 6
Joined: Thu Apr 07, 2016 4:50 pm

Re: CP newsletter: editorial or ideology?

PostFri Apr 15, 2016 1:34 am

It is not difficult to see that the editors of the Comparative Politics newsletter are probably somewhat partial to DA-RT. They are certainly entitled to take that position. But, I would probably not think of this as a conscious "bias" for the simple reason that, in the end, they have helped to circulate five excellent essays by different kinds of scholars in comparative politics who all conclude that DA-RT will ultimately do more harm than good for the field. I encourage all to read those five essays by the critics of "DA-RT" - along with the thoughtful responses of the journal editors, particularly of World Politics (Deborah Yashar) and Comparative Political Studies (Ben Ansell and David Samuels).

http://comparativenewsletter.com/files/ ... ng2016.pdf

Post Reply

Marcus Kreuzer
Villanova University
Posts: 26
Joined: Sat Apr 16, 2016 9:48 am

Re: CP newsletter: editorial or ideology?

PostSat Apr 16, 2016 4:39 pm

Two quick observations regarding the point Nancy raises of whether the CP Newsletter embodies a certain ideology:

1) Most people posting work in a qualitative tradition and generally we share horror stories of how our qualitative colleagues butchered a particular case study, lack contextual understanding, overlook boundary conditions, and generally engage the type of historical tourism that makes theorizing easy if not necessarily valid. I wonder to what extent such horror stories are little more than earlier incarnations of DA-RT. Qualitative scholars might be transparent when it comes to their data collection and analysis but they generally lack transparency when it comes to their theorizing and other pretesting activities. They take full advantage of the logical positivists division of analysis falling into a domain of confirmation and a domain of discovery where the latter is so serendipitous that it does not require any research transparency. But the entire push for process tracing is to go past this false dichotomy and require quantitative scholars to demonstrate greater research transparency in their theorizing. DA-RT, if broadly defined, could inconvenience (or improve depending of where you stand) quantitative scholars just as much as qualitative ones.
2) I also think that it is important to place DA-RT in the broader context where the natural and social sciences have been experiencing what is widely referred to as a replication crisis. Christie Aschwand captured the underlying problems by observing that it is easier to get results than it is to get answers. I would highly recommend two articles by her posted on FiveThirtyEight: Science isn't Broke and http://fivethirtyeight.com/features/sci ... nt-broken/ and Failure is Moving Science Forward http://fivethirtyeight.com/features/fai ... e-forward/

Post Reply

Rudra Sil
University of Pennsylvania
Posts: 6
Joined: Thu Apr 07, 2016 4:50 pm

Re: CP newsletter: editorial or ideology?

PostSun Apr 17, 2016 8:17 pm

Marcus, just so we don't waste too much time talking about this when we get together to enjoy our beers at McShea's, let me just quickly point out a crucial distinction here: despite the benefits some of us attach to process-tracing to highlight causal links inferred from quantitative research, there is no APSA committee telling quantitative scholars to submit their articles to some sort of standardized "process-tracing" repository as part of the submission process, and there is no statement that journal editors are being asked to sign on to. Of course you are right to note that something akin to transparency norms HAS been present in qualitative research in a very long time. This is precisely why the pushback against DA-RT is about whether it is necessary to have a standardized policy authorized by APSA and signed by key journals, and whether the putative additional benefits of such a policy (cutting down on "LaCour-type" situations, which tend to involve quantitative work) would outweigh the skewed costs to qualitative scholars. Maintaining a diffuse ethos of transparency that is pragmatically adapted by different research communities makes a lot of sense, and even falls broadly within the "scientific frame." But, insisting on uniform or equivalent procedures for all forms of data and all types of interpretive or analytic strategies is a different ballgame.

Post Reply

Marcus Kreuzer
Villanova University
Posts: 26
Joined: Sat Apr 16, 2016 9:48 am

Re: CP newsletter: editorial or ideology?

PostSun Apr 17, 2016 9:55 pm

Dear Rudy,
Your and various other posts imply the DA-RT involves imposing quantitative transparency standards on qualitative research, a kind of KKV déja vue all over again. I can share this concerns with some of the requirements especially the data repository/access requirement.
But is there not another way of looking at this.
Couldn't you see a scenario where, if qualitative transparency standards are formalized and made more explicit (we are still quite a ways from this), quantitative scholars would see DA-RT just as much as an imposition on their research as qualitative scholars do right now. Quantitative scholars increasingly do mix-method research and, as you and Amel Ahmed have pointed out elsewhere, the case studies in this research frequently don't meet the transparency standards of qualitative scholars. Furthermore, analytical transparency also involves more detailed discussion regarding the theorizing and test construction stage whose quality once again requires detailed qualitative knowledge. Bottom line, quantitative research involves more than just data collection and data analysis. Those pre-testing activities are covered by the transparency standards widely used by qualitative scholars. So the inconvenience (or benefits) that DA-RT imposes will be symmetrical if qualitative scholars succeed in formalizing their transparency standards more fully.

Post Reply

Rudra Sil
University of Pennsylvania
Posts: 6
Joined: Thu Apr 07, 2016 4:50 pm

Re: CP newsletter: editorial or ideology?

PostMon Apr 18, 2016 2:28 pm

Hi Marcus - I know you are a backer of DA-RT, and in the end, we'll just have to agree to disagree. I am generally not in favor of regulating scientific research through standardized procedures to begin with, instead relying on our collective evaluations and common sense to separate good, insightful research from poor, shoddy work, no matter what the approach. But, this particular approach to DA-RT does, in fact, come straight out of KKV ... in fact, it can be traced directly back to King's 1995 essay, "Replication! Replication!" What has changed is simply the situation: the QMMR section, initially intended to signal pluralism, has essentially narrowed the scope for what is considered "good" or "useful" qualitative research, and the NSF has helped in this process by giving Colin Elman a grant to promote qualitative data repositories. In reality, the same issues that were present in KKV remain in place now, and it all ultimately revolves around how broadly inclusive one wishes to be. In the end, I am more inclusive than you, and seek to embrace and encourage many forms of qualitative research (including ethnography) where the very idea of finding a rough equivalent for quantitative data & code sharing would make very little sense. But, to the extent that there is some value in convincing people by showing one's sources of empirical evidence, we all have a natural interest in doing that by whatever means are available and to the extent that is permissible when human subjects are involved. The more we try to regulate this stuff, the more we skew incentives in favor of one type of qualitative work.

You mentioned my piece with Amel Ahmed - well one of the things we emphasized was that much qualitative research has no business being mixed with quantitative work given the quite different fundamental objectives and foundational premises. Thus, we argued that the world of mixed-method research, if it is turned into a standard practice, will basically accommodate only one kind of qualitative scholarship, that proceeding from a worldview that has more in common with that of quantitaive work (i.e. small-N, Mill-style comparison, or process-tracing done explicitly to match the variables that seem to be most strongly correlated in a statistical analysis). Here, we have a DA-RT program that not only assimilates qualitative work to quantitative work (at least enough so that the kind of data from field notes is seen as similar) on the grounds that more transparency is required for replicability. True, that can apply to certain forms of qualitative work. But, not all. More importantly, in a Lakatosian world, far more important than replication of a single study would be to develop theories backed by evidence that simply are more convincing and explain more - people in various qualitative traditions do that by challenging each other's arguments on the basis of different kinds of empirical data. I fail to see what will be added to this process that justifies making a junior scholar struggling to fund fieldwork at a small liberal arts college find the time and energy to scan all the field notes (while carefully treading around IRB requirements) into repositories. There will never be perfect transparency in this business, but efforts to push the envelope a little further carry more risks than benefits for the whole range of research communities that some of us wish to keep afloat within a more inclusive political science.

Post Reply

Robert Kaufman
Rutgers University
Posts: 1
Joined: Thu May 19, 2016 2:39 pm

Re: CP newsletter: editorial or ideology?

PostThu May 19, 2016 3:22 pm

From: Robert Kaufman, President of the Organized Section on Comparative Politics

I believe that the debate over DA-RT has been a very healthy one, which has been joined with professionalism and good will by participants on all sides of the issue. This spirit was very well reflected in the recent symposium in the Comparative Politics Newsletter, and I would like to weigh in with a few thoughts of my own. At the outset of the debate about DA-RT, I was inclined to be sympathetic to the effort to specify the norms of knowledge production in our discipline, and I believed (and still do) in the sincerity of promises to respect the diverse research traditions in comparative politics. Upon further reflection, however, I have come around to the view, expressed by several critics who participated in the CP symposium and in the contributions by the editors of CPS and World Politics, that the costs of the enterprise outweigh the potential benefits.

In principle, the DA-RT initiative pursues the laudable goal of making the empirical bases of knowledge claims clear and available to other researchers. Moreover, as several participants have noted, providing a public accounting of the data and methods should be an appropriate step for quantitative researchers, although there is some room for legitimate debate about the time frame in which original data sets should be made available to others. The problem comes with proposals to extend and adapt these norms to the various approaches to qualitative research. Efforts to do so may be both possible and desirable in some instances, but the professional disincentives, ethical limits, and practical costs – particularly to young researchers – can be overwhelming.

To me, the most critical stumbling block is the Journal Editors Statement (JETS). It is one thing for members of the Association to debate and even try to frame norms of data access and research transparency. It is quite another for journal editors to add another filter to the already demanding gauntlet of peer reviews and editorial judgments that a paper must survive in order to reach publication. Editors, to be sure, do serve as gatekeepers, as Deborah Yashar has noted, and they have an obligation to insist on rigorous reviews. But it is far from clear that adding new requirements to the review process – even ones that are flexible and respectful of alternative research traditions – will improve on existing practices. The great risk, of course, is that it will result in a more limited and less creative research environment.

This risk is especially high when there is so little agreement about the specific standards that might be generally applicable across the research communities within comparative politics. The evidentiary standards for arriving at and documenting inter-subjective claims need to emerge through inter-subjective processes. Such processes, to be sure, are shaped by the hierarchical structure of our profession – and probably should be. Still, for the time being, it seems prudent to rely on the norms of research transparency and data accessibility that emerge from within the diverse research communities themselves, rather than ask journal editors and their boards to elaborate additional hoops for authors to pass through.

Post Reply

Tim Buthe
HfP/Technical Univ of Munich & Duke University
Posts: 32
Joined: Fri Feb 26, 2016 11:39 pm

Re: CP newsletter: editorial or ideology?

PostSun May 22, 2016 10:59 pm

Thank you to all the contributors on the thread. Allow me to re-direct the discussion a bit to solicit input from the many scholars of comparative politics who clearly have been following this thread, in order to solicit your input regarding the key issues with which the steering committee will need to grapple in the next few days when we try to decide how to define and delineate the 12-15 working groups that are supposed to be the core of stage 2 of the Qualitative Transparency Deliberations. PLEASE generously share your thoughts about the following question:

What are the aspects of research transparency for which you would most want to see an opportunity for differentiated deliberations during stage 2 of the QTD? Please don't hesitate to post even if you might only mention your top 2-3, and please be as specific as you can be. Adding a sentence or two to explain why will make your post even more valuable but such elaboration is by no means required.

Thanks!

Post Reply