Substantive Dimensions of the Deliberations

Forum rules

We encourage contributors to the Discussion Board to publicly identify by registering and logging in prior to posting. However, if you prefer, you may post anonymously (i.e. without having your post be attributed to you) by posting without logging in. Anonymous posts will display only after a delay to allow for administrator review. Contributors agree to the QTD Terms of Use.

Instructions
To participate, you may either post a contribution to an existing discussion by selecting the thread for that topic (and then click on "Post Reply") or start a new thread by clicking on "New Topic" below.

The transition to Stage 2 of the deliberations is currently underway but will take some time to complete. In the meantime, we very much welcome additional contributions to the existing threads in this forum.

For instructions on how to follow a discussion thread by email, click here.

Margaret Keck
Johns Hopkins University
Posts: 2
Joined: Fri Apr 08, 2016 8:22 am

No place for my work in this debate

PostFri Apr 08, 2016 9:08 am

You know, I have tried to write something about this controversy. But I find I really can't. It simply doesn't seem relevant to the work I have done over the last 35 years. I use footnotes extensively, with the idea of making as many of my sources as possible transparent to my readers, but many of them are references to unpublished documents that only tangentially relate to what I am discussing, in a language that most readers don't know. To analyze documents and interviews, I rely not just on language skills but on knowledge accumulated from 35 years of work in a region, on knowing the backstories of many of the people I am talking to, and on lengthy discussions over cold beers with people who have their own opinions about what I am studying. The DA-RT debate just doesn't compute in relation to the kinds of research I have always done, and it really doesn't seem worth the effort to try to make it fit. I'd much rather just publish elsewhere. I do share some of my research materials with other scholars. But I am perfectly comfortable with the idea that someone who does not speak the languages I speak, does not know the history I know, and does not have the kinds of social and intellectual networks I have been lucky enough to build up would not be able from my notes or appropriately archivable interviews to come to the same conclusions I have come to. Too bad. There are no shortcuts - one has to put in the time and do the work. That may not be a fashionable view, but I will stick with it.

Post Reply

Alan Jacobs
University of British Columbia
Posts: 38
Joined: Fri Feb 26, 2016 9:59 pm

[Steering Committee] No place for my work in this debate

PostWed Apr 13, 2016 2:17 pm

Thank you very much for your comments, Margaret.

You write in your post of a disconnect between the DA-RT framework and your work. When I read this, I immediately thought about how transparent I find your work. One particular way in which your work is unusually open – I am thinking here of your recent book Practical Authority, with Rebecca Abers – is in how explicitly you discuss the methodological difficulties that you confronted, the failure of initial hypotheses to find support in the empirical record, and the way in which the research strategy and the argument emerged as the investigation unfolded (including, in an extensive “Methodological Narrative” appendix). We often talk about case-study research as being rich with opportunities for inductive discovery and as a process that typically involves a back-and-forth movement between theoretical development and empirical analysis (as, of course, does much quantitative research). But in actual writeups of research findings, we do not often see the kind of transparency about that process that your book displays. It is also not a form of openness that has featured much in recent transparency discussions.

Would you be willing to comment on how and why you and Abers decided to be pursue this type of transparency in the book?

Post Reply

Guest

Re: No place for my work in this debate

PostWed Apr 13, 2016 3:13 pm

I wholeheartedly agree with the points made by Dr. Keck above, as I had the same reaction that my work does not find a place in this debate. When I learned of the DA-RT framework, my first thought was that this will further divide the field among journals and the US-Europe divide in international studies more broadly.

Beyond what I have seen posted already, I hope that the APSA Section for Qualitative and Multi-Method Research will pay particular attention to the problems associated with archival research in the US and overseas, if it has not already. In addition, I hope the section will consider the issues associated with archival materials in foreign languages (as Dr. Keck points out). Complying with disciplinary citation standards in political science has already been a problem with archival evidence. The DA-RT framework makes it even more of one in several dimensions.

Post Reply

Margaret Keck
Johns Hopkins University
Posts: 2
Joined: Fri Apr 08, 2016 8:22 am

Re: No place for my work in this debate

PostFri Apr 15, 2016 8:02 pm

Thank you, Alan Jacobs, for your kind words about Practical Authority. In that book, my co-author Rebecca Abers and I thought it was important to walk the readers through the research process as well as the conclusions. But by "walk them through," I do not mean make available to them every document or interview or set of fieldnotes or set of meeting minutes that went into that research (done over more than ten years and involving a lot of people other than the authors, most of whom were Brazilian). I do mean that it seemed necessary to explain HOW we did the research, what we thought we would find at the beginning, how we discovered that our initial expectations were not borne out, and what we did then. We did make survey data and research reports of the research done during the process available to other scholars, practitioners, and the public, on web sites and in presentations at meetings in Brazil. But the argument we eventually made in the book emerged from a frequently difficult struggle to make sense of theory and evidence at the same time. Making that process as transparent as possible (without boring our readers to death) opens the door to others who might challenge the decisions we made, or the interpretations we put on what we studied. The resulting debate, one would hope, could inspire others to do more research and formulate more interesting arguments. That sort of "transparency" is something I support wholeheartedly.

One further comment: There's a certain irony in the DA-RT discussion on archiving, which is focused on making research materials available to other social scientists through specialized sites, pre-sifted and translated and ready to be re-analyzed. I am currently engaged in finding archival homes for thousands and thousands of documents and other ephemera collected in the course of multiple research projects. These documents will be archived in libraries, where scholars and other people who are searching for materials in their subject areas can find them - in collections known to contain materials of this kind or via WorldCat. Some will be scanned and made available online. These will be original materials, in their original language, not translations. Although organizing papers for archiving is a lot of work, it's really useful, as anyone who has ever worked in a well-organized archive knows.

Post Reply

Guest

Re: No place for my work in this debate

PostTue Apr 19, 2016 1:49 pm

I would like to underscore and elaborate a bit on what Mimi Keck says:

For many of us, especially those who have been writing for years, these debates seem so disconnected from the realities of research. And we have plenty of other (inter-) disciplinary outlets if we chose to opt out of dysfunctional political science. Unfortunately, the price will be paid by emerging scholars, who will waste a lot of time and energy trying to digitize and archive everything in their research.

For those of us who work on topics and in environments where the focus is controversial (academically and/or politically) many sorts of "transparency" are either obvious (= must position oneself within theoretical debates which are equally ideological as academic) or dangerous (= immediate risks to interlocutors or possibly in the indeterminate future, because no one can accurately predict a coup or other type of policing). In my most recent book, I spent a few pages explicitly outlining these choices; it needed no more or less.

For those who have compiled resources which are truly a treasure trove, there may be better ways to archive them (as Mimi is doing). Even mundane resources can be valuable, albeit not worth institutional resources to scan/preserve. For instance, recently, I have been finding individual emerging scholars working on topics related to my earlier research who have been eager to receive packets of materials that I accumulated over the years but will never use again. I've hung onto them for decades on the off-chance someone might challenge my inferences--honestly, I'm not making that up--which NO ONE EVER did. It's time to triage, not post to a repository.

Post Reply

ReginaBateson

Re: No place for my work in this debate

PostTue Apr 19, 2016 4:06 pm

I would like to agree with Prof. Keck's most recent comment -- when researchers gather truly original primary-source documents that might be of use to others, why not deposit them with an established archive? There, they will be appropriately preserved and catalogued, so that scholars from a variety of disciplines can access them.

To the extent that DART and JETS is about creating public goods to be used by other scholars, I feel that qualitative researchers in political science are largely being asked to duplicate work already done by oral historians, archivists, and librarians. Either our notes and ephemera are "scratch work" not worth disseminating because they would make little sense to others, or they are valuable primary sources -- in which case it would seem to make more sense to archive our papers with established institutions.

I am all for thorough, detailed citations to primary sources. But especially when using documents that are already in a well-organized archive--perhaps even a digital archive!--it seems redundant (and potentially a violation of the archive's policies) to ask us to scan or photograph those documents (in full? in part?) and re-post them in what would be, essentially, yet another archive just for political scientists. Such a standard would require considerable unnecessary work by qualitative scholars, while also forcing us to work well outside our areas of expertise. Most of us are not trained as archivists or oral historians. Why pretend to be?

Regina Bateson
MIT

Post Reply

Guest

Re: No place for my work in this debate

PostWed Apr 20, 2016 6:04 am

[quote="mkeck"]But by "walk them through," I do not mean make available to them every document or interview or set of fieldnotes or set of meeting minutes that went into that research (done over more than ten years and involving a lot of people other than the authors, most of whom were Brazilian). I do mean that it seemed necessary to explain HOW we did the research, what we thought we would find at the beginning, how we discovered that our initial expectations were not borne out, and what we did then. We did make survey data and research reports of the research done during the process available to other scholars, practitioners, and the public, on web sites and in presentations at meetings in Brazil. But the argument we eventually made in the book emerged from a frequently difficult struggle to make sense of theory and evidence at the same time. Making that process as transparent as possible (without boring our readers to death) opens the door to others who might challenge the decisions we made, or the interpretations we put on what we studied. The resulting debate, one would hope, could inspire others to do more research and formulate more interesting arguments. That sort of "transparency" is something I support wholeheartedly.[/quote]

This seems to me what we could propose to strive for in certain subfields, instead of the blunt fact checking approach of Da-art.

Post Reply

Sarah Parkinson
Johns Hopkins University
Posts: 12
Joined: Mon Apr 18, 2016 4:32 pm

Re: No place for my work in this debate

PostMon Apr 25, 2016 3:42 pm

I wonder what more people think of Mimi Keck's idea of "walking" readers through research; as she and others noted, it underscores a very different approach to transparency than the definition to which many subscribe. It also relates to some of Joe Soss's comments at last year's APSA, which is that many researchers engaging in intensive fieldwork frequently want more space in publications to discuss the stories behind and processes/decisions undergirding their research.

Is what Keck suggests fully possible outside of books? Is it/should it be incentivized in article-length scholarship? What are the implications for scholars at different points in their careers?

Post Reply

Sean Yom
Temple University
Posts: 3
Joined: Wed Apr 20, 2016 10:43 pm

Re: No place for my work in this debate

PostTue Apr 26, 2016 3:35 pm

separkinson wrote:I wonder what more people think of Mimi Keck's idea of "walking" readers through research; as she and others noted, it underscores a very different approach to transparency than the definition to which many subscribe. It also relates to some of Joe Soss's comments at last year's APSA, which is that many researchers engaging in intensive fieldwork frequently want more space in publications to discuss the stories behind and processes/decisions undergirding their research.

Is what Keck suggests fully possible outside of books? Is it/should it be incentivized in article-length scholarship? What are the implications for scholars at different points in their careers?


I would guess, Sarah, that some scholars would not want to be completely transparent (or let's call it honest) in divulging to their readers, and students, what precise sequence of steps they undertook (and the mistakes they made) in order to end up with the publishable manuscript in article or book form. We want readers to be interested in the sausage, not how it's made. DA-RT is like the FDA ordering a manufacturer to tell consumers where it is getting the ingredients for the sausage, but that's it. Consumers still don't know, precisely, how it's being made in that opaque factory behind locked doors.

On the other hand, many of us do want to be more transparent, but publishing norms in article or book form prevent us from disclosing all of these messy mental steps and random empirical mishaps that occurred along the way of crafting our scholarship.

Post Reply

Tim Buthe
HfP/Technical Univ of Munich & Duke University
Posts: 32
Joined: Fri Feb 26, 2016 11:39 pm

Re: No place for my work in this debate

PostThu Apr 28, 2016 4:01 am

Thank you for your post, Sean. The question of how much our readers are interested in the "sausage-making," i.e., the research process rather than just the research product (and how much we as scholarly authors are interested in conveying information about that process) is certainly an interesting one.

It may be that many readers are only interested in the sausage and not the "sausage-making," i.e., information about the research process. And for popular books or intro-level textbooks focusing on the key research findings may be fine. But it seems to me that we as scholars--at least those of us who embrace the "scientific" ambitions of Political Science (appropriately broadly understood)--must be interested both in learning about other researchers' process and sharing information about our own. Such information is essential for learning from each other (including the mistakes). And it is impossible to really interpret and assess the research findings without knowing how they were obtained.

An important question then is: How can we provide the most important and most useful information without unduly high costs to readability? Marc Trachtenberg, in his article in the QMMR symposium on Transparency in Qualitative and Multi-Method Research (http://ssrn.com/abstract=2652097), not only identifies this tension between increasing transparency and writing for a broad audience; he also makes some suggestions for overcoming/reducing that tension, yet much more can and should surely be said about that topic.

Post Reply

Sarah Parkinson
Johns Hopkins University
Posts: 12
Joined: Mon Apr 18, 2016 4:32 pm

Re: No place for my work in this debate

PostFri Apr 29, 2016 2:13 pm

Thanks, Sean, Tim, and other posters; this is all very productive! One of the things that has surfaced repeatedly in these debates is the assumption that deciding what to make, being able to access ingredients, the "sausage making" process, and the culinary product are necessarily discrete "steps" that can be divorced from each other. For many research traditions--especially critical and interpretive ones (as Kathy Cramer and Tim Pachirat discuss in their QMMR essays)--this sort of distinction/separation simply doesn't hold.

If we're going to stick with the food metaphor, I wonder if talking about the proverbial distinctions between: 1.) Sharing exact ingredients; 2.) Providing a general "recipe" (the kinds of things that we do to gain access, the general and likely context variant types of "ingredients" used, general strategies of generating evidence, and ways of engaging with data), and; 3.) Sharing knowledge about what it takes to "make sausage" (meat grinder) versus "bake a cake" (mixer) would be useful.

Lunchtime.

Post Reply

James Heilman
University of Massachusetts-Amherst
Posts: 3
Joined: Tue Apr 26, 2016 8:33 am

Re: No place for my work in this debate

PostSat Apr 30, 2016 11:27 am

Two thoughts on this thread:

1) I like the idea of walking the reader through the research process. I wonder if our in-the-field practices will change so that we can more easily make our research process transparent for articles than we currently do. Training for graduate students would have to include some practices that field researchers could make use of so that when they start writing they are able to easily take field notes and make them transparent for an article or appendix to an article.

2) Some people in this thread have mentioned that one option is to not publish in journals that sign onto DART. As a graduate student I am concerned about how this could effect how others evaluate my work. If many of the top journals implement transparency requirements that are extra-onerous or impossible to fulfill for some types of qualitative research, then I might have to publish in journals that could be well recognized by peers in my sub-discipline but might not be as well respected by the political science discipline at large. Therefore, this could put me at a competitive disadvantage when I am on the job market. Since hiring committees are not discipline specific, I strive to publish in top-tier journals so that when people on a hiring committee debate my CV there cannot be an argument about the quality of the journals I have published in. Furthermore, once I am a faculty member working towards tenure, I will continue to want to publish in widely respected journals in order to make my tenure review as smooth as possible. I recognize that as new transparency standards are enforced the reputations of journals will change in ways we cannot predict. I hope that a sufficient number of journals that are respected throughout the discipline and publish a wide range of research topics and methodologies remain and that the discipline does not lose they type of journals.

Post Reply

Joe White
Case Western Reserve University, Political Science
Posts: 1
Joined: Thu May 19, 2016 2:22 pm

Re: No place for my work in this debate

PostThu May 19, 2016 10:59 pm

My name is Joe White and I do U.S. politics and policy and some comparative politics and policy - especially health care - at Case Western. Much of my work involves secondary research so I just cite the sources. I don't have to worry about knowledge of the language. But I have done a bunch of original qualitative research about Congress and especially federal budgeting. So in the course of that I have conducted somewhere over 300 confidential interviews over the course of the past quarter century.

Confidential interviews are gained on the basis of trust relationships. I promise full confidentiality. I am referred to people by those whom I have interviewed before. I go back to people whom I have interviewed before, because I want to understand change and can ask them to compare situations. I draw my own conclusions about who can be trusted in what ways, and then I have to provide information about my sources that is clear enough to provide some sense of their ability to speak to an issue and yet vague enough so that they cannot be precisely identified. They can have very good reasons not to want to be identified.

Here is an example: I did some work on budgeting during the GWBush years, and in particular decision-making within the executive branch. That involved interviewing within and about OMB. In what I wrote, I made claims about how the administration made decisions about budget totals. One source I called something like, "a highly ranked OMB political official" - you know the usual locutions.

The most reasonable true identity for such a person would be the OMB Director. The good news is, there were four of them during the administration. The bad news is, one is a sitting U.S. Senator who might get nominated for Vice President; a second is a long-serving former Governor who was frequently mentioned as a potential candidate for president and still might be some day; a third is out of politics for now but is running a significant interest group and might well go into a GOP administration even if he doesn't run for office again; and the fourth probably will never be a candidate but could be, oh, nominated for Secretary of the Treasury or go back to being White House Chief of Staff. If my source happened to be an OMB Director, am I really supposed to share my interview, with identifying information? But if I don't give the identifying information, how does anyone else know how well-positioned the person was to describe what I quote him describing?

I just don't see how DA-RT could possibly work reasonably with elite interviewing. I recognize the problem with asking people to trust me. Heck, I've constantly wished I could identify my sources - for example, when a politician makes an argument in public that s/he has directly contradicted in an interview. It bothers me to be sitting on more than a thousand pages of interviews that others don't get to see (well, once or twice I have shared material, with many strong promises of confidentiality, with people I think I know well enough to trust; but even that makes me nervous). But I talk to staff people who are not free to be public. I talk to bureaucrats who are in difficult situations balancing demands from their political bosses and Congress. I talk to advocates about how they pursue their objectives, which might not be quite the same as their public self-presentations I ask people about how they do their jobs, and ask for examples, and the examples would identify them or at least greatly narrow the possibilities.

Yes, you could conclude that I do "journalism" which is not "political science." But if the discipline forces its members to meet strict standards of sharing and transparency, then only journalists will be in a position to do some of the most basic investigations of political behavior. That's nuts.

Post Reply