Limits to Transparency

Forum rules

We encourage contributors to the Discussion Board to publicly identify by registering and logging in prior to posting. However, if you prefer, you may post anonymously (i.e. without having your post be attributed to you) by posting without logging in. Anonymous posts will display only after a delay to allow for administrator review. Contributors agree to the QTD Terms of Use.

Instructions
To participate, you may either post a contribution to an existing discussion by selecting the thread for that topic (and then click on "Post Reply") or start a new thread by clicking on "New Topic" below.

For instructions on how to follow a discussion thread by email, click here.

You are NOT logged in.

To have your post appear linked to your username, please log in first. If you have just registered, please wait to receive an activation email from qtdfora.

Post a reply


In an effort to prevent automatic submissions, we require that you confirm you’re human using Google’s reCAPTCHA service.

BBCode is OFF
Smilies are OFF

Topic review
   

Expand view Topic review: Limits to Transparency

Re: Limits to Transparency

Post by Bill Kelleher » Mon Dec 26, 2016 8:43 pm

nrsmith.ccny wrote:From the moderators: What can ethnographers generally not share about their research process and why can they not share those things?


I would like to respectfully point out the positivistic bias in the way this question has been phraised. The question presupposes a condition of objectivity when its focus is on "things" ethnographers can "not share." Thus put, attention is distracted from the deepest, most meaningful conflict between positivists and interpretive political science. The word "things" calls notes to mind, but not the researcher herself or himself.

The researcher is always a unique person who can't share who he or she is with another person who comes along. Kathy Cramer's fabulous comments on this thread add up to one point, no hay dos! That is, there is only one Professor Cramer -- so there is nothing that can be shared. As Ms. Cramer points out, even if they had her notes in hand, nobody would understand their meaning like she does. As she says, nobody can replicate her interviews, any more that they could replicate Professor Parkinson's work, or Professor Majic's.

While Manbridge's point that privacy must be respected is certainly important, the uniqueness of the person doing the research ought to be the defining principle of interpretive political science. Upon that basis, the notion that replication is somehow a necessary element of systematic and responsible professional research ought to be roundly rejected. The problem for us is how to formulate standards for assessing the validity and importance of research while also celebrating, and never denying, the uniqueness of the researcher.

I have discussed the vital place of uniqueness for interpretive political science in several posts on this site, but especially in my proposed Community Transparency Statement for Interpretive Political Science
viewtopic.php?f=21&t=195#p800

Re: Limits to Transparency

Post by separkinson » Fri Dec 02, 2016 3:01 pm

I think the answer to this question is: "it often depends," that is, it's entirely contingent on the project and I'm generally willing to trust the researcher's discretion unless their writing sets off alarm bells. If a researcher tells me that revealing a particular aspect of their work would be problematic, I'll believe them as long as it's not something basic. That is, I expect people to share their language abilities, time in the field, and some geographic description of where they lived, e.g. "a small town in the American Rust Belt." I also don't necessarily see not sharing everything as coterminus with "limited to transparency," though many people might frame it that way.

In general, there are both ethical, epistemological, and often legal reasons not to share ethnographic field notes or interview transcripts. I haven't seen a lot of disagreement on that front.

But to give a few examples of what I would not share from my own work:

I don't generally name the refugee camp where I lived in print (my interlocutors obviously know). Part of this move is to provide plausible deniability to those to whom I was closest if elites officers became angry with what I wrote and came asking who had helped me with my project. I also don't want it to turn into a place "where researchers go" because it was possible for me to live there; this has happened to at least one refugee camp in Lebanon and produced anguish among many who live there as a result. If someone winds up there as a result of their own research path, that's a function of their research, not them trying "the easiest path."

I often conceal precise dates and locations of interviews and excerpted field notes because someone (e.g. superior officers, state intel) might well have noted that I visited an office on an exact day. I usually use month or season instead and sometimes list the city rather than the camp or neighborhood.

I often interview people who are the only survivors of their unit/division or the only person living to have occupied a particular position. Naming that position would automatically "out" them, so I often render it more vague.

In writing, I don't reveal all of the security precautions I took when on the ground because then they would be useless when I return. I'll happily talk about them in person with my graduate students and with trusted colleagues.

Re: Limits to Transparency

Post by jane mansbridge » Wed Oct 26, 2016 11:30 am

I find it hard to specify in advance what researchers should and should not share 1) with the reader of the eventual article or book and 2) with the general public by depositing field notes and interviews in a public depository. One needs to share with the reader those moments observed or words from an interview that will give the reader an insight into an issue that the reader may not have had before. But one needs to be at the same time very careful ("paranoid" was the word aptly used in one of the QTD comments) about exposing those interviewed or observed to harm.
It may be possible to codify the considerations involved in two sometimes contradictory imperatives required for publishing a useful piece of research that appropriately protects participants other than the author. Those imperatives are not the same for depositories. My own conclusion in the past has been (and at the moment still is) that the imperative of protecting the people who have invited you into their confidence or their environment is the highest imperative. I hope that this conversation may help illuminate the appropriate considerations on all sides. It may be that, in the constantly changing environments in which we find ourselves, the best criterion will still be the researcher's independent judgment of what is appropriate in the particular situation that the researcher knows best.

Re: Limits to Transparency

Post by kathycramer » Thu Oct 20, 2016 5:18 pm

nrsmith.ccny wrote:From the moderators: What can ethnographers generally not share about their research process and why can they not share those things?


I am pasting in below a piece I wrote for the QMMR Symposium on Transparency in Qualitative and Multi-Method Research, in the Spring 2015 issue, under the title, "Transparent Explanations, Yes. Public Transcripts and Fieldnotes, No: Ethnographic Research on Public Opinion." I hope it is a useful contribution to this important discussion.



I am a scholar of public opinion. My main interest is in examining how people understand, or interpret, politics. For that reason, much of my work involves listening to people talk with others with whom they normally spend time. When I listen to the way they sort out issues together, I am able to observe what they perceive to be important, the narratives they use to understand their world, the identities that are central to these understandings, and other important ingredients of public opinion.

My work is therefore primarily qualitative, and usually interpretivist. By interpretivist, I mean that I am trying to capture how people perceive or attribute meaning to their worlds. I treat that effort as a necessary part of trying to understand why they express the opinions that they do.

Across the course of my career, transparency has been a professional necessity. My methods are rather unusual in the field of public opinion research, so the burden is on me to teach my readers and my reviewers what I am doing, why I am doing it, and how my work should be judged. Usually, public opinion scholars focus on individuals’ preferences and how to predict them, not on the process of understanding. In addition, we tend to be well versed in the strengths and weaknesses of polling data, but basically unfamiliar with conversational data.
Put another way, reviewers are likely to dive into my papers looking for the dependent variable and the strength of evidence that my results can be generalized to a national population. But my papers usually do not provide information on either of these things. Unless I explain why my work has different qualities to be judged, the typical reviewer will quickly tune out and give the paper a resounding reject after the first few pages.

So the first thing I have had to be transparent about is the fact that much of my work is not attempting to predict preferences. My work typically does not describe how a set of variables co-vary with one another to bring about particular values on a dependent variable. Indeed, I’m not usually talking about causality. These characteristics are just not what scholars typically come across in political science public opinion research. I have had to go out of my way to explain that my work uses particular cases to help explain in detail the process of a group of people making sense of politics. I have had to be up front about the fact that my goal is to help us understand how it is that certain preferences are made obvious and appropriate when objective indicators about a person’s life would suggest otherwise.

For example, in a piece I published in the APSR in 2012,1 reviewers helpfully pointed out that I had to bluntly state that my study used particular cases to study a broader question. In short, that article reported the results of a study in which I invited myself into conversations among groups of people meeting in gathering places like gas stations and cafés in com- munities throughout Wisconsin, especially small towns and rural places, so that I could better understand how group consciousness might lead people to support limited government when their objective interests might suggest otherwise. I had to take several paragraphs to contrast what I was doing against the more familiar positivist approaches. I wrote:[2]

"My purpose in investigating what people say in the groups they normally inhabit in a particular set of communities within one state is to better explain how the perspectives people use to interpret the world lead them to see certain stances as natural and right for people like themselves (Soss 2006, 316). It is motivated by the interpretivist goal of providing a “coherent account of [individuals’] under-standings as a prerequisite for adequate explanation” (Soss 2006, 319; see also Adcock 2003). In other words, to explain why people express the opinions that they do, we need to examine and describe how they perceive the world. In this article I explain the contours of the rural consciousness I observed and then specify its particularity by contrasting it with conversations among urban and suburban groups. That is, this is a constitutive analysis (an examination of what this thing, rural consciousness, consists of and how it works) versus a causal analysis (e.g., an examination of whether living in a rural place predicts rural consciousness—McCann 1996; Taylor 1971; Wendt 1998). The point is not to argue that we see consciousness in rural areas but not in other places, nor to estimate how often it appears among rural residents, nor to describe what a population of people thinks. Instead, the purpose here is to examine what this particular rural consciousness is and what it does: how it helps to organize and integrate considerations of the distribution of resources, decision- making authority, and values into a coherent narrative that people use to make sense of the world. This is not a study of Wisconsin; it is a study of political understanding and group consciousness that is conducted in Wisconsin (Geertz 1973, 22).

"To clarify the stakes, contributions, and implications of this study, allow me to contrast it with positivist approaches. I examine here how people weave together place and class identities and their orientations to government and how they use the resulting perspectives to think about politics. A positivist study of this topic might measure identities and orientations to government, and then include them as independent variables in a multivariate analysis in which the dependent variable is a policy or candidate preference. Such an approach is problematic in this case in the following ways. The positivist model specification assumes that values on one independent variable move independent of the other. Or if using an interaction term, it assumes that people with particular combinations of these terms exhibit a significantly different level of the dependent variable. However, the object of study, or my dependent variable in positivist terms, is not the position on an attitude scale. It is instead the perspectives that people use to arrive at that position. My object is not to understand the independent effects of identities and attitudes such as trust, or how people with different combinations of these compare to others, but to understand how people themselves combine them—how they constitute perceptions of themselves and use these to make sense of politics."

I include this excerpt to underscore that transparency in the sense of explaining in detail my data collection and analysis procedures, as well as my epistemological approach, has been a professional necessity for me. Without providing extensive detail on my research approach, many readers and reviewers would not recognize the value of my work. Indeed, on one occasion in which I did not take the space to provide such detail, one exceptionally uncharitable reviewer wrote, “I found this chapter very disappointing. This perhaps reflects my bias as a researcher who does formal models and quantitative analysis of data. Briefly, I, too, can talk to taxi drivers or my mother’s bridge crowd.” There were just 3 additional sentences to this person’s review.

Transparency has also been important for me for another reason. In interpretive work, we make evidence more valuable and useful the more context we provide. As we describe and examine the meaning that people are making out of their lives, we better equip our readers to understand and judge our claims the more information we provide about what leads to our interpretations. Positivist work has the burden of providing evidence that a given sample is representative of a broader population. Interpretivists must provide enough context that our interpretations are “embedded in, rather than abstracted from, the settings of the actors studied.”[3] Transparency is an integral part of the presentation of our results. For example, in a forthcoming book based on the study I describe above,[4] I explain the physical nature of the settings in which I met with rural residents, the lush green fields and blue skies of the com- munities in which they resided, what they wore, and how they responded to me as a smiley city girl arriving to their gas stations, etc., in my Volkswagen Jetta to convey in part the complexity of the perspectives with which the people I spent time with viewed the world. They were deeply proud of their com- munities and their groups of regulars, and at the same time resentful of the economic situations in which they found themselves.

In those respects, I value transparency. But in another respect, I do not. In particular, I do not consider making my field notes publicly available to be a professional duty or necessity. I am troubled by the recent push in our discipline to make available the transcripts of the conversations I observe and my fieldnotes about them to anyone and what it means for my future research. I have three specific concerns.
First, asking me to make my data publicly available assumes that any scholar could use it in the form it exists on my computer. That is, the assumption is that if I provide the transcripts of the conversations, and a key providing an explanation of important characteristics of each speaker, any scholar would be able to treat those words as representations of the conversation. I am just not confident that is possible. My fieldnotes are pretty detailed, but I do not note the tone or inflection of every sentence. I do not include every detail of the town or the venue in which I visit with people. I record details about what people are wearing, how the coffee tasted, and the pitch of some folks’ voices, but when I re-read a transcript, a thousand images, sounds, and smells enter my mind to round out the impression of the messages people were conveying to me and to each other. For this reason, I do not send assistants to gather data for me. I need to be there. I need to experience the conversation in all its fullness to get the best possible picture of where people are coming from.

Could a scholar still get something useful from my transcripts? Perhaps—the frequency of the use of some words, the general frames in which certain issues are talked about— all of that would be available in my transcripts. But my sense is that this push for transparency is not about broadening the pool of available data for people. It is about the assumption that making data publicly available will enable replication and the ability to more easily assess the validity of an argument. It is driven by the assumptions that the best social science is social science that is replicable and consists of conclusions that another scholar, given the same data, would arrive at independently. But I do not agree that the transcripts that I or my transcribers have typed out, and the accompanying memos that I have written would allow replication, nor would I expect another scholar to necessarily reach the same conclusions based on re-reading my transcripts.

Let me put it another way. The field of political behavior is pretty well convinced that Americans are idiots when it comes to politics. When you read the transcripts of the conversations I observe, it is not difficult to come away with the conclusion that those exchanges support that general conclusion. The words people have said, typed out on a page, do seem ignorant according to conventional standards. However, when the conversational data is combined with the many intangible things of an interaction—the manner in which people treat each other, the physical conditions in which they live—their words take on meaning and reasonableness that is not evident from the transcription of their words alone.

Let me reiterate that the purpose of my work is not to estimate coefficients. There is not a singular way to summarize the manner in which variables relate to one another in my data. Instead, what I do is try to characterize in as rich a manner possible how people are creating contexts of meaning together. Should I enable other scholars to look at my data to see if they would reach the same conclusion? I do not think it is possible to remove me from the analysis. In my most recent project, conversations about the University of Wisconsin-Madison became a key way for me to examine attitudes about government and public education. I work at that institution. I have a thick Wisconsin accent. My presence became a part of the data. Another scholar would not have all of the relevant data needed for replication unless he or she is me.


I am against this push for making transcripts and fieldnotes publicly available for another reason. Ethically, I would have a very difficult time inviting myself into conversations with people if I knew that not only would I be poring over their words in detail time and time again, but that an indeterminate number of other scholars would be doing so as well, in perpetuity. How do we justify that kind of invasion? You might say that survey research does this all the time—survey respondents are giving the permission for an indeterminate number of people to analyze their opinions for many years to come. But we tell respondents their views will be analyzed in the aggregate. Also, collecting conversational data, in person, in the spaces that people normally inhabit, with the people they choose of their own volition to do so, is not the same as collecting responses in a public opinion poll, even if that poll is conducted in person. When you are an interviewer for a public opinion poll, you are trained to be civil, but basically nondescript—as inter- changeable with other interviewers as possible. That is just about the opposite of the approach needed in the kind of research I conduct. I have to walk into a group as my authentic self when I ask if I can join their coffee klatch and take up some other regular’s barstool or chair. I am able to do the kind of work I do because I am willing to put myself out there and connect with people on a human level. And I am able to gather the data that I do because I can tell people verbally and through my behavior that what they see is what they get. If the people I studied knew that in fact they were not just connecting with me, but with thousands of anonymous others, I would feel like a phony, and frankly would not be able to justify doing this work.

My main worry with this push for transparency is that it is shooting our profession in the foot. I am concerned in particular about the push for replicability. Does the future of political science rest on replicability? I have a hard time seeing that. Perhaps because I work at the University of Wisconsin-Madison, in which tenure and public higher education are currently under fire, when I think about the future of our profession, I think about the future of higher education in general. It seems to me that our profession is more likely to persist if we more consciously consider our relevance to the public and how our institutions might better connect with the public. I am not saying that we should relax scientific standards in order to appease public will. I am saying that we should recognize as a discipline that there are multiple ways of understanding political phenomena and that some ways of doing so put us in direct connection with the public and would be endangered by demanding that we post our transcripts online. Why should we endanger forms of data collection that put us in the role of ambassadors of the universities and colleges at which we work, that put us in the role of listening to human beings beyond our campus boundaries? It is not impossible to do this work while making the transcripts and field notes publicly available, but it makes it much less likely that any of us will pursue it.

I do not think outlawing fieldwork or ethnography is the point of the data access initiative, but I fear it would be an unintended consequence resulting from a lack of understanding of interpretive work—the kind of misunderstanding that leads some political scientists to believe that all we are up to is talking with taxi drivers.

My interpretive work seeks to complement and be in dialogue with positivist studies of public opinion and political behavior. Its purpose is to illuminate the meaning people give to their worlds so that we can better understand the political preferences and actions that result. Understanding public opinion requires listening to the public. Transparency in this work is essential to make my methods clear to other scholars in my field who typically are unfamiliar with this approach, so that they can understand and judge my arguments. But I do not think that mandated transparency should extend to providing my transcripts and fieldnotes. My transcripts and fieldnotes are not raw data. The raw data exist in the act of spending time with and listening to people. That cannot be archived. The expectation for interpretive work should be that scholars thoroughly communicate their methods of data collection and analysis and provide rich contextual detail, including substantial quoting of the dialogue observed. There are many excellent models of such transparency in interpretive ethnographic work already in political science, which we can all aspire to replicate.[5]

1 Cramer Walsh 2012.
2 Cramer Walsh 2012, 518.
3 Schwartz-Shea and Yanow 2012, 47.
4 Cramer 2016.
5 See, e.g., Pachirat 2011; Simmons 2016; Soss 2000.
References
Adcock, Robert. 2003. “What Might It Mean to Be an ‘Interpret- ivist’?” Qualitative Methods: Newsletter of the American Political Science Association Organized Section vol. 1, no. 2: 16–18.
Cramer, Katherine. 2016. The Politics of Resentment: Rural Consciousness in Wisconsin and the Rise of Scott Walker. Chicago: University of Chicago Press.
Cramer Walsh, Katherine. 2012. “Putting Inequality in Its Place: Rural Consciousness and the Power of Perspective.” American Political Science Review vol. 106, no. 3: 517–532.
Geertz, Clifford. 1973. “Thick Description: Toward an Interpretive Theory of Culture.” In The Interpretation of Cultures: Selected Essays. New York: Basic Books, 3–30.
McCann, Michael. 1996. “Causal versus Constitutive Explanations (or, On the Difficulty of Being So Positive . . .).” Law and Social Inquiry vol. 21, no. 2: 457–482.
Pachirat, Timothy. 2011. Every Twelve Seconds: Industrialized Slaughter and the Politics of Sight. New Haven: Yale University Press.
Schwartz-Shea, Peregrine, and Dvora Yanow. 2012. Interpretive Research Design: Concepts and Processes. New York: Routledge.
Simmons, Erica. 2016. The Content of their Claims: Market Reforms, Subsistence Threats, and Protest in Latin America. New York: Cambridge University Press.

Soss, Joe. 2000. Unwanted Claims: The Politics of Participation in the U.S. Welfare System. Ann Arbor: University of Michigan Press.
Soss, Joe. 2006. “Talking Our Way to Meaningful Explanations: A Practice-Centered Approach to In-depth Interviews for Interpretive Research.” In Interpretation and Method, edited by Dvora Yanow and Peregrine Schwartz-Shea. New York: M. E. Sharpe, 127–149.
Taylor, Charles. 1971. “Interpretation and the Sciences of Man.” Review of Metaphysics vol. 25, no. 1: 3–51.
Wendt, Alexander. 1998. “On Constitution and Causation in International Relations.” Review of International Studies vol. 24, no. 5: 101–118.

Re: Limits to Transparency

Post by SMajic » Thu Oct 20, 2016 7:56 am

nrsmith.ccny wrote:From the moderators: What can ethnographers generally not share about their research process and why can they not share those things?


While ethnographers/POs who study similar sites may no doubt "compare notes" in a more metaphorical sense, I believe that they would be disinclined to share their field notes more generally-- these are often highly contextual, time-bound, and a product of particular relationships and interactions. If someone else were to access these notes and use them as "data," I am not sure how much sense they would make to the person who did not conduct the ethnography/observation firsthand, as the secondary user was not *there*.

Limits to Transparency

Post by nrsmith.ccny » Wed Sep 07, 2016 5:37 pm

From the moderators: What can ethnographers generally not share about their research process and why can they not share those things?

Top