nrsmith.ccny wrote:From the moderators: What can ethnographers generally not share about their research process and why can they not share those things?
I am pasting in below a piece I wrote for the QMMR Symposium on Transparency in Qualitative and Multi-Method Research, in the Spring 2015 issue, under the title, "Transparent Explanations, Yes. Public Transcripts and Fieldnotes, No: Ethnographic Research on Public Opinion." I hope it is a useful contribution to this important discussion.
I am a scholar of public opinion. My main interest is in examining how people understand, or interpret, politics. For that reason, much of my work involves listening to people talk with others with whom they normally spend time. When I listen to the way they sort out issues together, I am able to observe what they perceive to be important, the narratives they use to understand their world, the identities that are central to these understandings, and other important ingredients of public opinion.
My work is therefore primarily qualitative, and usually interpretivist. By interpretivist, I mean that I am trying to capture how people perceive or attribute meaning to their worlds. I treat that effort as a necessary part of trying to understand why they express the opinions that they do.
Across the course of my career, transparency has been a professional necessity. My methods are rather unusual in the field of public opinion research, so the burden is on me to teach my readers and my reviewers what I am doing, why I am doing it, and how my work should be judged. Usually, public opinion scholars focus on individuals’ preferences and how to predict them, not on the process of understanding. In addition, we tend to be well versed in the strengths and weaknesses of polling data, but basically unfamiliar with conversational data.
Put another way, reviewers are likely to dive into my papers looking for the dependent variable and the strength of evidence that my results can be generalized to a national population. But my papers usually do not provide information on either of these things. Unless I explain why my work has different qualities to be judged, the typical reviewer will quickly tune out and give the paper a resounding reject after the first few pages.
So the first thing I have had to be transparent about is the fact that much of my work is not attempting to predict preferences. My work typically does not describe how a set of variables co-vary with one another to bring about particular values on a dependent variable. Indeed, I’m not usually talking about causality. These characteristics are just not what scholars typically come across in political science public opinion research. I have had to go out of my way to explain that my work uses particular cases to help explain in detail the process of a group of people making sense of politics. I have had to be up front about the fact that my goal is to help us understand how it is that certain preferences are made obvious and appropriate when objective indicators about a person’s life would suggest otherwise.
For example, in a piece I published in the APSR in 2012,1 reviewers helpfully pointed out that I had to bluntly state that my study used particular cases to study a broader question. In short, that article reported the results of a study in which I invited myself into conversations among groups of people meeting in gathering places like gas stations and cafés in com- munities throughout Wisconsin, especially small towns and rural places, so that I could better understand how group consciousness might lead people to support limited government when their objective interests might suggest otherwise. I had to take several paragraphs to contrast what I was doing against the more familiar positivist approaches. I wrote:
"My purpose in investigating what people say in the groups they normally inhabit in a particular set of communities within one state is to better explain how the perspectives people use to interpret the world lead them to see certain stances as natural and right for people like themselves (Soss 2006, 316). It is motivated by the interpretivist goal of providing a “coherent account of [individuals’] under-standings as a prerequisite for adequate explanation” (Soss 2006, 319; see also Adcock 2003). In other words, to explain why people express the opinions that they do, we need to examine and describe how they perceive the world. In this article I explain the contours of the rural consciousness I observed and then specify its particularity by contrasting it with conversations among urban and suburban groups. That is, this is a constitutive analysis (an examination of what this thing, rural consciousness, consists of and how it works) versus a causal analysis (e.g., an examination of whether living in a rural place predicts rural consciousness—McCann 1996; Taylor 1971; Wendt 1998). The point is not to argue that we see consciousness in rural areas but not in other places, nor to estimate how often it appears among rural residents, nor to describe what a population of people thinks. Instead, the purpose here is to examine what this particular rural consciousness is and what it does: how it helps to organize and integrate considerations of the distribution of resources, decision- making authority, and values into a coherent narrative that people use to make sense of the world. This is not a study of Wisconsin; it is a study of political understanding and group consciousness that is conducted in Wisconsin (Geertz 1973, 22).
"To clarify the stakes, contributions, and implications of this study, allow me to contrast it with positivist approaches. I examine here how people weave together place and class identities and their orientations to government and how they use the resulting perspectives to think about politics. A positivist study of this topic might measure identities and orientations to government, and then include them as independent variables in a multivariate analysis in which the dependent variable is a policy or candidate preference. Such an approach is problematic in this case in the following ways. The positivist model specification assumes that values on one independent variable move independent of the other. Or if using an interaction term, it assumes that people with particular combinations of these terms exhibit a significantly different level of the dependent variable. However, the object of study, or my dependent variable in positivist terms, is not the position on an attitude scale. It is instead the perspectives that people use to arrive at that position. My object is not to understand the independent effects of identities and attitudes such as trust, or how people with different combinations of these compare to others, but to understand how people themselves combine them—how they constitute perceptions of themselves and use these to make sense of politics."
I include this excerpt to underscore that transparency in the sense of explaining in detail my data collection and analysis procedures, as well as my epistemological approach, has been a professional necessity for me. Without providing extensive detail on my research approach, many readers and reviewers would not recognize the value of my work. Indeed, on one occasion in which I did not take the space to provide such detail, one exceptionally uncharitable reviewer wrote, “I found this chapter very disappointing. This perhaps reflects my bias as a researcher who does formal models and quantitative analysis of data. Briefly, I, too, can talk to taxi drivers or my mother’s bridge crowd.” There were just 3 additional sentences to this person’s review.
Transparency has also been important for me for another reason. In interpretive work, we make evidence more valuable and useful the more context we provide. As we describe and examine the meaning that people are making out of their lives, we better equip our readers to understand and judge our claims the more information we provide about what leads to our interpretations. Positivist work has the burden of providing evidence that a given sample is representative of a broader population. Interpretivists must provide enough context that our interpretations are “embedded in, rather than abstracted from, the settings of the actors studied.” Transparency is an integral part of the presentation of our results. For example, in a forthcoming book based on the study I describe above, I explain the physical nature of the settings in which I met with rural residents, the lush green fields and blue skies of the com- munities in which they resided, what they wore, and how they responded to me as a smiley city girl arriving to their gas stations, etc., in my Volkswagen Jetta to convey in part the complexity of the perspectives with which the people I spent time with viewed the world. They were deeply proud of their com- munities and their groups of regulars, and at the same time resentful of the economic situations in which they found themselves.
In those respects, I value transparency. But in another respect, I do not. In particular, I do not consider making my field notes publicly available to be a professional duty or necessity. I am troubled by the recent push in our discipline to make available the transcripts of the conversations I observe and my fieldnotes about them to anyone and what it means for my future research. I have three specific concerns.
First, asking me to make my data publicly available assumes that any scholar could use it in the form it exists on my computer. That is, the assumption is that if I provide the transcripts of the conversations, and a key providing an explanation of important characteristics of each speaker, any scholar would be able to treat those words as representations of the conversation. I am just not confident that is possible. My fieldnotes are pretty detailed, but I do not note the tone or inflection of every sentence. I do not include every detail of the town or the venue in which I visit with people. I record details about what people are wearing, how the coffee tasted, and the pitch of some folks’ voices, but when I re-read a transcript, a thousand images, sounds, and smells enter my mind to round out the impression of the messages people were conveying to me and to each other. For this reason, I do not send assistants to gather data for me. I need to be there. I need to experience the conversation in all its fullness to get the best possible picture of where people are coming from.
Could a scholar still get something useful from my transcripts? Perhaps—the frequency of the use of some words, the general frames in which certain issues are talked about— all of that would be available in my transcripts. But my sense is that this push for transparency is not about broadening the pool of available data for people. It is about the assumption that making data publicly available will enable replication and the ability to more easily assess the validity of an argument. It is driven by the assumptions that the best social science is social science that is replicable and consists of conclusions that another scholar, given the same data, would arrive at independently. But I do not agree that the transcripts that I or my transcribers have typed out, and the accompanying memos that I have written would allow replication, nor would I expect another scholar to necessarily reach the same conclusions based on re-reading my transcripts.
Let me put it another way. The field of political behavior is pretty well convinced that Americans are idiots when it comes to politics. When you read the transcripts of the conversations I observe, it is not difficult to come away with the conclusion that those exchanges support that general conclusion. The words people have said, typed out on a page, do seem ignorant according to conventional standards. However, when the conversational data is combined with the many intangible things of an interaction—the manner in which people treat each other, the physical conditions in which they live—their words take on meaning and reasonableness that is not evident from the transcription of their words alone.
Let me reiterate that the purpose of my work is not to estimate coefficients. There is not a singular way to summarize the manner in which variables relate to one another in my data. Instead, what I do is try to characterize in as rich a manner possible how people are creating contexts of meaning together. Should I enable other scholars to look at my data to see if they would reach the same conclusion? I do not think it is possible to remove me from the analysis. In my most recent project, conversations about the University of Wisconsin-Madison became a key way for me to examine attitudes about government and public education. I work at that institution. I have a thick Wisconsin accent. My presence became a part of the data. Another scholar would not have all of the relevant data needed for replication unless he or she is me.
I am against this push for making transcripts and fieldnotes publicly available for another reason. Ethically, I would have a very difficult time inviting myself into conversations with people if I knew that not only would I be poring over their words in detail time and time again, but that an indeterminate number of other scholars would be doing so as well, in perpetuity. How do we justify that kind of invasion? You might say that survey research does this all the time—survey respondents are giving the permission for an indeterminate number of people to analyze their opinions for many years to come. But we tell respondents their views will be analyzed in the aggregate. Also, collecting conversational data, in person, in the spaces that people normally inhabit, with the people they choose of their own volition to do so, is not the same as collecting responses in a public opinion poll, even if that poll is conducted in person. When you are an interviewer for a public opinion poll, you are trained to be civil, but basically nondescript—as inter- changeable with other interviewers as possible. That is just about the opposite of the approach needed in the kind of research I conduct. I have to walk into a group as my authentic self when I ask if I can join their coffee klatch and take up some other regular’s barstool or chair. I am able to do the kind of work I do because I am willing to put myself out there and connect with people on a human level. And I am able to gather the data that I do because I can tell people verbally and through my behavior that what they see is what they get. If the people I studied knew that in fact they were not just connecting with me, but with thousands of anonymous others, I would feel like a phony, and frankly would not be able to justify doing this work.
My main worry with this push for transparency is that it is shooting our profession in the foot. I am concerned in particular about the push for replicability. Does the future of political science rest on replicability? I have a hard time seeing that. Perhaps because I work at the University of Wisconsin-Madison, in which tenure and public higher education are currently under fire, when I think about the future of our profession, I think about the future of higher education in general. It seems to me that our profession is more likely to persist if we more consciously consider our relevance to the public and how our institutions might better connect with the public. I am not saying that we should relax scientific standards in order to appease public will. I am saying that we should recognize as a discipline that there are multiple ways of understanding political phenomena and that some ways of doing so put us in direct connection with the public and would be endangered by demanding that we post our transcripts online. Why should we endanger forms of data collection that put us in the role of ambassadors of the universities and colleges at which we work, that put us in the role of listening to human beings beyond our campus boundaries? It is not impossible to do this work while making the transcripts and field notes publicly available, but it makes it much less likely that any of us will pursue it.
I do not think outlawing fieldwork or ethnography is the point of the data access initiative, but I fear it would be an unintended consequence resulting from a lack of understanding of interpretive work—the kind of misunderstanding that leads some political scientists to believe that all we are up to is talking with taxi drivers.
My interpretive work seeks to complement and be in dialogue with positivist studies of public opinion and political behavior. Its purpose is to illuminate the meaning people give to their worlds so that we can better understand the political preferences and actions that result. Understanding public opinion requires listening to the public. Transparency in this work is essential to make my methods clear to other scholars in my field who typically are unfamiliar with this approach, so that they can understand and judge my arguments. But I do not think that mandated transparency should extend to providing my transcripts and fieldnotes. My transcripts and fieldnotes are not raw data. The raw data exist in the act of spending time with and listening to people. That cannot be archived. The expectation for interpretive work should be that scholars thoroughly communicate their methods of data collection and analysis and provide rich contextual detail, including substantial quoting of the dialogue observed. There are many excellent models of such transparency in interpretive ethnographic work already in political science, which we can all aspire to replicate.
1 Cramer Walsh 2012.
2 Cramer Walsh 2012, 518.
3 Schwartz-Shea and Yanow 2012, 47.
4 Cramer 2016.
5 See, e.g., Pachirat 2011; Simmons 2016; Soss 2000.
Adcock, Robert. 2003. “What Might It Mean to Be an ‘Interpret- ivist’?” Qualitative Methods: Newsletter of the American Political Science Association Organized Section vol. 1, no. 2: 16–18.
Cramer, Katherine. 2016. The Politics of Resentment: Rural Consciousness in Wisconsin and the Rise of Scott Walker. Chicago: University of Chicago Press.
Cramer Walsh, Katherine. 2012. “Putting Inequality in Its Place: Rural Consciousness and the Power of Perspective.” American Political Science Review vol. 106, no. 3: 517–532.
Geertz, Clifford. 1973. “Thick Description: Toward an Interpretive Theory of Culture.” In The Interpretation of Cultures: Selected Essays. New York: Basic Books, 3–30.
McCann, Michael. 1996. “Causal versus Constitutive Explanations (or, On the Difficulty of Being So Positive . . .).” Law and Social Inquiry vol. 21, no. 2: 457–482.
Pachirat, Timothy. 2011. Every Twelve Seconds: Industrialized Slaughter and the Politics of Sight. New Haven: Yale University Press.
Schwartz-Shea, Peregrine, and Dvora Yanow. 2012. Interpretive Research Design: Concepts and Processes. New York: Routledge.
Simmons, Erica. 2016. The Content of their Claims: Market Reforms, Subsistence Threats, and Protest in Latin America. New York: Cambridge University Press.
Soss, Joe. 2000. Unwanted Claims: The Politics of Participation in the U.S. Welfare System. Ann Arbor: University of Michigan Press.
Soss, Joe. 2006. “Talking Our Way to Meaningful Explanations: A Practice-Centered Approach to In-depth Interviews for Interpretive Research.” In Interpretation and Method, edited by Dvora Yanow and Peregrine Schwartz-Shea. New York: M. E. Sharpe, 127–149.
Taylor, Charles. 1971. “Interpretation and the Sciences of Man.” Review of Metaphysics vol. 25, no. 1: 3–51.
Wendt, Alexander. 1998. “On Constitution and Causation in International Relations.” Review of International Studies vol. 24, no. 5: 101–118.