Respected and Esteemed Colleagues:
Please excuse my lateness to the party. (This digital carriage turns into a pumpkin at midnight tonight, I believe.)
Excuse, also, the length of this post. Rather than disassemble my thinking and ship it off to different conversation topics, I’ve kept it all intact in one place. But much of what I post here is also germane to the “Ethnography” and “Fieldnotes and Data Repository” sections of this site.
In this post, I would like to recycle portions of my essay, "The Tyranny of Light," in order to connect the purported title of this section--"Ontological/Epistemological Priors"--with the curious fact that the only thread under this section with a pulse is titled "Dishonesty in Research Raises Concern."
But first, I want to make the simple yet increasingly overlooked point that DA-RT does not equal transparency and transparency does not equal DA-RT. (I honestly feel like the preceding sentence should be read out loud in a super-fast, breathless voice like the disclaimers at the end of car dealership ads on the radio before the further reading of any post on this QTD site.) Instead, what DA-RT has done (and continues to do) is to catalyze very specific and very particular kinds of discussions about transparency (and ethics). There is nothing wrong with these specific and particular discussions in and of themselves, but they become unfortunate to the extent that they exert a kind of discursive alchemy under whose spell we soon forget that we are playacting on a stage not of our own making, and that the props provided are just that: provided props. DA-RT becomes unfortunate to the extent that people misrecognize it for transparency (and ethics) itself, rather than naming it for what it is: a specific, and frankly partisan, enactment of transparency (and ethics). This is why, in this post, I will try to use the clunky phrase “DA-RT catalyzed discussions of transparency” rather than just “transparency.” (End of super-fast, breathless disclaimer.)
So then: why is that DA-RT catalyzed discussions of transparency have moved us so quickly from “ontological & epistemological priors” to “dishonesty?”
As Tom Pepinsky rightly points out in this thread, it's not immediately obvious why DA-RT catalyzed discussions about transparency lead us to arguments about “dishonesty” instead of any number of other generative conversations we might be having.
And yet, I think a closer look at the history of DA-RT catalyzed discussions of transparency helps to illuminate:
1) why we find ourselves discussing dishonesty (a dead-end, I think: more on this later!) instead of any number of other things, and
2) how DA-RT sets the agenda (second face of power, folks!) for discussions—no matter how inclusive--that implicitly privilege some social research ontologies over others, invocations of neutrality notwithstanding.
So then, why the transposition of “ontological & epistemological priors” into debates about “dishonesty?”
It is crucial to understand that, on its proponents’ own account, the original motivation for both DA-RT and for the APSA Ethics Guidelines Revisions that authorized the DA-RT committee to do its work derive directly from concerns about replicability in empirical research conducted within positivist logics of inquiry (for a granular account of DA-RT’s origins, see Dvora Yanow and Peregrine Schwartz-Shea, link at end of this essay). Specifically, “APSA’s governing council, under the leadership of president Henry E. Brady, began an examination of research transparency. Its initial concerns were focused on the growing concern that scholars could not replicate a significant number of empirical claims that were being made in the discipline’s leading journals.” As the dominant DA-RT narrative has it, this emerging crisis of replicability in positivist political science was soon found to also exist, in different registers, for a range of scholars “from different methodological and substantive subfields.” Thus, while the DA-RT narrative acknowledges its specific and particular origins in concerns over replication of empirical studies conducted within positivist logics of inquiry, it moves quickly from there to claiming a widespread (discipline-wide?) set of shared concerns about similar problems across other methodological and substantive subfields (citations for the quotations in the original essay accessible via the link provided at the end of this post).
It’s not surprising, to me, that when concerns about replicability travel from their original ontological homes to forms of social research premised on ontologies that don’t privilege replicability as a “gold standard,” we begin to detect a shift in language from “replicability” to “dishonesty.” This is what some people might refer to as a problem with specifying DA-RT’s “scope conditions” (fancy jargon for, “Yo, your money don’t work around these parts.”) Because DA-RT is unwilling or unable to recognize and acknowledge the particular ontology that gave it birth—indeed, quite the opposite, it explicitly purports be ontologically motherless--all kinds of weird things start happening when it is forced to pack a suitcase and board a flight to ontologically distant territories. Weird things, for example, like an unhealthy and unhelpful obsession with honesty and dishonesty.
On this thread, Jane Mansbridge, Jesse Driscoll, Nancy Hirschman, and others have asked for specific examples of qualitative work charged with dishonesty. I can’t think of an especially great example in political science proper, but if we look just beyond the disciplinary castle walls, a lightning-rod, shrapnel-flinging exemplar of a case comes readily to mind: sociologist Alice Goffman’s 2014 University of Chicago book, On the Run.
I’m currently writing a seven-act play that includes a mock trial of this book, so let me just give you a taste of some of the charges that have been publicly leveled against On the Run by academics and the reading public:
At the heart of this controversy [over Alice Goffman's book] are the fundamental limitations of ethnography as a mode of inquiry. Ethnography can look like an uncomfortable hybrid of impressionistic data gathering, soft-focus journalism, and even a dash of creative writing.
-- Leon Neyfakh, Slate
That those flaws [in Alice Goffman’s book] managed to go unnoticed for so long reflects a troubling race-related blind spot among academic and media elites. The failure of On the Run is not only the failure of an individual book and an author, but of the system that produced them.
-- Paul Campos, Chronicle of Higher Education
And, perhaps most tellingly:
Qualitative “research” is useless because there is no way to tell if what is claimed is a reflection of reality or simply the “researchers” gullibility and biases, or even if it’s all a fabrication…. At least [quantitative research] can be put to the test in replication studies, as is increasingly done in social science. To use a book like Alice’s as a guide to understanding social problems is to put enormous trust in her judgment and honesty—even when she openly admits to being a politically motivated advocate. There’s no way to verify many of her claims.
-- Anonymous comment, Marginal Revolution.com
In list form, these charges look something like this:
- inventing events
- embellishing facts
- changing her versions when challenged
- making egregious factual and methodological mistakes
- refusing to provide the means to verify any empirical claims, including failing to file her dissertation, keeping it under wraps years
later, destroying her data, etc.
- misrepresenting her site
- misrepresenting her relationship to informants
- participating in a conspiracy to commit murder
Now, an actual discussion of the merits of On the Run is beyond the scope of this post (for that, please read my play!), but suffice to say two things here: none of the current proposals put forward by DA-RT would do anything to address any of these concerns. As I wrote in “Tyranny of Light,” and as Catherine Boone, Mneesha Gellman, and others have pointed out in this thread, the idea of a foolproof “verification” device for ethnography is patently absurd. It is the performative equivalent--as Mark Beissinger superbly points out on this thread--of making numerical datasets openly available without disclosing the micro-decisions that went into the alchemy of coding that turned the messiness of the world into those “transparent” numbers. As Beissinger puts it: “If we are talking about real transparency in research, large-n researchers would need to provide extensive documentation on every single coding in their datasets. This is simply not being asked because it is not practicable--even though the real instances of fraud that we are aware of have come from falsified codings in large-n data sets.”
So then, would the depositing of Alice Goffman’s fieldnotes for On the Run in a publicly available database placate the critics of her book? Unlikely.
But, really, why stop with requiring ethnographers to post their fieldnotes, diaries, and personal records? Why not also require the ethnographer to wear 24 hour, 360 degree, Visual and Audio Recording Technology (VA-RT) that will be digitally livestreamed to an online data repository and time-stamped against all fieldwork references in the finished ethnography? Would the time-stamped, 24 hour, 360 degree VA-RT then constitute the raw “data” that transparently verifies both the “data” and the ethnographer’s interpretation and analysis of those data? VA-RT for DA-RT!
VA-RT dramatizes a mistaken view that the ethnographer’s fieldnotes, diaries, and personal records constitute a form of raw “data” that can then be checked against any “analysis” in the finished ethnography. The fallacy underlying the mistaken proposal that ethnographic fieldnotes, diaries, and other personal records should be posted to an online repository derives from at least three places.
The first is an extractive ontology inherent in a view of the research world as a source of informational raw material rather than as a specifically relational and deeply intersubjective enterprise. Fieldnotes, and even VA-RT, will always already contain within them the intersubjective relations and the implicit and explicit interpretations that shape both the substance and the form of the finished ethnographic work. Quite simply, there is no prior non-relational, non-interpretive moment of raw information or data to reference back to. What this means is not only that there is no prior raw “data” to reference back to, but that any attempt to de-personalize and remove identifying information from fieldnotes in order to comply with confidentiality and human subjects concerns will render the fieldnotes themselves unintelligible, something akin to a declassified document in which only prepositions and conjunctions are not blacked out.
Second, fieldnotes, far from being foundational truth-objects upon which the “research product” rests, are themselves texts in need of interpretation. Making them “transparent” in an online repository in no way resolves or obviates the very questions of meaning and interpretation that interpretive scholars strive to address.
And third, neither fieldnotes nor VA-RT offer a safeguard “verification” device regarding the basic veracity of a researcher’s claims. The researcher produces both, in the end, and both, in the end, are dependent on the researcher’s trustworthiness. For it would not be impossible for a researcher to fabricate fieldnotes, nor to stage performances or otherwise alter a VA-RT recording.
The notion of a “data repository,” either for ethnographic fieldnotes or for VA-RT, is dangerous both because it elides the interpretive moments that undergird every research interaction with the research world in favor of a non-relational and anonymized conception of “information” and “data,” and because it creates the illusion of a fail-proof safeguard against researcher fabrication where in fact there is none other than the basic trustworthiness of the researcher and her ability to communicate that trustworthiness persuasively to her readers through the scaffolding and specificity of her finished work.
(An aside that could really be a whole different post: it’s fascinating to compare Goffman’s On the Run with another highly celebrated recent ethnography, Matthew Desmond’s Evicted. Responding in part to the concerns about the “verifiability” of Goffman’s book, Desmond had his own book extensively “fact-checked.” And yet, in his quest for facticity, Desmond writes himself completely out of his own ethnography, so much so that when he appears in the book as an actor, he refers to himself in the anonymized third-person, as “a friend” who helped one of his informants move. We are left to ask: which account is ultimately more trustworthy and persuasive? Desmond’s exhaustively “fact-checked” book in which the reader has no sense at all for how his own presence and positionality shaped the research world he interacted with to get his “facts,” or Goffman’s non fact-checked book in which she offers a careful and detailed (transparent?!) account of her own involvement with her research world and “subjects?” A useful thought exercise indeed.)
“Ok, ok,” you say. “We get it. Conversations about dishonesty are a dead-end. What instead?”
Indeed, what instead? If we were to push back against the agenda-setting second face of DA-RT’s power, if we were to push back against the weirdness that comes with DA-RT’s cross-border ontological travels, what are some things that a non-DA-RT catalyzed conversation about transparency and ethics might lead us to talk about?
In this thread, Tom Pepinsky suggests that instead of talking about dishonesty we should be talking about “improving communication.” To that unobjectionable goal, I would add that we might also be talking about:
1) our relationships with and obligations to the individuals, social worlds, and ecosystems we study and interact with, whether they live in the middle of civil wars or rural Wisconsin (Cramer; Parkinson and Wood; Pachirat)
2) “the systems of patron-client ties, nepotism, and old boy’s networks that keep our institutions [read, “Political Science”] stuck in time, resistant to change from within, and impervious to social problems from without” (Fujii);
3) “asymmetric conditions of knowledge production in the field” (Htun); and
4) a “broader, long-standing, and ultimately unresolvable struggle over foundational questions that social scientists cannot ‘sidestep’” (Sil, Guzman, and Calasanti) and that cannot be managed by any procedural antipolitics machine, no matter how self-consciously inclusive the people in charge of that machine might be.
This alternative conversation agenda is just a start. It is an an agenda for conversation, of course, that has been central to the work many of us have been doing long before DA-RT arrived on the scene. It is an agenda for conversation that our work—with its attention to reflexivity, positionality, and power—makes it impossible for us not to talk about. We can only hope, as Jesse Driscoll puts it so eloquently on this thread, that others might also be persuaded and compelled by “the smell of truth” in such an alternative agenda.
Timothy Pachirat
Link to my Tyranny of Light essay, along with highly recommended essays by: Kathy Cramer; Sarah Parkinson & Elisabeth Wood:
http://scholar.harvard.edu/files/dtingl ... 2015-1.pdfLink to other referenced essays by Lee Ann Fujii; Mala Htun; and Rudra Sil, Guzman Castro, and Anna Calasanti:
https://dialogueondartdotorg.files.word ... ng2016.pdf Link to essay by Dvora Yanow and Peregrine Schwartz-Shea:
https://www.cambridge.org/core/journals ... 27FF2B7287