Page 1 of 1

Transparency may add noise, obstruct valid inference

Posted: Mon Dec 26, 2016 2:01 pm
by akuperman
I have a somewhat different concern about the proposal to require that interview transcripts be made public – ostensibly to promote cumulation of knowledge by enabling verification by future researchers – because I fear it could have the opposite effect as explained below.

My own interview research has been mainly with elites in recent or ongoing internal conflicts – i.e., the political and military elites of rebel/revolutionary/liberation movements and of the states they oppose. In my experience, the patterns of frankness of interviewees fall into three broad categories:

- Very frank from the start of the interview. This is typical of older, retired officials who have little to gain or fear.
- Initially reticent and parroting party line, but over time, as the interviewer demonstrates expertise and sympathy, becoming more frank.
- Parroting party line and refusing to open up throughout. This is typical of current and aspiring politicians.

As a result, transcripts typically are a mix of the following:

- Truth.
- Falsehood followed by Truth.
- Falsehood.

The job of the expert interviewer is two-fold: First, to elicit as much truth as possible from the interviewees using techniques that have been explained in the academic literature and elsewhere. Second, to separate the truth from the falsehood, using a variety of techniques that rely on the interviewer’s expertise of the case and research method. For example, if I know you lied in your last assertion, then I should take your next assertion with a grain of salt. Or if I hear a change in your tone of voice, or if you start blinking rapidly, or if you abruptly end the interview, it may signal a switch from honesty. By employing such techniques, the expert interviewer is separating the signal from the noise – and thus aiding the cumulation of knowledge.

The proposed requirement to make public the transcripts of interviews would effectively add back the noise to the signal. One can imagine a future researcher looking at the transcripts – without benefit of the interviewer’s case expertise, method expertise, and face-to-face presence with the interviewee – and reaching a very different conclusion based on weighing the noise and signal equally. Indeed, the future researcher could employ an ostensibly more “scientific” method, such as computerized textual analysis, to make inferences that would be far less valid – due to the well know problem of “garbage in / garbage out.”

This may not be the biggest concern with the proposal, but it is a substantial one, because it means the proposal could have an effect opposite to its stated intention.