Pursuing transparency in qualitative research
Posted: Tue Nov 08, 2016 6:58 pm
It is not possible to replicate qualitative work in the same way as quantitative work, especially given how idiosyncratic fieldwork can be. No one is realistically going to replicate interviews that are done either with groups of people or specific individuals. Obtaining data often depends on serendipity (accidentally running into a terrific interviewee), or taking part in specific opportunities provided by events like conferences or observing happenings like a demonstration. Working in dangerous places is even more precarious and contingent. But a reviewer can have a sense of the project and the depth of the data based on similar work they have done in other contexts.
I am very much against posting interview transcripts. I am presently conducting field research in Algeria. If my interviewees were to think I was going to make their interviews public, even if anonymized and even if the IRB allowed it (which they won’t), they would never agree to the interviews in the first place. Even if there are no security issues involved, there are privacy issues, issues of reputation, of pride, of not wanting to malign other people needlessly, and even of libel to consider.
There are considerations of context and background which factor into one’s interpretation of one’s data. There are popular myths and stereotypes that come through in interviews and if someone not familiar with the context reads isolated interviews, they may think that the author has misrepresented a certain reality, when in fact, they have weighed various forms of evidence against one another and come to a conclusion that may contradict the interviews. It may damage relations with interviewees if one were to publicly challenge them and explain the discrepancy.
To give one example, I have done enough interviews and found enough statistical data to learn that donors and the UN did play a role in gender quota adoption in Algeria, but some activists, who were funded themselves by donors, said categorically that the donors did not play a role because they wish to provide a certain image of what happened. While I convey in my writing that it was mainly domestic actors that played the dominant role, if I were to provide the transcript of some interviews, it would show that they think domestic actors were the only actors. I know their motivation in saying this (not wanting to appear under donor influence and wanting to challenge the view that donors were the only actors) but I don’t want to challenge their view publicly and directly because these are key actors who I am sure I will interview in the future. My written work will challenge their view but only indirectly. I will cite other sources that provide what I consider a balanced perspective. However, someone reading such an interview transcript might assume I misrepresented the situation, when in fact I did not. But to explicitly explain such a discrepancy would damage my relations with my informants or it may be tangential to the study. Providing selected interview transcripts is inadequate because the reviewer does not have the full context of the interview in relation to other interviews, survey data, and other sources based on living and experiencing the situation. The interview is still only partial evidence and cannot be taken on its own as evidence of something. It has to be seen as part of a whole and not a standalone evidence in the way one might regard a statistical analysis.
So how does one evaluate qualitative evidence without replicating it? Ideally a reviewer has a familiarity with the context or a similar context. If for example we are talking about interview based evidence, in order to adequately evaluate the research, the reviewer will need to know some of the following:
• The duration, time, and general location of the study
• 1) What kinds of people were interviewed; 2) how many; 3) How representative are they of the general population or specific population if it is a targeted study; 4) do they represent a variety of demographic groupings and if not, why not and does it influence the findings; 5) do they represent a variety of views if they are elite interviews.
• How was the data analyzed.
• Limitations of the study. Possible biases in the study.
• Is there triangulation in the study? Are the findings backed up in other ways through related surveys, participant observation, other fieldwork experiences, similar research in other locations, in secondary sources, or in other primary sources such as newspaper articles? Are the findings backed up through other methods, e.g., content analysis, crossnational research, survey research. If not, what explains the discrepancy?
• What do historical or earlier studies tell us and if the pattern has changed, why has it changed?
• If the findings differ from other neighboring regions, other studies, or other contexts, can the author account for why their evidence is different.
• Remaining puzzles or unexplained findings that come from the evidence.
It is not any one piece of evidence that matters. Is is the context and how one explains how the evidence fits into that context. It is also a question of depth of knowledge of the context that is important, which is ultimately intangible.
I am very much against posting interview transcripts. I am presently conducting field research in Algeria. If my interviewees were to think I was going to make their interviews public, even if anonymized and even if the IRB allowed it (which they won’t), they would never agree to the interviews in the first place. Even if there are no security issues involved, there are privacy issues, issues of reputation, of pride, of not wanting to malign other people needlessly, and even of libel to consider.
There are considerations of context and background which factor into one’s interpretation of one’s data. There are popular myths and stereotypes that come through in interviews and if someone not familiar with the context reads isolated interviews, they may think that the author has misrepresented a certain reality, when in fact, they have weighed various forms of evidence against one another and come to a conclusion that may contradict the interviews. It may damage relations with interviewees if one were to publicly challenge them and explain the discrepancy.
To give one example, I have done enough interviews and found enough statistical data to learn that donors and the UN did play a role in gender quota adoption in Algeria, but some activists, who were funded themselves by donors, said categorically that the donors did not play a role because they wish to provide a certain image of what happened. While I convey in my writing that it was mainly domestic actors that played the dominant role, if I were to provide the transcript of some interviews, it would show that they think domestic actors were the only actors. I know their motivation in saying this (not wanting to appear under donor influence and wanting to challenge the view that donors were the only actors) but I don’t want to challenge their view publicly and directly because these are key actors who I am sure I will interview in the future. My written work will challenge their view but only indirectly. I will cite other sources that provide what I consider a balanced perspective. However, someone reading such an interview transcript might assume I misrepresented the situation, when in fact I did not. But to explicitly explain such a discrepancy would damage my relations with my informants or it may be tangential to the study. Providing selected interview transcripts is inadequate because the reviewer does not have the full context of the interview in relation to other interviews, survey data, and other sources based on living and experiencing the situation. The interview is still only partial evidence and cannot be taken on its own as evidence of something. It has to be seen as part of a whole and not a standalone evidence in the way one might regard a statistical analysis.
So how does one evaluate qualitative evidence without replicating it? Ideally a reviewer has a familiarity with the context or a similar context. If for example we are talking about interview based evidence, in order to adequately evaluate the research, the reviewer will need to know some of the following:
• The duration, time, and general location of the study
• 1) What kinds of people were interviewed; 2) how many; 3) How representative are they of the general population or specific population if it is a targeted study; 4) do they represent a variety of demographic groupings and if not, why not and does it influence the findings; 5) do they represent a variety of views if they are elite interviews.
• How was the data analyzed.
• Limitations of the study. Possible biases in the study.
• Is there triangulation in the study? Are the findings backed up in other ways through related surveys, participant observation, other fieldwork experiences, similar research in other locations, in secondary sources, or in other primary sources such as newspaper articles? Are the findings backed up through other methods, e.g., content analysis, crossnational research, survey research. If not, what explains the discrepancy?
• What do historical or earlier studies tell us and if the pattern has changed, why has it changed?
• If the findings differ from other neighboring regions, other studies, or other contexts, can the author account for why their evidence is different.
• Remaining puzzles or unexplained findings that come from the evidence.
It is not any one piece of evidence that matters. Is is the context and how one explains how the evidence fits into that context. It is also a question of depth of knowledge of the context that is important, which is ultimately intangible.