Substantive Dimensions of the Deliberations
We encourage contributors to the Discussion Board to publicly identify by registering and logging in prior to posting. However, if you prefer, you may post anonymously (i.e. without having your post be attributed to you) by posting without logging in. Anonymous posts will display only after a delay to allow for administrator review. Contributors agree to the QTD Terms of Use.
Instructions
To participate, you may either post a contribution to an existing discussion by selecting the thread for that topic (and then click on "Post Reply") or start a new thread by clicking on "New Topic" below.
The transition to Stage 2 of the deliberations is currently underway but will take some time to complete. In the meantime, we very much welcome additional contributions to the existing threads in this forum.
For instructions on how to follow a discussion thread by email, click here.
-
Gary Marks
University of North Carolina at Chapel Hill - Posts: 1
- Joined: Mon Apr 18, 2016 9:23 am
Are qualitative researchers held to a higher standard?
What would this mean if applied to quantitative research? What would it mean for those, like myself, who do a fair amount of quantitative as well as qualitative analysis? It would certainly mean more than providing the final do file which is merely the tip of iceberg in the process of inference. The final do file is just the end result of a quantitative scholar’s efforts. It says little about how one actually draws analytic conclusions from the data. A full account of the process would be to provide an account of all the models and variable adjustments that were used to analyze the data, i.e. the complete set of do files.
If the invocation to transparency were interpreted literally in this way, I would expect quantitative scholars would find this as burdensome as qualitative scholars would find writing a series of annotations explaining how their sources support qualitative claims.
I wonder if we are holding qualitative researchers to a higher standard?
Post Reply
-
Paolo Spada
Southampton - Posts: 11
- Joined: Wed Apr 06, 2016 8:24 pm
Re: Are qualitative researchers held to a higher standard?
I think that in the attempt to solve the problems of statistical observational research we might generate the same problems for qualitative research.
I do not know what the audience is so I have written a modular post:
-> for those that know little about experiments, statistics and are allergic to numbers start below at 1,
-> for those that have familiarity with these methods jump to 2
1: One step back to explain what I mean
The ideal standard of experiments is to trace everything, announce the model ex-ante and generate the data via the experiment, so both the model and the data are replicable. The researcher after having deposited the model in a public repository could be effectively removed from the equation and substituted by a robot executing a program. The more an experiment is 'automatic' the better.
Observational statistical analysis is not so different from story telling
In observational statistical analysis (i.e. non experimental stuff in which I use secondary data or I collect data) the code and actually the dataset itself that is uploaded in journals, as Gary notices, are often a subset of all the data, robustness checks and alternative specifications that have been tried. For example the dataset I developed in 8 years of research going back and forth in the field has hundreds of variables, but when I try to publish stuff out of it I select 20-30 variables that are used in the paper or the reviewers go crazy. The paper becomes too complex for anybody apart me to comprehend (that is actually one of the review I got in my first rejection). I am telling a story with statistics on the basis of my experience and field knowledge, and I am extremely clear about the limit of my observational studies exactly because I use also lots of experiments and qualitative research, and I know that what I am doing is as 'filtered' by me subconsciously as my own qualitative research. To my eyes there is not much difference between qualitative research and statistical observational study.
Experiments are more rigorous, but..
Experiments are, potentially, more rigorous, but they can only answer so much and I see them as a surgical tool that works well only if a lot of qualitative and quantitative research has already been done on the topic. If not most experiments are explorations that very often leave me with the "everybody that has done two minutes of research in that field knows that" feeling. So we need a lot of qualitative and observational research to prepare good experiments. And by the way I am not saying anything radical here, any good intro 101 class on experimental analysis teaches to prepare a lot before fielding an experiment. it's obvious stuff.
One of the major critique that experimentalists have toward observational statistical analysis is that it is not automatic enough
The major critique that experimentalists have toward traditional statistical analysis is that if the researcher is not rigorous they can try 100 different models using different subsets of explanatory variables and then select a subsets of combinations that fits their story and the most common robustness checks. And even if the researcher is super-rigorous they might affect the model selection subconsciously. And experimentalists are right, and the format of current publications and the fact that there is a premium in churning out papers instead on the value of the actual findings exacerbates the whole problem. If nobody reads my 500 robustness checks and it takes lots of time to do them well, the best ones require new data collection and require even years, I have zero career incentives to do them, and I have strong incentives to package a very coherent, simple and clean story that is a sandbox with low external validity. Da-rt pushes only to make sure that the internal coherence of the sandbox is ok. It is a good step forward because studies have shown that we do not even have coherent sandboxes, but it does not solve the real problem that is about the overall perverse incentives.
2: The problem with imposing the 'tip of the iceberg uploading' standard on qualitative research
The interesting editorial of Lupia makes the case that we are making a big fuss, and that editors are reasonable and we would not have to upload every single note and scribbling, that is what he calls the comic book version of Da-art, but we would simply have to upload a few things that support the crucial elements of our argument. This to me appears to be an exact description of what Gary calls 'the tip of the iceberg" approach that is currently the standard for statistical study. I fear that by pushing a similar 'tip of the iceberg uploading' standard on qualitative research we will inject the bad incentives currently messing-up statistical studies on qualitative research. I.e. we would promote a mickey mouse version of research that can be justified with a self-selected sample of interviews and whatever else we need to upload. The cost of uploading content will reverse engineer the arguments. If I can't justify X because it would require too much uploading, or I do not have anything clear to upload because X emerged organically from 100 cold beer conversations, as one colleague describes in another post, I would have an incentive not talk about X even if it is really important and would propel the debate forward, or I would fake an interview. This is exactly what the current problem with statistics is, to publish a study needs to achieve significance, thus researchers have an incentive to focus only on research that achieves that. In experiment null results disappear, in observational studies the incentive to play with the data even just subconsciously is gigantic.
More nuanced standards?
So I am wondering if it makes sense to generate an approximation of the observational statistics standard (upload just the tip of the iceberg) for qualitative research EVEN if we find a way to not generate a crazy burden on the researcher. For example create a three level replication standard: A) experimental (this should be higher than what Da-rt requires, we need a repository of models that are submitted ex-ante so that we can tie even further the hands of the researchers and we can also share info on null results experiments that are not published), B) observational (Da-rt seems to work pretty well for that, it is ridiculous that we can't replicate published studies), C) qualitative (some sort of simplified Da-rt that would consider the cost of uploading). This diversified approach, maybe with multiple levels, was my prior entering this discussion. However now I am starting to have doubts due to risk described in the previous section. But maybe such problems can be compensated with other intervention not at the journal level.
Super high standards for type A research, independently of the method, everything else goes in type B
I am also wondering if the crazy high standard that Gary describes almost as an absurdity should actually be implemented. So that in my example I would have to upload the whole diary of my statistical observational study research process from start to end. The diary should be documented and show all the dead-end statistical models I have tried. Everybody I talked to and has influenced my thinking (those are interviews). Good ethnographers keep such diaries, so this solution is outlandish only because the current practice is different. And in my case it would certainly force me to think and rethink of every choice I have made and I would have to make available all my 100+ variables. And we would have to justify much more. It's really annoying, costly, and we will have to start keeping diaries over diaries, but it would trace everything. At such point the standard would be incredibly high for everybody. For sure quantitative focused PhDs would then take as much time as qualitative and multi-method ones, so it would level the plain-field in a certain way. The problem with this is that those that can hire RAs or have lots of resources will have an even greater advantage than what they have now, as Stephanie McNulty has mentioned in another post. But the devil advocate in me asks, what would be the systemic effect? Would the gain from improving the quality of research of those with lots of resources be greater then the loss of adding additional burden on those that are in universities with none? Or the loss in diversity would be such that the epistemic quality of the whole field will be reduced? I study deliberative democracy, and I am a fan of the diversity theorem, and thus I prefer the second answer, but I can see that is not as non controversial as it might appear prima facie. People that got a job in high resource universities have won a tournament, it's like the premier league, as unfair as it is, they are quite good at what they do and enforcing a rule to make what they do even better might generate a systemic improvement, even if it decreases the ability of the secondary leagues to compete.
Or maybe we need something else
Or maybe there is a way to promote good practice at the systemic level in another way without imposing any standard on qualitative research. I do not know what that could be, beyond the obvious that is promote mentoring, but I sense that it should come earlier, not at the journal stage. Maybe what we need is a better social science IRB process that is designed not for drug testing, but to promote high quality research.
What do you think Gary? Others?
Post Reply
-
Guest
Re: Are qualitative researchers held to a higher standard?
However, journal editors should not be put in a position of power to determine whatever these standards should be. If there's a problem with particular scholars manipulating data, then target that problem rather than creating a detrimental infrastructure.
Post Reply
-
Guest
Re: Are qualitative researchers held to a higher standard?
Arguably, the requirement to produce a historian like, thick, discursive footnote situating a causal inference in the broader historiographical context and differentiate it from competing interpretations would probably strike a fear in quantitative scholars just as intense as DA-RT seems to inject in its qualitative skeptics. I think this is what Gary is alluding to here. (To get a flavor of such footnoting, I recommend Anthony Grafton's delightful "The Footnote. A Curious History."
If this were the case, it would underscore that many (but not all) DA-RT requirements would affect scholarship independently of its methodological orientation. It would ask more of quantitative scholars by requiring them to be more transparent about their pre-testing, theorizing and offer more than just perfunctory drive-by literature reviews. And it would ask more of qualitative scholars by requiring to be more transparent in the post-theorizing, "testing" causal inference stage of analysis.
I would say that at this stage, the quantitative scholars have a head-start on qualitative ones because DA-RT requirements are better articulated for the testing, causal inference stage of analysis. But there is nothing to prevent qualitative scholars to catch up by asking their quantitative colleagues to be more transparent about how their theorizing generated the hypotheses they end up testing. The sign that we might have succeeded would be similar push back against DA-RT from quantitative scholars as we currently see from many qualitative scholars.
Post Reply
-
Guest
Re: Are qualitative researchers held to a higher standard?
I think this makes total sense and maybe we could write together a quick document that makes such point clearly and is immediately released by the organizers and invites the rest of the discipline to come comment on it in these boards. As much as I enjoy the high quality discussion that is occurring in this forum, the community of 30-40 people currently contributing is too small to be representative, thus I urge the organizers to redouble their efforts to invite people to these boards.
The idea of the previous guest seems a great one to start conversations and make people working exclusively on statistical analysis think about how does it feel to be asked to upload things that they consider part of their 'back-office' process.
Post Reply
-
Guest
Re: Are qualitative researchers held to a higher standard?
*I write "what some see" here because in my reading, the implementation of DA-RT by the editors of both the APSR and CPS is not as onerous as critics of the standards expect that it might be.
Post Reply
-
Guest
Re: Are qualitative researchers held to a higher standard?
Post Reply
-
Guest
Re: Are qualitative researchers held to a higher standard?
Post Reply
-
Alan Jacobs
University of British Columbia - Posts: 38
- Joined: Fri Feb 26, 2016 9:59 pm
Re: Are qualitative researchers held to a higher standard?
One interesting question suggested by one of the posts above is about the role of peer reviewers, as opposed to editors or editorial rules, in the pursuit of transparency. There has been an interesting discussion of who the transparency "gatekeepers" ought to be in a thread over here. Should it be editors? Reviewers? Or should authors be left to make judgements about transparency (perhaps, e.g., where human subject protection is at stake) themselves?
These questions are also closely related to what one contributor has referred to as the "DA-RT subsidarity principle that transparency evaluations should be made at the most local or ontologically most proximate level." See here. What do people think of this principle? How might editors operationalize it? What counts as the "most local" or "ontologically proximate" level?
One might also ask whether reviewers sometimes need access to original data in order to do the job we ask of them. In the context of quantitative analysis, Nicole Janz has made a provocative argument on this point.
We'd welcome further thoughts on all of these issues (either here or in the more specifically relevant thread).
Post Reply
-
Guest
Re: Are qualitative researchers held to a higher standard?
I agree completely. I'm an Associate Professor in security studies with lots of publications under my belt, and frankly, the idea of having to meet the DA-RT requirements makes me want to throw in the towel and go start another career. Pressure has been increasing on all sides - teach more students, publish more articles, get more grant money - and then to have this on top of it? It makes qualitative research seem unduly and unreasonably difficult.
Post Reply
-
Catherine Boone
London School of Economics - Posts: 6
- Joined: Thu Apr 07, 2016 3:33 pm
Re: Are qualitative researchers held to a higher standard?
Post Reply
-
Guest
Re: Are qualitative researchers held to a higher standard?
Post Reply
-
Marcus Kreuzer
Villanova University - Posts: 26
- Joined: Sat Apr 16, 2016 9:48 am
Re: Are qualitative researchers held to a higher standard?
Here are a few suggestions on how DA-RT could be integrated more fully in the review process and thereby shift transparency evaluations from editors to reviewers.
1) Some journals currently solicit ratings for an article's originality, rigor etc. Why not do the same and ask reviewers for transparency ratings. To increase the validity of any rating, the reviewer might also be asked to briefly elaborate on his/her ratings.
2) The "DA-RT subsidarity principle" for qualitative work should be pretty straight forward. The more knowledge a reviewer has of the literature, case(s), or sources the more "local" he/she becomes. Maybe the reviewers could be asked to briefly identify his local knowledge.
3) The "DART subsidarity principle" for quantitative work is more complex because it makes more heroic assumption about the uniformity of data. (i.e.conditional independence, unit homogeneity) Here reviewers might asked to assess to what extent the data does or does not meet those two conditions.
4) Transparency should not be viewed as a threshold requirement that either is or is not met. No quantitative data set will ever fully uniform and no case study data will fully available. Transparency should always be evaluated on a continuum and acknowledging departures from benchmarks should not become sufficient to reject a piece. Instead, such acknowledgments should serve as background information for better judging its broader validity.
5) I also think that there is potential problem of "faux transparency" where reviewers use overly strict transparency expectations to reject submissions. This might be the fear that informs many posts. An effective solution to this would be to have reviewers review each other's reviews. This would encourage reviewers to be more explicit in defending their criticisms and might even let them to re-evaluate them in light of other reviewer's comments. Plus, it would it both easier for editors to reach informed decisions and shift their editorial discretion to people more knowledgeable on the subject. This would be consistent with the subsidarity principle.
6) The opposite of the subsidarity principle would be to improve the effectiveness of post-publication discussion. See more. https://www.qualtd.net/viewtopic.php?f=10&t=56#p149
Post Reply
-
Guest
Re: Are qualitative researchers held to a higher standard?
Post Reply
-
Guest
Re: Are qualitative researchers held to a higher standard?
Conceptualization and measurement are the foundation of both qualitative and quantitative research. These steps in the research process are also often some of the least transparent, and offer the greatest opportunities for various forms of funny business. Anybody who has been around the block a few times realizes that there are people out there conducting quantitative research who iteratively work through different approaches to conceptualization/measurement of a phenomenon until finding one that generates a set of data that will support their pet hypothesis. Yet DA-RT seems largely silent on these issues.
In essence, qualitative researchers are expected to make explicit all sorts of links between observed phenomena in the world and their descriptive inferences regarding that phenomena, yet quantitative researchers are allowed to have the steps linking observed phenomena in the world to entries in a tabular data set remain a total black box. If the descriptive inferences that are the foundation of your research are reified into a quantitative data set, they no longer need to be made transparent. But if your descriptive inferences are not reified into a quantitative data set, then they should be meticulously sourced and subject to scrutiny.
I have a hard time seeing how this isn't a massive double standard.
Post Reply
-
Guest
Re: Are qualitative researchers held to a higher standard?
And yet, the applying the DART initiative to qualitative work would require a huge amount of time/effort/resources (for example, by providing embedded links to sources about a paper's core claims)-- and this, all in the name of transparency when it is not at all clear to me that qualitative research is suffering from any lack of transparency in the first place. Now this is not to say that we could not be more self-reflective when it comes to questions of (say) selection bias in our interview base, or in the types of historical evidence that we use. But DART just imposes unreasonable and unnecessary costs. And this in a moment when qualitative work is getting increasingly short shrift in the discipline.
I do like Marcus Kreuzer's efforts to be constructive about DART! But I also worry that many of the suggestions are going to be difficult to implement. For example, I already feel over-burdened; being asked to peer review other peer reviewers seems like it would add an extra burden without having a clear effect. It would also add another month at least in terms of article turnaround time, when many journals already take ages and ages. This is perhaps not a problem for senior scholars, but is deeply stressful for juniors.
Similarly, I worry that self-ratings of local knowledge, while useful in theory, are going to be difficult to implement in practice. How many people have spent time in a given African village, or worked with a given set of files at a given archive in London? I think editors currently try to get as 'local' as possible. I am frequently asked to review pieces on countries that I have done research on, even when the subject matter of the papers is quite far afield from my own research interests; similarly, I am also asked to review pieces because of my in-depth knowledge of certain datasets. I interpret this as journal editors' efforts to get some reasonable degree of local knowledge.
Post Reply
-
Elliot Posner
Case Western Reserve University - Posts: 8
- Joined: Tue May 10, 2016 10:42 pm
Re: Are qualitative researchers held to a higher standard?
I’m leaning toward differentiated transparency standards. However, I worry about dividing research traditions into positivist and interpretivist camps for this purpose (See Büthe and Jacobs’ Conclusion in QMMR, Spring 2015). When in the muck of early empirical research (but also during later stages), this division can be a false dichotomy. More demanding transparency standards have the potential to become a bridge between the traditions by encouraging honest discussions (and the development of norms) about where and how judgment and interpretation (are appropriate to) intersect with typical “positivist” small-n analysis. Separate transparency standards might have unintended consequences that lead to unnecessary fragmentation.
Post Reply
-
Tim Buthe
HfP/Technical Univ of Munich & Duke University - Posts: 32
- Joined: Fri Feb 26, 2016 11:39 pm
Re: Are qualitative researchers held to a higher standard?
Since the QTD(s) are supposed to move, after the end of this week, to the more differentiated "stage 2" deliberations, let me take Elliot's point about "leaning toward differentiated transparency standards"--but being worried about artificial distinctions--as an occasion to encourage all of you who have read and/or contributed to this thread to help the Steering Committee decide:
What should be the dimensions for differentiating? What aspects of qualitative research warrant separate working groups to develop a better, differentiated understanding of the meaning of research transparency, the trade-offs involved, and practicalities of research transparency? Research contexts and specific empirical methods are just examples of possible dimensions for the more differentiated deliberations.
The QTD Steering Committee would like to get as much input as possible into this question--ideally by the end of this week. We've created a separate thread here for replies on this issue.
Post Reply
-
Guest
Re: Are qualitative researchers held to a higher standard?
Post Reply
-
Guest
Re: Are qualitative researchers held to a higher standard?
Post Reply