Preregistration (or lack thereof)
Posted: Tue May 10, 2016 1:21 am
I am curious about the following issue -- that of preregistration, or rather why it's lacking from most stated DA-RT principles and statements (and what that means).
DA-RT is presented with the intention of enhancing transparency in the conduct of empirical political science research. That research (which some may criticize as too quantitative, or hypothetico-deductive, or what not) idealized by this vision is what I have called the "deductive template" in past work -- the 4-step blueprint where (1) we derive axiomatic and falsifiable hypotheses, next (2) predict all the implications of those propositions, then (3) collect all data and relevant cases, and (4) analyze data/cases by comparing predictions with observations. The revised APSA Ethics Guide lays out data access and production transparency as important points, but both deal only with the third step -- how we generate, code, and source our information. Analytic transparency has to do with the fourth step, by asking us to be more explicit in how we generate defensible conclusions from the data.
In some circles, especially those of us working with experiments, though, there is an additional initiative that covers the first two steps, namely preregistration. Preregistration would, in theory, require all of us to deposit all of our proposed causal hypotheses at the start of any study (if we can even call them that) into some depository, at the very least; then, when the study concludes, we are prevented from going back and changing the hypotheses so that we can market our research as having predicted or tested that proposition all along. In other words, procedure matters more than results. There are obvious reasons why others of us, such as comparative-historical researchers sensitive to complex causation, laugh at this: our finished theoretical contributions (think Moore's classic multi-pathway model of industrialization, democratization, and dictatorship) are the product of exhaustive research that fleshes out the mechanisms, links, and temporalities that we cannot possibly imagine. Virtually no study of historical path-dependence involving complex causal relationships is a study of elegant hypothesis confirmation, with the scholar having hypothesized all those nuanced linkages and complexities from the start.
I doubt any of us have ever actually carried out a multi-year study that resembled, in any way, the step-by-step proceduralism of this research model. But the real question is this: if DA-RT proponents we really want to ensure proper conduct, why no push for preregistration? Did it slip off the agenda at the initial DA-RT meetings starting in 2010? It is like DA-RT proponents are pushing a halfway model of "best practices" that still allow for great malfeasance on part of the researcher when pursuing the very template of practices idealized by DA-RT. If we really want transparency, shouldn't we demand preregistration?
DA-RT is presented with the intention of enhancing transparency in the conduct of empirical political science research. That research (which some may criticize as too quantitative, or hypothetico-deductive, or what not) idealized by this vision is what I have called the "deductive template" in past work -- the 4-step blueprint where (1) we derive axiomatic and falsifiable hypotheses, next (2) predict all the implications of those propositions, then (3) collect all data and relevant cases, and (4) analyze data/cases by comparing predictions with observations. The revised APSA Ethics Guide lays out data access and production transparency as important points, but both deal only with the third step -- how we generate, code, and source our information. Analytic transparency has to do with the fourth step, by asking us to be more explicit in how we generate defensible conclusions from the data.
In some circles, especially those of us working with experiments, though, there is an additional initiative that covers the first two steps, namely preregistration. Preregistration would, in theory, require all of us to deposit all of our proposed causal hypotheses at the start of any study (if we can even call them that) into some depository, at the very least; then, when the study concludes, we are prevented from going back and changing the hypotheses so that we can market our research as having predicted or tested that proposition all along. In other words, procedure matters more than results. There are obvious reasons why others of us, such as comparative-historical researchers sensitive to complex causation, laugh at this: our finished theoretical contributions (think Moore's classic multi-pathway model of industrialization, democratization, and dictatorship) are the product of exhaustive research that fleshes out the mechanisms, links, and temporalities that we cannot possibly imagine. Virtually no study of historical path-dependence involving complex causal relationships is a study of elegant hypothesis confirmation, with the scholar having hypothesized all those nuanced linkages and complexities from the start.
I doubt any of us have ever actually carried out a multi-year study that resembled, in any way, the step-by-step proceduralism of this research model. But the real question is this: if DA-RT proponents we really want to ensure proper conduct, why no push for preregistration? Did it slip off the agenda at the initial DA-RT meetings starting in 2010? It is like DA-RT proponents are pushing a halfway model of "best practices" that still allow for great malfeasance on part of the researcher when pursuing the very template of practices idealized by DA-RT. If we really want transparency, shouldn't we demand preregistration?