Qualitative Transparency Deliberations

[ReginaBateson]

As has been raised in prior discussions, JETS gives journal editors great discretion in deciding whether, and how, to implement transparency and data access requirements. I think we should take a step back and debate whether this role should be assigned to journal editors. Or should other individuals in the profession--like reviewers--serve as gatekeepers and DART-enforcers?

Journal editors do not necessarily have personal experience with particular research methods, not can they be expected to have expert knowledge of all research settings. Reviewers, on the other hand, have presumably done similar types of research, and they should be familiar with the standards and best practices of different subfields and epistemic communities. They should also be much more familiar with the case under study. So why not leave the question of transparency and data access to the reviewers, not editors? When an article goes out for review, either the main text or appendices would address transparency and data access. If the author does not plan to make the data publicly available, he/she would explain and justify that decision. In their reviews, the reviewers could agree or disagree with the author's rationale. This could be further discussed back-and-forth in R&R if necessary, until the reviewers, the editor, and the author settle on a course of action.

Finally, when data will be shared, it would seem to make a good deal of sense for the full data to be shared with the reviewers, not submitted to the journal only after an acceptance. The reviewers are best positioned to evaluate the quality of the data and the reasonableness of the author's interpretations. If we want transparency to lead to increased rigor and accountability, sharing data BEFORE publication seems crucial, because that is when a manuscript faces its most intense evaluation from other experts.

[malahtun]

I agree that editors are not necessarily in the best position to be the gatekeepers, but I'm not sure reviewers are either. off top of my head: this scenario could lead to 1) lead to massive variation in how otherwise similar articles are treated even by same journal depending on who the reviewers are and their particular beliefs and expectations; 2) lead to a potential stalemate when reviewers differ and the author has to answer to many different audiences; 3) involve a coordination problem among reviewers, who for a single submission will differ in their views of the article and what they expect in transparency.
mala

[Guest]

Generally, I subscribe to what could be called epistemological subsidarity, that is, research transparency should be assessed as the most local level possible. Practically, this would mean the scholars most intimately familiar with the subject matter and methodology of the reviewed article. So, this would mean shifting the responsibility of assessing research transparency from editors to reviewers which is what we do currently anyway.
I would like to make one suggestion to strengthen this practice and propose one caveat.
The practice could be strengthened by adding research transparency to the check-list of criteria used to evaluate a submission. Many journals currently ask us to evaluate contributions for their originality, rigor, breadth etc. Why not add research transparency as another criteria and have use rate it. It might encourage reviewers to pay more or more explicit attention to it.
Let suggest a caveat to the broader epistemological subsidarity principle. What if journal editors would require that each submitted article have at least 50% of citations that contain actual page numbers. Page numbers arguably are the less onerous, least controversial research transparency benchmark, yet all too frequently, it is rarely met. Many of us probably have been frustrated by the drive by literature reviews that reduce an entire theoretical debate to five pageless citations or by empirical implications that are supported again by pageless citations. At the risk of over-generalizing, this benchmark might pose a slightly more formidable challenge for qualitative than qualitative submissions.