To participate, you may either post a contribution to an existing discussion by selecting the thread for that topic (and then click on "Post Reply") or start a new thread by clicking on "New Topic" below.
The transition to Stage 2 of the deliberations is currently underway but will take some time to complete. In the meantime, we very much welcome additional contributions to the existing threads in this forum.
For instructions on how to follow a discussion thread by email, click here.
https://ajps.org/2016/05/10/ajps-to-awa ... ce-badges/
the badges are presented by the writer of the article just as a visual cue to easily identify what paper has passed their replication procedures and uploaded the data and code.
Here is the piece of the article that talks about exceptions:
"Of course, some articles published in the American Journal of Political Science will not receive the Badges. For example, many formal theory manuscripts and virtually all of the normative theory manuscripts that are submitted to the Journal do not contain any empirical analyses. Such work is exempt from the AJPS Replication Policy, so the Open Practice Badges are not relevant to these manuscripts. And there are certain situations in which a manuscript may be given an exemption from the usual replication requirements due to the use of restricted-access data. In such cases, authors still are asked to explain how interested researchers could gain access to the data and to provide all relevant software code and documentation for replicating their analyses. Manuscripts in this situation would not receive the Open Data Badge, but they would be awarded the Open Materials Badge. Even with allowances for exceptions, we anticipate that the vast majority of the articles published in the American Journal of Political Science will receive both Badges."
I wonder if they thought that some of the really interesting types of qualitative research that have been discussed in QTD might not get these badges... I guess they might be added to exceptions.
But more generally this type of gamification device is a paternalistic carrot/stick weak incentive structure.
I am open to see how it will affect the journal, and ready to accept that these incentives work and will improve the rate of replication/transparency in quantitative research, but part of me also resists to these types of approaches.
My hunch is that it will be useless for quantitative research, the only outcome will be stigmatizing the exceptions. But I am ready to be proven wrong.
- Posts: 26
- Joined: Sat Apr 16, 2016 9:48 am
While I share your concerns, the idea of badges is intriguing if the could be earned for more than just data access and replication. Let's assume that you could earn badges for the type of research transparency that is more relevant to qualitative research, they could convey important information and become tools for gradually establishing new transparency norms without enforcing them from upon high. Take a simple example, let's assume that you could earn a citation badge for having more than 70% of your citations including and actual page number. Or you could earn a badge for active citation. Such badges would provide small carrots to improve research transparency. Bottom line, badges are not the problem but what research activities can earn them is.
- Posts: 11
- Joined: Wed Apr 06, 2016 8:24 pm
But the discussion about top down incentive structure and choice architectures is always tricky. There is a growing debate around these gamification approaches and many view these incentive structures as the dark side of gamification and contrast it with playfulness (see for example all the interesting work of Eric Gordon). I am still on the fence.
Setting aside philosophy and freedom, from a purely instrumental standpoint there are plenty of examples in which incentives backfired.
For example project repurpose of AFL-CIO was born because during the first Obama campaign they a/b tested a badge scheme and discovered that it reduced the motivation of people. Instead repurpose substituted the badges with micro-empowerment. Participants earned points by canvassing that could be spent in targeting the effort of the campaign itself instead of getting a meaningless badge.
[link to repurpose: http://repurpose.workersvoice.org/]
Successful implementation of these schemes more and more use the micro-empowerment approach (for example stack overflow increases the administrative privileges of participants that perform community enhancing activities).
Not sure if there is a way to apply the micro-empowerment approach to journals. Probably in the review track there is, creating categories of super-reviewers that get additional editorial privileges the more they review. In the publication track I am not sure. Maybe the possibility of getting additional word count could be a proper incentive. But all these things need very careful field testing because they have the habit of creating unintended consequences as the infamous Haifa experiment on fines for parents that arrived late to pick up kids at school showed.
And why instead not introducing more participatory decision making from readers/reviewers/authors in journals?
Why for example discussions like this one cannot be promoted among the readers/authors/reviewers of CPS to define their rules?
Maybe we need more deliberation instead off nudges/incentives...
- Posts: 11
- Joined: Wed Apr 06, 2016 8:24 pm
http://fivethirtyeight.com/features/eve ... 538twitter
Univ of New Mexico
- Posts: 16
- Joined: Wed Apr 06, 2016 9:20 am
Marcus' suggestion of something as simple as requiring more citations with page numbers--- and applying this across qualitative and quantitative work--- actually makes a lot of sense to me. I often read papers where I feel like authors make hand-waves to citations that they have read very superficially at best (else they would have shown a little more nuance in their characterization of a given argument). Taken in moderation, this strategy seems a fairly low-cost way to improve the quality of scholarship. But of course, as with everything else in life, I wouldn't want to see this strategy taken too far. Put more concretely, I'm not sure it's really worth all the extra time/energy to figure out where, precisely, Barrington Moore wrote "no bourgeosie, no democracy."