tag:blogger.com,1999:blog-7803121573679180423.post8149797514866140119..comments2022-03-24T16:31:47.356-07:00Comments on Incurably Nuanced: Preregistration is a Hot Mess. And You Should Probably Do It.Unknownnoreply@blogger.comBlogger4125tag:blogger.com,1999:blog-7803121573679180423.post-42767889620752500212018-03-11T09:43:15.223-07:002018-03-11T09:43:15.223-07:00I think we need not just a reboot on the vocab but...I think we need not just a reboot on the vocab but also some careful thinking about the different problems that we're hoping preregistration can help us solve. One point I was trying to make is that because different people have used "preregistration" to mean different things over the last few years, we've lumped together all the different problems that these different things can solve. So now, we assume that preregistration almost automatically confers a whole range of benefits, from combating publication bias (which was touted as a benefit of "preregistration" meaning registered reports, but isn't very connected to "preregistration" meaning uploading a document with some predictions in it so one or another repository), to differentiating studies that test a priori directional predictions from studies that ask open-ended research questions, to differentiating analyses that were data-independent from analyses that were data-dependent.<br /><br />I agree that RRs may sometimes or even often fail to anticipate all relevant researcher dfs. But I would predict that articles published via a registered report mechanism are more likely than articles published as a regular journal article with a preregistered study to transparently and clearly disclose all deviations from the preregistered analysis plan. In other words, I suspect that because the RR format usually requires researchers to write up their planned methods section and planned analyses and then to disclose any and all changes to the editor and potential reviewers, there is more accountability built into that system, and authors are more likely to be careful about clarifying any necessary changes that emerged to the analysis plan. (Of course, this prediction of mine could and arguably should be tested empirically!)Alisonhttps://www.blogger.com/profile/03499440763063735416noreply@blogger.comtag:blogger.com,1999:blog-7803121573679180423.post-22945794324815756442018-03-11T09:33:29.796-07:002018-03-11T09:33:29.796-07:00Thanks, David...I agree that "what's the ...Thanks, David...I agree that "what's the best way forward?" is exactly the question we need to be asking -- how do we clarify our language to reduce confusion, but also how do we make sure that we're distinguishing between the different potential goals that preregistration can serve, and how do we design effective strategies for creating and reviewing preregistrations that ensure we are actually getting the benefits we want from them? It would be great to collaborate with COS on possible ways to move in that direction!Alisonhttps://www.blogger.com/profile/03499440763063735416noreply@blogger.comtag:blogger.com,1999:blog-7803121573679180423.post-77881200883471587242018-03-08T10:10:37.702-08:002018-03-08T10:10:37.702-08:00Good points. It would be terrific if wise folks di...Good points. It would be terrific if wise folks did a reboot on the vocab in this domain. <br /><br />Psych Science recently changed to a system in which the action editor vets the author's badge requests, including for the prereg badge. I hope that will contribute to improving the quality of the preregs associated with articles that get that badge. But just as important is ensuring that researchers have a good grasp on the purpose of preregistration (to differentiate exploration from hypothesis testing).<br /><br />BTW, I would not be surprised if even RRs sometimes fail to anticipate all relevant researcher degrees of freedom.<br /><br />Steve Anonymoushttps://www.blogger.com/profile/07485890516098106807noreply@blogger.comtag:blogger.com,1999:blog-7803121573679180423.post-756540481787669002018-03-08T07:23:24.074-08:002018-03-08T07:23:24.074-08:00Hi Alison. Thanks for leading this discussion abou...Hi Alison. Thanks for leading this discussion about the risks of fractured communication. I know this is being discussed on facebook and twitter too, I wonder what the best way forward is to help establish shared vocabulary and reduce confusion, so that expectations can be reasonably managed. All of the points you made are spot on, and there is even more confusion in some areas (for example, some Registered Reports don't have an associated preregistration, some preregistrations are not actually in a registry or a functional equivalent of one, and on and on!). We're happy to help amplify your message and want to do what you think is necessary to meet the goal of increasing clarity (and by extension, rigor!).Anonymoushttps://www.blogger.com/profile/17828418862149203477noreply@blogger.com