The Only Heuristic You'll Ever Need

I don’t know about you, but when the shouting gets shouty, I like to wrap myself in a warm blanket of thoughtful nuance. Fortunately, I have here in front of me a set of six manuscripts that do exactly that, and they are headed your way in the latest special section on improving research practices in the forthcoming September issue of Perspectives on Psychological Science. 

I have talked before about the tendency for humans to love a good cognitive shortcut, and I suspect that cognitive shortcuts act as both antecedents to and consequences of the shouting matches that sometimes erupt in the ongoing conversation on research practices. One of my favorite drinking games these days* is to take a shot every time somebody claims “everyone knows X” or “nobody is arguing Y” or “I don’t think anyone would do Z.” It turns out that this is a prime example of the false consensus effect—a heuristic that leads people to overestimate the extent to which other people share their own beliefs, preferences, and behaviors. We tend to use our own beliefs and behaviors as a guesstimate and generalize from there.

Meanwhile, if I simplify the landscape of perspectives into two sides, I’m more likely to perceive the “other” side as unified, homogeneous, and extreme in their positions, and I contribute in turn to other people’s perceptions that there are only two sides. These and other heuristics tend to sink us further into polarizing arguments and unhelpful finger-pointing, and impede our ability to have constructive discussions, learn from each other, change our own minds, and build consensus.

Moreover, cognitive shortcuts also played a major role in creating the problems with our methods and practices that we are now confronting (p < .05, anyone?). As I note in my introduction to our new special section (available here, in UC’s open access repository, if you’d like a sneak peek): The single most important lesson we can draw from our past in this respect is that we need to think more carefully and more deeply about our methods and our data. Heuristics got us into this mess. Careful thinking will help get us out. The only heuristic you'll ever need in science is this: Don't rely on heuristics. 

And this is why the papers in this special section feel like a warm blanket of thoughtful nuance to me: Together, they highlight the importance of thinking carefully at each phase of the research process, from selecting among multiple possible research strategies, to analyzing one’s data, to aggregating across multiple studies to build a more comprehensive picture of a given topic area.

They hammer home the importance of thinking carefully about tradeoffs when choosing one research strategy over another (e.g., running fewer studies with larger samples or more studies with smaller samples), echoing and building on recent calls to fully consider both the pros and cons of a given research strategy when seeking to design smart changes for one’s own lab or for the field as a whole (see e.g., Finkel, Eastwick, & Reis, in press; Gelman,2013; Ledgerwood, Soderberg, & Sparks, in press). They push us to more carefully examine and transparently communicate the assumptions we make when we analyze our data. And they unpack some of the idealized assumptions underlying various meta-analytic techniques—including p-curve and p-uniform, as well as traditional methods—and show us what happens when those assumptions are violated, as they often are in the real world. (Don’t worry, there’s a better way to do meta-analysis, and the last article in the special section explains how.)

Most importantly, the articles all provide concrete advice both on how we can be more careful and more transparent about the assumptions we make throughout the research process, and on how we can continue to improve our research practices in a thoughtful, smart, and nuanced way. 

So if you’re feeling tired of the shouting, and you’re ready for some nuance, stay tuned: The following articles are coming your way, open access, very shortly.

 

*Just kidding!**

**Or am I?

 

Baby Steps

If you hadn’t noticed by now, I can be indecently peppy. Over at PsychMAP, when my (brilliant and thoughtful) co-moderator gloomily surveys the imperfections of our field and threatens to drown his sorrows in drink, I find myself invariably springing in like Tigger with a positive reframing. We’ve found flaws in our practices? What a chance to improve! An effect didn’t replicate? Science is sciencing! Eeyore’s tail goes missing? Think of all the opportunities!

Every now and then, though, I stare at this field that I love without recognizing it, and I want to sit down in the middle of the floor and cry. Or swear. Or quit and open a very small but somehow solvent restaurant that serves delicious food with fancy wine pairings to a grand total of six people per night.

The thing that gets me to land mid-floor with a heavy and heart-sinking whomp is not discovering imperfections about our past or noting the lurching, baby-giraffe-like gait of our field’s uneven but promising steps toward progress—it’s when the yelling gets so loud, so polarized, so hands-over-my-ears-I-can’t-hear-you-la-la-la that it drowns out everything else. It’s when Chicken Little and the Ostrich have a screaming match. It’s when people stop being able to listen.

I’ve written before about some of the unintended and damaging consequences that this kind of tone can have, and here’s another: Add these loud debates to the shifting standards and policies in our field right now, and the average researcher’s instinct might be, quite reasonably, to freeze. If it’s unclear which way things are going and each side thinks the other is completely and hopelessly wrong about everything, then maybe the best course of action is to keep your head down, continue with business as usual, and wait to see how things shake out. What’s the point of paying attention yet if nobody can agree on anything? Why start trying to change if the optimal endpoint is still up for debate?

If you find yourself thinking something along these lines from time to time, I am here to say (passionately and melodramatically from where I sit contemplating my future prospects as a miniature restauranteur, plopped in the middle of the floor):

There is every reason to start now, and zero reason to wait.

Here’s the thing: We don’t have to agree on exactly where we’re going, and we certainly don’t have to agree on the exact percentage of imperfection in our field, to agree that our current practices aren’t perfect. Of course they’re not perfect—nothing is perfect. We can argue about exactly how imperfect they are, and how to measure imperfection, and those discussions can sometimes be very productive. But they’re not a necessary precondition for positive change.

In fact, all that’s needed for positive change is for each of us to acknowledge that our research practices aren’t perfect, and to identify a step—even a very small, incremental, baby step—that would make them a little better. And then to take the step. Even if it’s the tiniest baby step imaginable in the history of the world. One step. And then to look around for another one.

So for example, a few years ago, my lab took a baby step. We had a lab meeting, which was typical. We talked about a recent set of articles on research practices, which was also typical. But this time, we asked ourselves a new question: In light of what we know now about issues like power, p-hacking, meta-analysis, and new or newly rediscovered tools like sequential analyses, what are some strategies that we could adopt right now to help distinguish between findings that we trust a lot and findings that are more tentative and exploratory?

We made a list. We talked about distinguishing between exploratory and confirmatory analyses. We talked about power and what to do when a power analysis wasn’t possible. We generated some arbitrary heuristics about target sample sizes. We talked about how arbitrary they were. We scratched some things off the list. We added a few more.

We titled our list “Lab Guidelines for Best Practices,” although in retrospect, the right word would have been “Better” rather than “Best.” We put a date at the top. We figured it would evolve (and it did). We decided we didn’t care if it was perfect. It was a step in the right direction. We decided that, starting today, we will follow these guidelines for all new projects.

We created a new form that we called an Experiment Archive Form to guide us. (It evolved, and continues to evolve, along with our guidelines for better practices. Latest version available here.) 

And starting with these initial steps, our research got better. We now get to trust our findings more—to learn more from the work that we do. We go on fewer wild goose chases. We discover cool new moderators. We know when we can count on an effect and when we should be more tentative.

But is there still room for improvement? Surely there is always room for improvement.

So we look around.

You look around, too.

What’s one feasible, positive next step?

---

Some places to look:

•        Braver, Thoemmes, & Rosenthal (2014 PPS): Conducting small-scale, cumulative meta-analyses to get a continually updating sense of your results.

•        Gelman & Loken (2014 American Scientist): Clear, concise discussion of the “garden of forking paths” and the importance of acknowledging when analyses are data-dependent.

•        Judd, Westfall, & Kenny (2012 JPSP): Treating stimuli as a random factor, boosting power in studies that use samples of stimuli.

•        Lakens & Evers (2014 PPS): Practical recommendations to increase the informational values of studies.

•        Ledgerwood, Soderberg, & Sparks (in press chapter): Strategies for calculating and boosting power, distinguishing between exploratory and confirmatory analyses, pros and cons of online samples, when and why to conduct direct, systematic, and conceptual replications.

•        Maner (2014 PPS): Positive steps you can take as a reviewer or editor.

•        PsychMap: A Facebook group for constructive, open-minded, and nuanced conversations about Psychological Methods and Practices.