Research is important but it’s not necessarily accurate

Rumpydog posted yesterday about a study comparing dog people to cat people. I was intrigued, but also (as usual) prepared to be hypercritical of research assumptions and methodologies, and in particular the “commonsense” taken-for-granted beliefs that shape the initial formulations of hypotheses. I love good research and an armful of endnotes, but part of what I love about it is looking for flaws. (Although when you point out my typos I will be embarrassed and probably need a bunch of cookies for comfort.)

Anyway, I read the study and then posted a little rant on poor Rumpydog’s site:

Okay, that was fascinating. And I love that they studied this! BUT I think they are relying on some notions about dogs and cats that would not necessarily hold up to scrutiny (they talk about dominance in canine social groups as if it a function of individuals rather than being fluid across situations, and refer to dogs as “deferential” and “submissive” when these are also qualities of individuals somewhat correlated to breed).

Also, they fail (as far as I saw on my quick read) to take lifestyle factors into account, which is a major fact0r in whether one considers cats or dogs to be more appropriate as (potential) companions.

Also, they treat gender as a binary rather than a spectrum, which means that some of their conclusions are iffy. For example, one conclusion is that there is a robust difference between cat and dog owners in terms of both SDO and competitiveness, while a few paragraphs previous to that, they clearly state that these differences are not necessarily as clear within genders. (They do say this could also be because of the small sample size, which is totally understandable.)

It was interesting that they eliminated the “both / neither cat and dog people” from the study. I’m a “both,” so I can’t really speak to one side or the other. But I have a longstanding (and formerly professional) interest in canine behaviour, so I find that the assumptions about dominance, submission, and obedience always strike me.

It’s interesting that they are positing “dominant human / submissive dog” as the other side of “less dominant (submissive?) human / independent cat.” I think some of the preliminary assumptions are not well-founded, that their understanding of gender means their discussion is not entirely credible, and that their initial hypotheses should have been more thoroughly examined / discussed.

On the other hand, cat people, dog people, cat/dog people, and no-pet people all have stereotypical ideas about their own group and the other groups, so it was pretty cool to read this study and see how people are approaching the questions! Thanks for sharing it and letting me have my morning rant. 🙂

I was feeling kind of like the bitchy auntie, but then I came across this as I ate my brekkie today: Religious Kids are More Selfish and Sadistic, According to Science (the summary of the actual research can be found here). There’s a brief discussion of a study at the top of the article, but where it gets interesting is further down, where the author breaks down the study and critiques various aspects, such as the lack of clearly defined definitions of some of the research terms (like “religion”), failure to control for cultural non-religious factors, failure to control for how religion is itself a cultural factor, and so forth.

I was so excited! I was like “hey! that was me yesterday! only this person is way more articulate!” It makes me feel really good to see people engage critically with research—because that’s how it gets done better next time.

Here is one of my all-time favourite infographics from Compound Interest: A Rough Guide to Spotting Bad Science. I hope you enjoy it as much as I do! I have it bookmarked and keep going back to it when a study irritates me and I can’t quite figure out why. It reminds of some of the major areas to examine when trying to figure out how credible the research and conclusions are.

Advertisements

2 thoughts on “Research is important but it’s not necessarily accurate

  1. La Quemada

    As a (social science) researcher, I love this. It’s so great to have people bring a critical eye to research, especially to the way research gets picked up in the media. I like the infographic too. The one thing I would add is that peer review (#12 on the Rough Guide) is not always a guarantee of high quality. I have seen so many cases where two reviewers look at the same article and have completely opposite positions about its quality. So even among researchers, at least in social/educational sciences, there is actually still a lot of misunderstanding about research design and the definition of variables (such as “religion” or “submissiveness”).

    Liked by 1 person

    Reply
    1. nissetje Post author

      Totally agree. I don’t think there’s a single item on that list that would guarantee the research is good, but as a gestalt, it’s a pretty helpful guide to evaluating what we read, especially online, and especially since it seems like most people don’t go to the original research to make a decision but rather take a news article’s word for it.

      Like

      Reply

What do you think about this?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s