A lot of jingoistic nonsense has sprung up since 9/11, in particular the belief that Muslims are somehow inherently more violent or supportive of terrorism that other populations. This, of course, is nonsense, and I'll show you why in a moment. But I also want to use this opportunity to make a wider point about the use of opinion polls as a pseudoscience tactic, completely with some interesting examples and a funny video!
Thanks to 9/11 and the subsequent conflicts in Afghanistan and Iraq, the internet is now full of "islamophobes" venting about the "violence of Muslims", and pushing for all sorts of extreme measures, from banning Muslim immigration to nuking Iran. Central to their reasoning is the idea that Muslims fanatically hate the West, that Islam is a violent, aggressive religion bent on world domination, and that Muslims support terrorism.
Some perceived support for these views comes from opinion polls such as those of the Pew Global Attitudes Project, which suggest that support for "terrorism" is widespread in many Muslim countries - 44% in Pakistan, 58% in Lebanon, 88% in Jordan and so on. Now, on the face of it, those figures look pretty damning, but there are three major problems with coming to a conclusion from such a poll: control, interpretive bias and creativity.
- - - - - - - - - - - -
Control:
Let's suppose I want to test a new drug to see if it's clinically effective. I could give the drug to a number of patients and see if they get better, but then how do I know that they wouldn't get better anyway? This null hypothesis - that patients might get better anyway - needs to be tested by creating a control group.
Likewise, the poll results seem to indicate strong Muslim support for terrorism, but is this any different than the sort of support we'd expect to see in non-Muslims? Well as it happens, no it isn't, as even the Christian Science Monitor (now there's a website I wasn't expecting to quote) point out that: "Only 46 percent of Americans think that "bombing and other attacks intentionally aimed at civilians" are 'never justified,' while 24 percent believe these attacks are 'often or sometimes justified.'" Those figures were researched by Terror Free Tomorrow, which you may be surprised to know is a right-wing outfit strongly supported by John McCain.
So 54% of Americans feel that attacks on civilians - terrorism - can be justified, and around a quarter feel that bombing civilians is often or sometimes justified. That falls neatly inside the range of opinion from most Muslim countries, and in fact on paper makes Americans considerably more violent than Muslims from Indonesia or Al-Qaida's current home, Pakistan (respectively, 74 and 86% of respondents agreed that terrorist attacks are "never justified").
So once we add a control group - Americans - the picture radically changes. We still don't have the full picture - we'd need to include many other nations for a start - but we can immediately see that removing or ignoring the control group leads to a very skewed interpretation of the results.
Interpretive Bias:
In science, and in life generally, the answers you get are only as good as the questions you ask. This is particularly true in polls. Here's a question put to Muslims around the world in 2005 by the Pew Research Center, which they described as "The most basic measure of support for terrorism": "Some people think that suicide bombing and other forms of violence against civilian targets are justified in order to defend Islam from its enemies. Other people believe that, no matter what the reason, this kind of violence is never justified. Do you personally feel that this kind of violence is often justified to defend Islam, sometimes justified, rarely justified, or never justified?"
Picking out Pakistan as a fairly typical example, 25% said "often justified", and 19% said "sometimes". Now, you could (and people have) spin that as "44% of Pakistanis support terrorism", but look again at the question, and in particular the use of certain key-words - "justified", "defend", and "enemies". Do you think that bombing civilians could sometimes be justified if it was carried out in defense of, say, democracy? Or your nation? In order to defend yourself from your enemies? Is this really "support for terrorism", or is it just a case of people being willing to accept a degree of violence in defense of their way of life?
We can address support for attacks on U.S. troops similarly - if America launched an unprovoked, apparently imperialist war on a European nation, then began the systematic exploitation of that nation's resources for the benefit of its own economy, there is no doubt that many Europeans would support the use of violence. Now, it doesn't matter if you don't think that's why America went to war - the fact of the matter is that many Muslims (and others) do, and so within that context the poll results don't demonstrate an extremist, violent tendency as much as they do a natural instinct to defend one's territory.
Now, I'm not suggesting that the people of Pew intentionally biased their poll. The fact is, that it's almost impossible to phrase a question in a way that doesn't include some sort of innate bias. The key thing is not to lose the relationship between what the question says, and your conclusion. If the first thing you should ask in a poll is "where is the control", the second is "what does the question actually ask?"
Creativity:
The best illustration of pollster "creativity" is the following clip from the British sitcom Yes Minister, in which Sir Humphrey runs rings around a bemused Bernard while demonstrating how to conduct an opinion poll. The genius of this exchange is that even though you can see exactly where Sir Humphrey is going, you can't help but be dragged along towards his conclusion.
Back in 2003, Chris Mooney wrote an excellent and strongly-worded article criticizing American polling guru John Zogby, and the lack of critical thinking shown by journalists who mindlessly repeat incredible poll results. Referring to a poll conducted on behalf of an animal rights group, which claimed over half of Americans supported equal rights for chimps, Mooney commented, "by hiring the renowned pollster John Zogby, the group had essentially purchased an objective fact, one that entered into the conventional wisdom via the nation's leading Sunday magazine."
Zogby, and pollsters like him, have a history of being sponsored to conduct polls which then have a tendency to produce a desirable result. As Mooney points out, "what these polls have in common is that they reveal "findings" that their sponsors wish the public to believe as facts. And Zogby's standing as a reputable pollster buys instant credibility." This represents a paradigm shift in the very concept of a poll - originally created in order to find out about public opinion, they have been converted into a tool for manipulating it. An uncritical media then laps up the results for quick and easy stories.
That's not to suggest that the results are faked. Rather, the polls are creatively designed in such a way that people are directed towards the "correct" answer. In the case of the Chimpanzee Rights Poll, it was reported that 51% of Americans believed that primates should be entitled to the same rights as human children. Chris Mooney describes the actual options presented in the poll: "In brief, they could say that chimps ought to be treated 'like property,' 'similar to children,' 'the same as adults' or 'not sure.'" Given those options, it's unsuprising that 51% of people were steered into answering 'similar to children', but notice that even now, the poll results ("51% of people believe chimps should be treated in a manner similar to children") emphatically do not match the press release ("51% of people believe primates should have the same rights as human children").
Carl, in the comments, points out that creativity doesn't just apply to the questions, but can also be used in audience selection. Opinion polls should, in theory, select audiences representative of the general population, and a considerable bias could be introduced by selecting the audience. Magazine and internet polls are likely to be particularly bad at this.
- - - - - - - - - - - -
Now, opinion polls can be very useful for genuine research, but the polling industry has been corrupted by lobbyists, and pollsters like Zogby that are happy to sell the "right" results. We shouldn't dismiss all opinion polls, but there are four basic questions worth asking yourself when you see one (if anyone has any other suggestions, I'd be happy to hear them in the comments):
1) Where is the control? Am I seeing the full picture? If the poll is talking about the attitudes of one group, or was conducted in one time frame, do we know what the results of other groups or other time frames are?
2) What is actually being asked? Does the reported conclusion match the question? Does the wording in the press release match the wording in the poll?
3) Is there a bias? How is it being asked? Are the questions loaded? Are the available answers adequate and unbiased? Who commissioned the poll, and what are their motives?
4) Who is being asked? Is the survey sample representative of the general population? (Hat tip to Carl in the comments)
Returning to the example with which I started this little piece, I'll leave you with the following quote from the CSM: "In truth, the common enemy is violence and terrorism, not Muslims any more than Christians or Jews. Whether recruits to violent causes join gangs in Los Angeles or terrorist cells in Lahore, the enemy is the violence they exalt."
That is all. If you find this useful, please pass it on.
http://layscience.net/trackback/167
A good, thought-provoking post, Martin. I applaud you and your cognitive abilities.
A very good article. A few additional points: Although I agree that one should consider who commissioned the poll, and their possible motives (one cannot be SURE of motives without inside information), one needs to be very careful. For instance, if a poll concludes that a majority of those polled believe B, it would be incorrect for me to dismiss the poll purely on the known fact that the organization that commissioned the poll believes B. The fact that an organization perhaps goes to some lengths disseminating a supportive poll doesn't mean the poll is necessarily flawed, though it certainly could be.
The methodology perhaps most ripe for manipulation, because it is not an exact science, is the selection of a representative sample of the population whose viewpoints are to be determined. It's all-too easy for a disreputable polling firm to steer the results in the way the client desires, merely by subtly biasing the selection process towards people who are more likely to support it than the population it's deemed to represent. This is polling's fatal weakness, for if skillfully done (and I believe it often is), it's almost impossible to detect.
And then there's push-polling, in which many people are asked a few brief, highly negative and often misleading questions in order to influence the people being polled, typically in an election campaign.
A classic example occurred during the 2000 Republican presidential primary, in which large numbers of people were asked, "Would you be more likely or less likely to vote for John McCain for president if you knew he had fathered an illegitimate black child?" The implication is that McCain did indeed father such a child, otherwise the question is pointless, right? Well, no. McCain did not father an illegitimate black child (and TECHNICALLY the question doesn't claim that he did), but the person being "polled" was clearly supposed to believe that he did, and thus conclude that George W. Bush is the better candidate.
Push poll results are often, perhaps usually, not published, as merely asking the loaded questions is enough. However, sometimes they are. Signs of a push-poll, other than the highly negative, possibly loaded questions themselves, are a very large sample, and a very few short questions (in order to "poll" as many people as possible).
As your article implies, polling done for public consumption is a minefield, and the consumer should tread warily.
Yes, you're definitely right that you shouldn't dismiss a poll (or any piece of evidence) purely because of the sources. It's just something that people should be aware of.
I like the McCain example. Another good example of push-polling would be the old practice of people publishing exit polls while elections are in progress. Push-polling is something I didn't explicitly point out above, but it comes neatly under question bias.
Selection bias is an interesting one. I think it applies a lot more to things like Magazine polls (e.g. 84% of Guns Weekly readers support relaxing gun legislation), or to polls where a large component of the voting comes from the internet (which I suspect has a slight left-wing bias due to the younger age profile), and to any polls where the respondents are self-selecting. I'm actually not sure it happens much in public opinion surveys though, partly because for a lot of their "paid surveys" it's probably easier to just load the questions. I'll make a note of it though.
Martin is the editor of layscience.net.
Follow Me!
RSS | Twitter
Very interesting post, but I have a couple of worries.
Firstly, I'm not so sure this article should have the BPSDB icon on
it: there is a big difference between bad science and pseudoscience
and, granting that all your criticisms are correct, what you describe
is bad science, not pseudoscience.
Secondly, there is clearly a problem in your "control" section,
specifically with your definition of "terrorism". I'm an ex pol sci
freak who did quite a bit of research into Islam and violence, and
I've never seen any peer-reviewed paper or serious commentator define
terrorism simply as 'attacks on civilians'. The RAND Terrorist
Incident Database, for example, uses the following definition: "For
the purpose of this database, terrorism is defined by the nature of
the act, not by the identity of the perpetrators. Terrorism is
violence calculated to create an atmosphere of fear and alarm to
coerce others into actions they would not otherwise undertake, or
refrain from actions they desired to take. Acts of terrorism are
generally directed against civilian targets. The motives of all
terrorists are political, and terrorist actions are generally carried
out in a way that will achieve maximum publicity." Of course, whether
or not your comparison in the 'control' section holds up or not
depends on the exact questions asked in the different polls but there
seems to be a reason to worry here.
Lastly, while it's clearly absurd to think Muslims are somehow
'naturally violent' or somesuch, it's possible to posit a
cultural-historical mechanism for a relatively greater incidence of
violence for Muslims. It's been a while since I analyzed the Rand
date, but, while Muslims are *not* uniquely prone to use terrorism, a
very high proportion of all the terrorist incidents in the last couple
of decades has been perpetrated by explicitly Islamist groups.
Regarding the BPSDB link, well while it's not about a pseudo-science case in particular, it is about the sorts of tactics that many of the people covered in BPSDB used, so I thought why not bung it in. Given that the BPSDB project seems to be gradually fading to death anyway, I'm sure it's not a big deal, but point noted.
The definition of terrorism varies. The point is that really, the questions being asked of Muslims are no different than those being asked of Americans - do you condone the use of violence against civilian targets in defense of your land and faith. Of course, you're right that the definition of terrorism could dramatically change the way you interpret the poll, but that's kind of my point. Certainly in my view the RAND definition covers what American and Muslim respondents talk about in the poll. For example, the bombing of a target like (hypothetically) Al Jazeera would be an act of terrorism. Hell, Hiroshima was an act of terrorism by that standard.
On the last point, one of the biggest problems I have with all of this stuff is that people don't seem to distinguish between different branches of Islam. To refer to all Muslims in response to violence associated with the global salafi jihad movement is like using the IRA as an example of all Christian Certainly a lot of terrorism is coming from Muslim nations, but then a lot has come from the West too over the last 300 years. Ultimately, I think it's more sensible to talk about the causes of terrorism and fundamentalism in general.
PS- Let me know if you want me to amend the name or add a URL to the comment above.
Martin is the editor of layscience.net.
Follow Me!
RSS | Twitter