Defenders of Wildlife just released a poll, conducted by Tulchin Research, on sage grouse conservation. Defenders claims the poll provides evidence that registered voters in Colorado, Nevada, Wyoming and Montana are in favor of two key things: listing the greater sage grouse under the Endangered Species Act if current state and municipal conservation efforts prove inadequate; and that political candidates also support listing if such state and municipal plans are inadequate.
In reality, the poll asks such loaded questions and in an order that “primes” people to respond in a certain way that it is of limited, if any, validity. Let’s take the first question, which asks:
“If current state and local conservation plans are inadequate and fail to protect the species from the threat of extinction, would you support or oppose placing the sage-grouse on the Endangered Species list to prevent it from going extinct?”
In response, 67% of voters support listing, 28% oppose listing, and 6% don’t know.
This response should not be surprising because the question is loaded so heavily in favor of the desired response: listing the sage grouse under the Endangered Species Act. The question casts aspersions on “current state and local conservation plans” by placing this term in close proximity to, and defacto paired with, the unflattering words “inadequate” and “fail.” Meanwhile, the Endangered Species Act is put in a very favorable light because the term “Endangered Species list” is in close proximity to, and defecto paired with, the noble phrase “prevent it from going extinct.”
In the academic study of survey research and polling, this technique is known as “framing.” According to two of the leading experts on polling, Michael Link, currently chief methodologist for research methods at The Nielsen Company, and Robert Oldendick, professor of political science at the University of South Carolina:
“How questions are asked and the response categories provided are crucial to determining results. Common sense tells us that the use of loaded words or the phrasing of a question can affect the pattern of responses to a survey question.”
And as the Pew Research Center points out:
“The choice of words and phrases in a question is critical in expressing the meaning and intent of the question to the respondent and ensuring that all respondents interpret the question the same way. Even small wording differences can substantially affect the answers people provide.”
The second key question in Defenders of Wildlife’s sage grouse poll was asked directly following the question about state and local conservation plans, and it is phrased as:
“Would you be more likely or less likely to vote for a candidate for office who supports protecting the sage-grouse?”
In response, 51% of voters are more likely, while 31% are less likely.
As with the first question, this second question relies on a tried-and-true trick to garner the desired response. In survey research and polling, “priming” is the term used to describe this trick because respondents are primed to give a desired response by the preceding question or questions. Respondents are primed to favor candidates that support protecting the sage grouse (which implicitly means protecting with the Endangered Species Act because of how the preceding question was framed), because the order in which these two questions were placed.
According to Dr. Philip Garland, Vice President of Methodology for Survey Monkey, the popular online survey development company:
“The order questions appear in your survey can directly impact the responses you gather. One of the more well-known examples (Iyengar & Kinder, 1987) of question order effects is in the domain of politics…This phenomenon is called priming. Respondents are primed to think about one issue while answering the subsequent question.”
Again, the Pew Research Center provides useful insight on the issue of the order in which poll questions are asked:
“Once the survey questions are developed, particular attention should be paid to how they are ordered in the questionnaire. The placement of a question can have a greater impact on the result than the particular choice of words used in the question.”
The only thing that can be concluded from Defenders of Wildlife’s highly flawed sage grouse poll is that asking loaded questions in a certain orders will produce desired and predictable results that have little if any validity and objectivity.
Another recent flawed poll on sage grouse conservation was released in July by Pew Charitable Trusts’ Western Lands Initiative. The poll purports to show that large majorities of the public in Oregon, Colorado, Nevada and Montana support protecting the greater sage grouse. In reality, the poll asks such loaded questions in favor of the Bureau of Land Management’s plans to protect sage grouse that very little if anything of value can be concluded. Yet the poll appears valid in part because it was conducted by a Democrat firm, Benenson Strategy Group, and a Republican firm, Public Opinion Strategies.
The upshot of these two flawed polls on sage grouse conservation is that desired responses can easily be obtained by the ways in which questions are phrased, coupled with and the order in which questions are asked. If nothing else, these two polls are good case studies in flawed polling methodology.