Poll Accuracy and Lessons from 2006

The federal election polls are all over the place, prompting us to wonder which is the most accurate. Based on the last federal election, the best pollster is Nik Nanos, now at CPACE-Nanos, then with SES. However, there's a lot more to polling than a simple metric. Here are excerpts from an article that analysed the 2006 election polls (with a fair bit of paraphrasing to make it easier to read):

In 2006, the biggest winner was clearly SES Research, whose final nightly tracking poll for CPAC on January 22 had all four parties’ popular vote within one-tenth of a percentage point of the outcome.

The big losers in 2006 were the Strategic Counsel and Ipsos Reid, which both missed the Liberal rebound in the closing four days of the campaign.

Nanos/SES has a policy of not doing seat projections, because there is no established formula in the polling industry to make accurate predictions. There are accepted standards when people do polls. There’s more of an art to doing seat projections.

Outside of internal polls, the most accurate seat prediction models during the 2006 election were the ones that did a provincial breakdown, such as Democraticspace.com.

On January 17, 2006, Strategic Counsel showed the Conservatives at 42 percent support nationally, with the Liberals trailing badly at 24 percent — blowout numbers. In contrast, the Nanos/SES poll from the same day had the Tories at 36.6 percent and the Liberals at 31.5 percent [which was accurate].

Nanos/SES eliminated undecided voters from his outcomes rather than redistributing them.

The Strategic Counsel opening question (which asked respondents which party has the most momentum toward a federal election) could have created a pro-Conservative bias in subsequent answers.

The Strategic Counsel has a preference for placing the ballot question later in the interview while Nanos/SES places the ballot question near the beginning. In 2006, Nanos/SES was the only pollster that asked an open-ended ballot question without prompting for parties or party leaders. This allows Canadians to verbalize their choice on their own voting preference as opposed to choosing from a list.

Prime Minister Martin’s last-minute efforts to negatively attack Stephen Harper during the end of the 2006 election campaign created a small upswing for the Liberals.

The question, ‘Do you support a change in government?’ underestimated Liberal support, because the people who answered ‘yes’ to that question would be hypocritical if they had just told the pollster that they wanted change, and then said that they would vote Liberal, so the number of people who will answer ‘Liberal’ declines.

Turnout affected the Liberal numbers. The more people who voted, the better the Liberals did.

The reason the seat projections weren’t all that great in Ontario was that they were applying Ontario-wide numbers to rural seats, which underestimates the Conservative vote, and to urban seats, which underestimates the Liberal vote.

Thirty percent of NDP voters said the thought of a Conservative majority would cause them to reconsider their vote.

The inconsistency across polls throughout the campaign seemed to apply more to the Liberal numbers than the Conservative numbers. This is a product of very high levels of ambiguity amongst conditional Liberal voters who were torn between censuring the Liberals and fear of the Conservatives and what they might bring.

Although people tend to prefer larger samples, it only marginally increases the accuracy. Larger samples are more important as a tool to improve the accuracy of sub-samples or regions. Factors such as question, wording, question order, and sample design have a greater impact on the accuracy of research. People tend to place less weight on smaller survey samples but in the last two elections, Nanos/SES, which had the smallest samples for both, was the most accurate.

Update: Methodology explains higher support for Liberals in Nanos poll