The big news of the weekend – a poll published in Scotland on Sunday that claimed to show a lead for staying in the union – was altered in a way that could have manufactured that result.
Business for Scotland has now seen the raw data from polling company Savanta ComRes and we can reveal that the poll deliberately omitted information which could have dramatically overturned the result.
Savanta ComRes, who carried out the poll for the Scotsman organisation, excluded a weighting mechanism that changed a previous poll last month from a Yes minority to a majority of 53%.
We spoke exclusively to Savanta ComRes who explained that the question of voting likelihood ”wasn’t asked purely because we knew we wouldn’t end up using it for weighting.”
To be clear, a unionist newspaper commissioned a poll and agreed that it would not be carried out with the normal weighting procedure. It is therefore reasonable to assume that (given the impact of that weighting on their previous poll) they did this knowing that such a move could manufacture a result that would favour the union. Thus they published a poll that did not in anyway break the sequence of 23 polls showing independence support at 50% or above.
Savanta ComRes political research director Chris Hopkins admitted to Business for Scotland today that excluding the information was ‘’unusual’’ and ‘’not ideal’’.
Clearly the circumstances surrounding the poll leave the door open to accusations that information was ignored to make the results look bad for the independence side.
Mr Hopkins denied there was anything ‘’conspiratorial’’ about the decision, pointing to a disclaimer in Scotland on Sunday which said: ‘These figures are not weighted for voter turnout, with further polling expected this week to show a clearer impact of the inquiry on Scottish independence voting intention. These figures on Scottish independence are not directly comparable with previous polls on the subject, due to this and the nature of the poll.’’
When we told Mr Hopkins that not every media outlet that reported the poll mentioned the disclaimer he said: ‘’There is very little I can do about that.’’
He insisted that the ‘likelihood of voting’ weighting had been abandoned, in consultation with the Scotsman, because ‘’we did not have the resources to do it within the timeframe’’.
When Business for Scotland today asked for the result to be republished with the weighting applied Mr Hopkins initially said there were no plans to do so. When pressed he agreed to give the matter more thought. But when the polling company published the raw data today – as it is required to do under British Polling Council rules – it was revealed that respondents had not even been asked about their likelihood of voting.
When the same weighting was applied to a previous Savanta ComRes poll in February which returned results almost identical to the raw data for Sunday’s poll it turned a No lead into a 53% majority for Yes
Mr Hopkins said: ‘’The question wasn’t asked purely because we knew we wouldn’t end up using it for weighting. That will mean it is not possible to retrospectively apply that weight.’’
When the ”likelihood of voting” weighting was applied to a previous Savanta ComRes poll in February which returned results almost identical to the raw data for Sunday’s poll it turned a No lead into a 53% majority for Yes.
In an interview with Business for Scotland today Mr Hopkins said this particular weighting is not generally expected to make a significant impact on a poll’s overall result. But when asked about February’s poll – also for the Scotsman organisation – he confirmed the ‘likelihood of voting’ weighting transformed a No majority into a Yes majority. He said: ‘’We do not normally see this weighting having a major impact but for some reason it did so in that February poll.’’
Of course, if there was one way to make sure the weighting didn’t have the same effect as it had in February it would be not to include it
In the normal course of events the ‘likelihood of voting’ is one of a number of weightings applied to poll results. These include age, sex, region and recollections of past voting in the 2019 general election and the 2014 independence referendum. All those weightings had been applied to Sunday’s poll. In fact the likelihood of voting was the only weighting which was not applied.
Of course, if there was one way to make sure the weighting didn’t have the same effect as it had in February it would be not to include it.
Sunday’s was the first poll on Scottish independence Savanta ComRes had carried out without the weighting applied … and the first to claim that the No vote was leading. Twenty one polls in a row had shown the Yes vote ahead until the 22nd showed the two side were tied.
Mr Hopkins said there had not been enough time to apply the weighting to Sunday’s poll. The company and the client for the poll – The Scotsman organisation – had agreed to press ahead with publication despite knowing in advance that the weighting would not be added.
A poll by Panelbase for the Sunday Times, also carried out after Wednesday, did find the time to apply weighting by filtering out anyone who said they were less than 8/10 likely to vote. We believe the time it takes to do that would could reasonably be estimated to be between 5 and 10 minutes.
When we asked My Hopkins if he understood suspicions given the weighting’s quite dramatic impact on the previous poll he conceded we ‘’had a point’’ and agreed to think about publishing the weighted result and to tell Business for Scotland if he would do so. However, it has now transpired from his statement that republishing was never a possibility.
Sunday’s poll was described as the first to be carried out after both Alex Salmond and Nicola Sturgeon had given evidence to the Holyrood inquiry into the handling of allegations against Mr Salmond. It ‘’suggested’’ a 52% lead for No and 48% for Yes but now we know that result was created by the selected methodology and it does not relate to any other polls asking the same question.
Professor Sir John Curtice, a leading political analyst and the president of the British Polling Council, said that the most recent results were “unchanged from equivalent figures” found by the same polling firm – Savanta ComRes – in early February. He added: “Turnout weighting had a particularly marked impact in that poll.”
Mark Diffley, a former Ipsos MORI director who now runs The Diffley Partnership, said: “I don’t get why it wasn’t filtered for turnout – it’s straightforward to do and ‘further polling out this week’ does not seem to be a reason not to do it.”
Pollster Mark McGeoghegan said: “You cannot take the difference between two polls to be reflective of real-world change, never mind assign a reason to that change, if you’ve changed the weighting. Don’t compare incomparable polls.”
Business for Scotland Chief Executive Gordon MacIntyre-Kemp who regularly commissions polling, said “The problem with that statement from Scotland on Sunday about further analysis being done on the data is that it makes absolutely zero sense. If they did not ask the intention to vote at the time of the poll it is just not possible to retrospectively weight the data. So no further analysis can show a clearer picture.
Mr MacIntyre-Kemp also said the raw polling data throws up as many questions as it answers. ”The first thing that jumped off the page at me was that Savanta ComRes interviewed 1,015 people but the independence question – although labelled “BASE: All respondents” – only has a base of 845 respondents. That’s not far off what I’d expect if they had actually applied a ‘likelihood to vote’ filter. If they didn’t, as they now claim, the base number should be far higher.
”Looking back at their previous poll last month they applied an intention to vote and that left them with 885 from their original 1,002 respondents in that poll. At that point, they had pretty much the same result as the poll published yesterday – Yes 43, No 46, DK 11. Then they applied their turnout weight/ model and that turned it into a Yes lead at 53%
”I can’t prove that they did ask the question and just left it out of the data they published but it looks highly suspicious. As a result we will be writing to the British Polling Council and asking them to investigate if Savanta ComRes has just published incomplete data that contradicts their statement that they did not ask the question, which in turn contradicts the earlier statement that suggested they did ask for intention to vote but just didn’t have time to apply the weighting.”
Oh what a tangled web we weave….