No-one likes getting it wrong, least of all this blogger. So although we called the UK General Election right in terms of the winner, who would win the most votes and most seats, and the Scottish National Party landslide, a Conservative overall majority was just outside the bell curve of our possible results.
We apologise for this error. But when the basic data's wrong, what can you do?
Pick up on warnings, that's what. Of which there were plenty. Engage our political, historical, qualitative analysis - and our human antennae, not just our statistical training. The storm of numbers helped distract us from our gut instinct, which always had Ed Miliband - student politics, EdStone and all - completely and utterly failing the blink test of all true leadership. This involves asking: do you think, in your heart of hearts, that this is the team you want to lead you and the country for five years?
Well, no.
So now it's time to assemble some initial reflections - to analyse, to dissect and to understand. What else are academics for? This will involve a lot of hard work and heartsearching over the next year or so leading up to the Scottish, Welsh and London elections due in 2016, but a basic picture is beginning to emerge. What we can do for now is compare and contrast the warnings we should have heeded about the reliability of opinion polling with some of what we do know about the actual results.
In this vein, we offer you five reasons for the unexpectedly easy Conservative victory.
1. The SNP. This was the main near-term tactical triumph of the Conservatives' campaign. Those of us who've had our ear to the ground in England heard it all the time in person, and we quote: 'I can't stand that woman [SNP leader Nicola Sturgeon]. She's a new Mrs Thatcher'; 'the Scots are going to come and nick all our money'; 'I don't want to be talked down to by a load of Marxists'; 'why should we dance to their tune?'; 'if they hate us so much, why didn't they just leave when they had the chance?' It was the issue that cut through on the doorstep and ran right through the qualitative data as if it were a stick of seaside rock, the issue that made it through to the noise from the London news debate to ordinary people. It worked a treat, channelling the dark energy of Conservative strategist Lynton Crosby's fears and hatreds to a tee. He's done it before in Australia, of course, lambasting and demonising imaginary refugees who were supposed to have thrown their children into the sea, and helping the Conservatives towards respectability in 2005 with his 'are you thinking what we're thinking?' sloganeering. This time, imaginary hordes of thieving Scots played the same role. It was a stroke of nasty, manipulative, cynical and grotesque... genius. Labour's amateur hour herbivores didn't have a clue what hit them.
Strangely - and full marks again to Crosby for noticing this - this message worked best in the Conservative-Liberal Democrat marginals, frightening off middle-of-the-road English voters who were scared of the SNP dictating its own terms to a Labour government. The Liberal Democrats' complete and utter humiliation was the key to the Conservative majority. If Mr Cameron's party had only picked up the expected ten to fifteen Liberal Democrat seats, there'd now probably only be a minority government. What they did - and it appears that eviscerating their 'colleagues' was always part of the plan, even back in 2010 - was say to Liberal Democrat voters: 'you don't have the luxury of this choice, you have to save the country from the madness of the Celtic Fringe'. So they talked about the SNP in Cornwall and Somerset, about as far from the epicentre of Glasgow's electoral earthquake as you can get and still be in the British Isles. Disgraceful stuff, and a totally false prospectus: but also very, very effective.
2. Shy Tories. Hats off to the Number Cruncher Politics blog on this one, who saw it all with such clarity that we'll have its URL on permanent auto-update next time. This answer to our many conundrums reminds us of the Scottish independence referendum, which the polls also had neck-and-neck in the week before polling. 'No' won by 10.6%. That contest was decided by the 'shy noes' - people who would never consider putting a poster up in their window, but were determined to thwart the ambitions of what they saw as 'radical Glasgow'. The same types turned up last week to thwart the prospect of what to seemed to them like an unacceptably left-wing government that threatened to undermine the very bases of economic recovery: budgetary retrenchment, welfare reform and political stability. Remember, here, that many voters find it difficult to admit even to themselves, and to their close kith and kin, that they are going to vote Conservative: each voter is at war within themselves, and may not actually be dissembling when they refuse to answer pollsters' questions (or act differently to their expressed intention).
One very noticeable feature in this election was that many marginals in Middle England moved away from Labour, even as that party did a little better in England's North West and in London (hello, Lancaster and Ealing Central). Middle class families who'd always voted Labour simply switched to the Conservatives to avoid a left-wing combination of Mr Miliband's party and the SNP. These people probably considered voting Labour, or thought that they probably would, but swerved away in the last few weeks. There is a hint of late swing in a few pieces of the data - for instance, Lord Ashcroft's on-the-day poll (opens as PDF) which saw the Conservatives leading by four points, a gain from the tie that he had reported just before the voting. But that's not the main, or even a big, reason why the polls got it all so wrong. Most of us knew the shy Tories were in there, and we could easily read off the errors that polls had made in the European and by-elections over the past couple of years: but this was beyond even those (three to five per cent) errors. We thought that a bigger electorate would mean that the mistakes were ironed out. We were wrong, because the 'shy Tories' weren't just dyed-in-the-wool Conservative voters who didn't want to admit their choice to us: they were often 'Labour people' who were going over en masse to the Conservatives. That alone should chill Labour hearts over the ten to fifteen years of Opposition that they may well now endure.
3. Pollsters' herding. Pollsters knew that they faced an enormous challenge this time. They were looking at a more fluid electorate, and a more regionalised, as well as socially fragmented, battleground. How to construct the samples? How to weight them by demographics, and past voting choice? It was a nightmare - and one that caught up with them on election night, when Labour's vote appeared to splinter all over the place - to the SNP, the Greens, to Plaid Cymru, and even to the Conservatives. Actually, it's becoming clear that Labour was always behind given the composition of the likely electorate. The resilience of a party's vote, and how many of their apparent supporters will actually turn up, is a very hard thing to measure, as ComRes research pointed out in the run-up to the poll. Basically, and very provisionally, we suspect that had young people voted in the same numbers that older people did, Labour's vote would have held up much better. They didn't, they don't usually, and they're unlikely to in the future. Get used to it. But many of them said that they would, and turnout levels as projected from polls were much higher than than they turned out to be in reality.
Slightly more worrying is the polling industry's tendency to huddle together for fear of getting it 'wrong' on the day. Labour pushed forward, not backwards, in the polling data from the campaign's last couple of days: a phenomenon that fooled many of us into thinking that the debacle wouldn't be as bad as 1992. Well, that was all a statistical artefact, created by pollsters tacking to the centre. There's plenty of academic evidence of herding, and it's intuitively understandable that no pollster wants to get too out of line. They're commercial organisations: drop a big howler, and you're likely to look silly and lose work. It's as simple as that. One pollster, Survation, has even bravely come out and admitted that it suppressed a voting survey that got the eventual result pretty much bang on - because it 'looked wrong'. This, to be fair to them, is not just a matter of commercial necessity. It's a classic example of 'anchoring' in action, reflecting the process by which human beings try not to stand out from the crowd, and find it difficult to shrug off conventional thinking and social norms. Labour and the Conservatives neck-and-neck? How exciting, everyone's saying it. I'll just put these contrary results to one side, then. Understandable: but deeply problematical, given how the polls shaped and then decided this election.
4. Labour's fundamentals. Let's be frank here. Labour's reputation on the economy was shattered in 2010, and has never recovered. That's the true reason why the fear of a government that would be both 'weak' and 'left-wing', partly because it was backed by the SNP, could gain such a quick strangehold on the public imagination. It was voters' underlying scratch-they-couldn't-itch all the time. This wasn't a foregone conclusion: Labour did not wallow quite so low in the water between the beginning of the economic crisis in 2007 and the last-but-one General Election. Britain's relatively rapid recovery in 2009 and 2010, partly achieved via the Keynesian policies and international activism of Messrs Brown and Darling, didn't seem to go down too badly among the public. Years of small public sector deficits (and a few of surplus), attended by falling debt-to-GDP ratios, meant that the Government had a bit of elbow room to tackle the crisis head-on. What really hammered Labour here was the way the debate was framed in 2010 - 'there's no money left' memo and all - once the coalition was able to say 'we're clearing up the mess of Labour's overspending'. Total nonsense, of course: but put together with Labour's leadership problems, this caused many voters to continually blame Labour for their hardships and struggles: a kind of 'shadow incumbency' that the party never really freed itself from.
Another problem Labour clearly had - and this was the mainstay of our initial reaction to the result - is cultural. They are 'too London', too metropolitan, too elitist. Many voters probably think Mr Miliband went to a private school, just like Mr Cameron (he didn't). Labour's leadership speaks in incomprehensible jargon, far from the depressed and left behind places where UKIP thrived. You know what? We thought, from all our data - every single spreadsheet and time series - that UKIP took the largest number of its votes from the Conservatives. In the end, on the day, they didn't. They swallowed up hundreds of thousands of traditional working class Labour voters, costing the party those marginal seats that even the relatively puny swing to Labour should have seen them gaining. That was the main difference between a Hung Parliament and a Conservative majority. And until Labour finds a language in which to speak to older, angrier, less well-educated white voters, it will continue to fail among these struggling socio-economic groups.
5. Dislike of coalitions. Coalitions are little understood among the public, and very few of them can offer a definition of what they are - or even how they work. Most are hostile to the very idea, and the whole Conservative-Liberal Democrat hookup in 2010 happened only via the chance of Gordon Brown being so unpopular, and so far behind, that many voters thought they could take the 'risk' of the Liberal Democrats and still get a new government. So it proved. But they were not converted to the idea of parties governing together. The stable coalition that Mr Cameron and his Liberal Democrat deputy Nick Clegg had managed might be all right - but the experiment was on probation, and the messy compromises and constant drum-beat of accommodation was tiresome, even on the relatively harmonious terms that the parties managed between 2010 and 2015.
The very fact that the race appeared to be neck-and-neck for so long made most Britons pause in the ballot box and say: 'do I really want a minority Labour administration backed up by the SNP and the Welsh and Northern Irish Nationalists?' The only answer to that was no: that seemed like an even more unstable and chaotic mixture than that the country had 'enjoyed' for five years. So voters acted accordingly, offering Mr Cameron the one single tiny path that he could ever have found to even the very small majority he enjoys today. We always like here to see elections as the effecting of the will and wisdom of the crowd. This time, we can sum it up like this: 'chaos or competence'. Which, funnily enough, was precisely one of the Conservatives' main election slogans.
All of this has come as some shock to your average cafe-dwelling leftist. They thought that a new world was coming into being: a multi-party continental fantasyland, in which parties would have to work together, in which voters would choose based on a buffet of different policy options, in which a 'left alliance' of all anti-Conservative parties could seize power, enact voting reform and generally make the world a softer, greener, more pluralistic place.
It wasn't. The two biggest parties' vote did not fall: it rose. The United Kingdom Independence Party won exactly one seat - exactly the same as the Greens. The SNP are now irrelevant to the ruthless and effective deployment of power at Westminster, having played just precisely that role that Mr Cameron allocated them in his own drama. He will now discard and ignore them for five years. The Conservatives are now firmly in the box seat: they will repeal the Human Rights Act, enact boundary reform, bring in their absurd 'English votes for English laws' scheme, stuff further (and ruinous) financial financial devolution down the throats of Cardiff and Edinburgh, silence the trade unions, and generally rule the roost for some time to come.
Until, that is, Labour wakes up from its university city dreaming, and starts to represent that deep England that took such fright at the SNP.
We advise you not to hold your breath.
