Nate Silver: How a math geek used big data to predict the US elections

This year’s US elections saw statistician Nate Silver crunching numbers and spewing predictions guided by big data and smart algorithms — and the math geek is pretty famous today.

Silver was already fairly well known for his work on baseball and political forecasting especially on New York Times owned blog Fivethirtyeight, but this election has shot his name through the stratosphere.

In 2008 Silver called 49 out of 50 states, missing only Indiana (which went to Obama by 0.1%.) This year, according to projections, his data driven model correctly predicted 50 out of 50 states.

Earlier in the election race Silver came under fire on Twitter for predicting that President Barack Obama had a 73.6% chance of defeating the Republican challenger, Mitt Romney.

“Prediction is the name of Silver’s game, the basis for his celebrity,” writes Dylan Byers of Politico. “So should Mitt Romney win on Nov. 6, it’s difficult to see how people can continue to put faith in the predictions of someone who has never given that candidate anything higher than a 41 percent chance of winning (way back on June 2) and — one week from the election — gives him a one-in-four chance, even as the polls have him almost neck-and-neck with the incumbent.”

The article also quotes MSNBC’s Joe Scarborough who added that: “anybody that thinks that this race is anything but a tossup right now is such an ideologue, they should be kept away from typewriters, computers, laptops, and microphones for the next 10 days – because they’re jokes.”

I imagine that Silver fans are currently asking “who is the joke now?”. The below map tweeted by Michael Cosentino probably echoes what Silver’s fans are thinking.

For the Nate-haters, here’s the 538 prediction and actual results side by side twitter.com/cosentino/stat…

— Michael Cosentino (@cosentino) November 7, 2012

Silver’s model is based on data and his job is to apply logical patterns to that data as ReadWrite’s Dan Rowinski summarises:

“The important thing here is the way Silver creates his models. Essentially, he takes data sets and applies logical analytical methods that take his conclusions to deeper understanding. Like any good statistician, he is not looking for a specific outcome, but rather looking for the most finely tuned pattern recognition engine to predict future results.”

In light of the results and Silver’s accuracy it is interesting to explore how far we have come in terms of data predictions and what it means for future elections and big world events, when the numbers can be given with such certainty ahead of the actual votes.

“By 2016, if the networks are paying attention, don’t be surprised to see that the talking heads are all Nate Silver clones. Every media organization will now want its own state poll-based algorithm, especially given how much traffic Silver has driven to the New York Times‘ website,” writes Mashable’s Chris Taylor.

According to an AllThingsD report, the New York Times saw a surge in traffic to its site just for Silver, “A lot of the traffic is coming just for Nate,” said executive editor Jill Abramson. A statement from NYT’s PR staff says that 20 percent of visitors to the Times’ site “are including a trip to Silver’s blog as part of their journey”.

“That mathematical models can no longer be derided by ‘gut-feeling’ pundits. That Silver’s contention — TV pundits are generally no more accurate than a coin toss — must now be given wider credence,” says Taylor.

Below is a video of the blogger explaining how he got it right.

Image credit: Wikipedia