Searching for wisdom in the Twitter cloud

Twitter works for journalists as a fantastic alert system, and becomes an interesting news source itself when events develop so fast that the media are not able to cover it. But is there any real wisdom in the Twitter cloud?

A few weeks ago, together with a colleague, I was ‘ashed in’ at Amsterdam airport; he could not fly to Johannesburg and my flight to Cyprus was cancelled, all due to the ash cloud from that infamous Icelandic volcano Eyjafjallajokull.

Twitter proved to be a great help though, and following the tweets gave us an idea who was flying where or not, what the ash cloud was doing, and what the weather report was looking like. On Twitter, we could easily find some answers.

Journalists who followed the tweets also got a good picture about what was going on, and searching the piles of tweets brought up interesting links, from explaining the background of the phenomenon, to the amount of losses various companies were incurring due to it. We were all surfing the crowd in order to discover some truth.

But is there a collective wisdom in the Twitter cloud?

Although I like Adams Smith ‘s simile about the ‘invisible hand’, which creates order in the market, the Twitter cloud is more like a huge pub were everybody is talking and shouting while the music is playing loud. In some corners, interesting things are happening more or less in a spontaneous and uncoordinated manner.

Last year I tried to analyse the tweets about a Turkish Airlines crash at Schiphol Airport near Amsterdam, and to compare that collection of tweets with several Cover It Live sessions about the same event which relied on Twitter for information as well.

The collection of tweets did not show any clear pattern of topics or consensus about what happened. On the contrary, the Cover It Live sessions showed after some time, consensus about the number of people in the plane and the number of fatalities. So I concluded that surfing the crowd, trying to establish some truth, is difficult if there is not a minimum sense of community and direct interaction between the ‘Twitterati’.

Other research showed that the social network of followers and ‘followees’ does not reveal a structure, but underneath it is a structured interactive network of friends (people who were sending direct messages). See for example: Social networks that matter: Twitter under the microscope, by Huberman, Romero and Fang Wu.

Perhaps I was expecting too much. Twitter is not about knowledge. It is a thermometer deep into the buttocks of society. It is about the mood, and popular hash tags tell us what is cool and what is not. But if you know the mood about a certain topic, perhaps you can predict an outcome based on the number of tweets.

This is the idea behind the study of Sitaram Asur and Bernardo Huberman, “Predicting the Future with Social Media”. The researchers, both working for Hewlett-Packard, analysed at the beginning of 2009, 2.89-million tweets sent by 1.2-million users about 24 movies.

Statistical analysis showed that there was a relationship between the number of tweets in the week before the movie was released and the first weekend box-office revenues. For the horror movie ‘The Crazies’, the statistical model predicted a revenue of $16.8-million. The box-office revenue was $16.06-million.

Compared to the Hollywood Stock Exchange, where people can bet on success or failure of a movie, the model performed significantly better.

After the first weekend people had seen the movie, they of course tweeted about it and they were either positive or negative. For the second weekend, the number of tweets and their content showed that positive responses improved the sales and negative responses decreased sales.

Herbert Blankensteijn, a Dutch journalist and blogger for one of the national newspapers, wondered if you could apply this model to the actual Stock Exchange to predict the rise and fall of stock prices.

Asur and Huberman had no answer, and for the moment I don’t want to bet my money on that. But the application of the model for marketing is obvious. With Twitter, a company can try to predict the sales of their product.

It also proves that there is some wisdom in the Twitter cloud.