Social sentiment can provide serious insight for big data: here’s how

As social media explodes, valuable trends are increasingly available to those able to discover them. Yet sorting out signal from noise is only getting harder. Smarter technology is helping businesses and cities do so.

What book should I read? What movie should we see? What’s the best restaurant in this neighbourhood?

Not so long ago, answers to these sorts of questions were best found through the world’s largest, oldest social network, better known as “word of mouth.”

Few forms of marketing are as potent. Yet the breadth of advice from one’s friend and family is limited, and sometimes hard to access.

These days, with the rise of digital social networks, consumers and business alike have a nearly limitless source of this kind of opinion, but face a growing challenge of finding out which opinions to trust.

Whether a simple “like” of a book on Facebook, a rant about a movie on Twitter, or a considered review of a restaurant on Yelp, there’s a rising torrent of opinions available to help shape our understanding of what’s hot and what’s not.

Social media is mushrooming in size, spreading from its early roots with younger adults into mainstream consumers. Every minute, there are some 1.7-million Facebook posts, a third of a million tweets, plus some 2.8 million YouTube views, according to Edelman Digital’s 2013 Social Media Trends.

Digitally speaking, these are just the tip of the iceberg. The spread of these networks and the data they create is gaining speed as smart phones increasingly become consumers’ dominant means of engagement.

Without tools to help navigate this deluge of data, any potentially valuable insight is just noise.

Curtain calls

The issue is not limited to consumers. Companies, non-profits and even cities are also reckoning with social media overload, in their efforts to harvest valuable insights from the whirlwind of opinions, tweets and posts.

There is a better way. While individuals are easily overwhelmed by this swirl, computers excel at the task, tirelessly sorting through huge pools of information, whether its financial data or, as it turns out, tweets.

In recent years, sophisticated analytics and natural language processing have matured to the point where computers can make sense of even the sometimes-cryptic shorthand of Twitter posts.

Scanning millions of these short messages can help gauge positive, negative and neutral opinions for all manner of products, policies, or people.

Consider the film industry, where strong advance buzz can lure execs to place a film in more screens upfront. Similarly, slow-burn movies may start out disappointingly but build a following slowly as word spreads. In this case, movie execs need early evidence that sentiment is on the rise and whether it is positive or negative.

Trend spotting

As technology’s ability to extract a valuable signal from the noisy flow of social media improves, it’s offering a tool to help brands identify trends sooner than their competitors.

Another wrinkle in the emerging discipline of social sentiment analysis is that many signals are emerging in relatively obscure niches across the web.

The rise of the cycle chic trend — which involves cycling in stylish street clothes, often atop classic bicycles — largely took root online and spread slowly at first but then gathered momentum. Soon, a variety of companies begin to serve the growing number of chic cyclists and others attracted to the movement.

For trend savvy fashion and accessory designers, early signals offers a head start to design and sell into a rising trend on the way up, before it saturates the mainstream.

Streamlining cities

Away from film and fashion preferences, social media also captures very real, often frustrating, aspects of daily life. Indeed, as much as we’re likely to post praise about a preferred product, it’s only human nature that we are more vocal, more often about daily hiccups in our routine.

Comments about traffic and commuter delays, for instance, are deep, persistent signal in any real-time sampling of social media posts. And while most individuals fire off these complaints in frustration, with little hope they’ll make a difference, cities are exploring how these e-gripes can help optimize city operations.

In Europe, a recent IBM Social Sentiment Index analyzed the public’s signaling about traffic in key cities.

The findings? In cities such as Barcelona, Spain and Eindhoven, Netherlands with heavy investment in public transportation, commuter sentiment is generally positive. Where driving is more prominent, accidents and construction — particularly during rush hour — generate some of the most negative sentiments seen among commuters.

For cities trying to make sense of these trends, these findings can help in both the near-term, and over the long run. Hour to hour, spikes in Twitter or social network postings in a given region can help deploy police and first responders to unkink the effects of an accident.

Longer term, this guidance can help shape decisions about public investment by shifting resources to modes and transport plans that will deliver the greatest satisfaction to the largest number of commuters. In Lyon, France, the city has formalized this effort, with a coordinated hub of digital services called L’Optimod to help improve mobility.

Consumers and citizens are actively telling us where their preferences are, and where problem spots are.

For filmmakers, fashionistas and city planners alike, the challenge is finding the best ways to listen and respond.

The tools to do so—by turning the din of social chatter into actionable signals—are here today and can help deliver better products and services day-to-day.