The sarcasm machine: Measuring sentiment in social media mentions

At the end of last year, I wrote a few posts on measuring social media, at the time when Google Analytics had recently launched Social Engagement Reporting (SER) in order for site owners to measure the impact of social media mentions in and outside of the website, as well as “real-time” which allowed users to see what was happening on their site at that exact moment. The real glitch, however, was how we expected computers to accurately report on social media metrics like sentiment when there are complex semantic occurrences like sarcasm.

Thankfully, we can park the Chinese room analogy I used, by casting our eyes on BrandsEye’s recently launched “Crowd” product which uses real flesh and bone to accurately gauge sentiment online.  BrandsEye Crowd claims that it “combines the benefits of three powerful elements: the all-knowing wisdom of the crowd, managed through the minute attention to detail of computers, and focused by the strategic judgement of trained professionals” in order to successfully decipher whether or not a user is being sarcastic in his or her output on social media platforms like Twitter.

Although some argue above and below the figure, my typical industry rates for accuracy of social media tools is around 70%; hell, even normal human beings still argue about whether someone is being sarcastic or not. What apparently makes Crowd that much better than its competitors is the fact that human intervention allows BrandsEye to achieve accuracy of up to 90-95%, or so their MD, Tim Shier, claims:

“This system is a radical innovation in the online reputation management space. We know of crowdsourced rating in academia, especially economics, but not for commercial brands. We’re very proud of the system, which is already showing incredible accuracy, and we believe can get to between 95-100% perfect with a bit more refinement. In this space accuracy is all-important — accurate data quality gives you quality insights, which helps you make better business decisions.”

Crowd’s solution is made up of two parts: an Elo rating system which allows the system to rank the respondents, as well as a custom-built artificial intelligence system that does the distribution of the mentions and the overall correlation of the data. The rating system allows Crowd not only to rate the quality of the responses, but also to reward raters who produce quality responses.

As the proliferation of social media tools like Facebook and Twitter happen, social media analysis will become even more important in the chase for engagement. Accuracy in terms of deciphering that analysis is going to be the difference between knowing you’re digitally shit, and knowing your digital shit – Brandseye’s Crowd looks like it will be able to tell you.