Great comments though. It’s always good to get the insights from people working in the field.

With computers you get scale but probably more mistakes, with humans you can’t scale but you probably get more accuracy. Regardless, if you are analyzing the English language, no one can claim the holy grail because both humans and computers make mistakes. You have to use the right tools for the job.

Not strictly about sentiment but it’s something we try to deal with at Networked Insights too: http://bit.ly/gRb8T

An algorithm may “learn over time” but in order to do so it needs consistent re-training by a user base or trained group of taggers. The more complex the sentiment categories get, the more human input is required to build accuracy over time. This training factor does not take away from the fact that a working machine solution to certain problems is far more scalable than a human one.

I think the key is transparency–in terms of how you do your training and to what degree human input is required. If interested, I wrote a little about our own training process here: http://adaptivesemantics.com/blogs/Building_consistency

I’m not sure I understand this. Surely if your individual sentiment calculations are correct, then in aggregation they would show correct sentiment too?

So, if you have ten positive tweets and five negatives, then you would show a bar chart with positives at ten, and negatives at five, or give a ‘score’ of 10:5 or 200% or 2:1.

If not then you’re aggregating incorrect sentiment to give… incorrect sentiment.

The other aspect is how you interpret auto-sentiment. With twendz, we’re trying not to present it as the gospel truth, but rather to help you to identify trends as an early-warning indicator, or a snapshot of a point-in-time for your brand. This is why we don’t display the sentiment for individual tweets, but instead as an aggregate since this encompasses a greater accuracy.

You’re right, though. It’s not a measurement metric, it should instead be treated as an indicator or a gauge.

Perhaps ‘sympathetic’ would be a better word here.

Let’s take your example:
” I used the tool as a test when Jade Goody died. I noticed it would class as ‘negative’ tweets that said “sad that Jade Goody died””.

This is actually correct behavior.

Sentiment, as it applies in this context is scored based on the tonality of the post, which in this case is negative. The author is sad as the post clearly shows, hence, the mention is scored as such.

Perhaps, it’s the use of the labels “positive” and “negative” that are misleading.

SocialMention’s sentiment scoring mechanism gauges tonality of the post as a means to flag content – it does not however, utilize a complicated language analysis … such a system wouldn’t be very accurate anyways given the extremely complex nature of internet speak.

Jon

Does that make you happy?

Random sampling? As you say, it’s been around for years. From what I’ve learned recently, it seems to me that, to accurately and reliably measure, you either use humans, or you use techniques that have ‘been around for years’ because they’re proven, statistically robust measures.

It doesn’t really matter that we’re dealing with ‘online’ here. There’s nothing that different. It’s just more volume. The same maths models and statistical techniques apply.

And I suppose you could say humans have been around for quite some time too.