Twitter - Data Mining The Stream of Consciousness

I may be a little late to the party, but over the last couple of weeks I’ve been making use of the application that has set many of the technorati’s hearts aflutter - twitter.

For those that have either been living under a rock, or alternatively actually have a life, Twitter is a simple service that allows you to send and receive small, frequent updates answering just one question - “What are you doing?” In place of the more familiar “friending” that you might find on social networks, twitter allows you to “follow” individuals - In essence, subscribing to their updates. If you want to see what you friends are doing, follow them on twitter. If they want to see what you are doing, they follow you. It’s easy to see that this straightforward mechanism actually allows two very distinct types of relationship - one being a bi-directional relationship (friendship) where you both know what is going on with the other, and the other relationship being a one-way “informational” relationship, where you want to be a consumer of ideas and information from someone else.

Okay, so now you know the how. The big question is of course, why?

Having used the service for a few weeks, I can definitely say there is a certain kind of “connectedness” that you feel when you get involved in (or rather, know about) the minutiae of people’s lives. I’ve personally found that large, transient cities like San Francisco are teeming with people that you know from seeing them at other places - or as Nivi puts it, “I know lots of people that I don’t know”. Twitter, in this social group, is a kind of glue that actually allows you to get to know these people better. Suddenly you find when you next see someone, you have something more personal to talk about other than the standard “how’s work?” or “how are you?”

But there’s another aspect that hasn’t yet really been explored in the world of Twitter. With the exception of “private” direct messaging, almost everything said on twitter is open and public. Information about your profile, who you follow and where you are location is easily accessed. The twitter public stream is not just a stream of public consciousness, it’s a rich source of information about the relationships between people and what they’re interested in. Potentially, you can even determine what’s hot and what’s not within certain demographic categories.

The information contained within Twitter, if properly utilized, would be a great source of demographic information for advertisers and would essentially enable a much more targeted, social version of Google’s ad sense. Given the 140 character limit versus the 160 characters of a text message, I’m sure this is something the creators of twitter are thinking about - there’s plenty that can be squeezed into 20 characters. And given how open the information on twitter is, you can be sure that if they don’t do it soon, somebody else will.

Update: Since writing this article, I’ve come across a good article listing the top twitter clients as found by scanning the public timeline. Whilst it’s not as detailed a treatment as I outline above, it’s a good example of mining using the twitter data set.