Classify Twitter's Tweets Based On Naive Bayes Algorithm

Introduction

Naive Bayes classification algorithm of Machine Learning is a very interesting algorithm. It is used as a probabilistic method and is a very successful algorithm for learning/implementation to classify text documents. Any kind of object can be classified based on a probabilistic model specification. This algorithm is based on Bayes' theorem. It is not a single algorithm but a family of algorithms. It comes under the category of supervised learning. It is used to predict information based on training datasets.

Earlier, I had explained about K-Nearest Neighbour algorithm. Conceptually, the K-NN algorithm is based on the Euclidean Distance formula, however, Naive Bayes is based on the concept of probability.

Explanation

Let us take Twitter's tweets and build a classifier based on the given tweets. This classifier will tell whether a tweet is under the category of "Politics or Sports" or not.

The basic example of the tweet data will be classified based on the containing texts (Table1).

Tweet Id	Text	Category
294051752079159296	99 days to go until the start of #ct13. Did you know Chris Gayle is the only player in the event\u2019s history to be dismissed for 99?	Sports
291019672701255681	On Jan 10, PM #Abe received a courtesy call from Mr. Yoshihiro Murai, Governor of Miyagi Prefecture. \nhttp://t.co/EsyP40Gl	Politics
305581742104932352	Video of last week's hot topics: #2pack, #Draghi, pensions & #drug tests. @Europarltv video http://t.co/9GVBa315vM	Politics
291520568396759041	10 off the over, 10 required! Captain Faulkner to bowl the last over, in close discussion with veteran Warne. The final spot on the line #BBL02	Sports

I have the below "training" data from Twitter's feed (Table2).

Tweet Id	Category	Text
306624404287275009	Sports	99 days to go until the start of #ct13. Did you know Chris Gayle is the only player in the event\u2019s history to be dismissed for 99?
306481199130505216	Sports	Tonight's Scottish First Division match between Dumbarton and Raith Rovers has been postponed due to a frozen pitch
304353716117590016	Politics	@GSANetwork raises awareness & stands up to stop #LGBT #bullying in school & online. http://t.co/FWIG5vvVmi @glaad
304844614517547008	Politics	Blasts Deja Vu. How many times have we been in this exact moment? Failed or ignored intel/no CCTV/blame game and innocents dead.

Below is the tweet test data that is unclassified (Table3).

TweetId	Text
301733794770190336	RT @aliwilgus: @tweetsoutloud How serious is NASA's commitment to the SLS and Orion programs, and the future of human space flight beyond ...
301576909517619200	RT @FardigJudith: This line in the President's State of the Union Address spoke to me. Check it out & share your #SOTU #CitizenRespo
256056214880919553	What is your favorite place to play badminton? Do you have a specific club in mind? Give them a shoutout! #badminton #clubs
300248062209691648	Sam wins the first game v Safarova #FedCup #AusvCze http://t.co/yjyZLnjr

I will classify or categorize the data of table2 by using the Naive Bayes Algorithm with the help of Python 3 code. Let us extract the important words from the tweeted sentences below.

def extract_tweet_words(tweet_words):
words = []
alpha_lower = string.ascii_lowercase
alpha_upper = string.ascii_uppercase
numbers = [str(n) for n in range(10)]
for word in tweet_words:
cur_word = ''
for c in word:
if (c not in alpha_lower) and (c not in alpha_upper) and (c not in numbers):
if len(cur_word) >= 2:
words.append(cur_word.lower())
cur_word = ''
continue
cur_word += c
if len(cur_word) >= 2:
words.append(cur_word.lower())
return words

Get Training data from a tweet.

def get_tweet_training_data():
f = open('training.txt', 'r')
training_data = []
for l in f.readlines():
l = l.strip()
tweet_details = l.split()
tweet_id = tweet_details[0]
tweet_label = tweet_details[1]
tweet_words = extract_words(tweet_details[2:])
training_data.append([tweet_id, tweet_label, tweet_words])
f.close()
return training_data

Get test data from tweet that will be classified.

def get_tweet_test_data():
f = open('test.txt', 'r')
validation_data = []
for l in f.readlines():
l = l.strip()
tweet_details = l.split(' ')
tweet_id = tweet_details[0]
tweet_words = extract_words(tweet_details[1:])
validation_data.append([tweet_id, '', tweet_words])
f.close()
return validation_data

Get a list of words in the training data.

def get_words(training_data):
words = []
for data in training_data:
words.extend(data[2])
return list(set(words))

Get the probability of each word in the training data of tweet.

def get_tweet_word_prob(training_data, label = None):
words = get_words(training_data)
freq = {}
for word in words:
freq[word] = 1
total_count = 0
for data in training_data:
if data[1] == label or label == None:
total_count += len(data[2])
for word in data[2]:
freq[word] += 1
prob = {}
for word in freq.keys():
prob[word] = freq[word]*1.0/total_count
return prob

Get the probability of a given label.

def get_tweet_label_count(training_data, label):
count = 0
total_count = 0
for data in training_data:
total_count += 1
if data[1] == label:
count += 1
return count*1.0/total_count

Apply the Naive Bayes Model like below.

def label_tweet_data(test_data, sports_word_prob, politics_word_prob, sports_prob, politics_prob):
labels = []
for data in test_data:
data_prob_sports = sports_prob
data_prob_politics = politics_prob
for word in data[2]:
if word in sports_word_prob:
data_prob_sports *= sports_word_prob[word]
data_prob_politics *= politics_word_prob[word]
else:
continue
if data_prob_sports >= data_prob_politics:
labels.append([data[0], 'Sports', data_prob_sports, data_prob_politics])
else:
labels.append([data[0], 'Politics', data_prob_sports, data_prob_politics])
return labels

Print the labeled or categorize the test data like below.

def print_labelled_data(labels):
f_out = open('test_labelled_output.txt', 'w')
for [tweet_id, label, prob_sports, prob_politics] in labels:
f_out.write('%s %s\n' % (tweet_id, label))
f_out.close()

Read the training and test data like below.

training_data = get_tweet_training_data()
test_data = get__tweet_test_data()

Get the probability of each word.

word_prob = get_tweet_word_prob(training_data)
sports_word_prob = get_tweet_word_prob(training_data, 'Sports')
politics_word_prob = get_tweet_word_prob(training_data, 'Politics')

Get the probability of each label.

sports_prob = get_tweet_label_count(training_data, 'Sports')
politics_prob = get_tweet_label_count(training_data, 'Politics')

Normalize for stop words.

for (word, prob) in word_prob.items():
sports_word_prob[word] /= prob
politics_word_prob[word] /= prob

Label the test data and print it.

test_labels = label_tweet_data(test_data, sports_word_prob, politics_word_prob, sports_prob, politics_prob)
print_labelled_data(test_labels)

The output example of this algorithm will be something like below.

TweetId	Category
301733794770190336	Politics
301576909517619200	Politics
305057161682227200	Sports
286543227178328066	Politics

I have attached the complete Python code with test data, training data, and output categorized/labeled data. You can also generate the output data by running this Python code of Machine learning.

Pre-requisites for the running this code include -

Python 3.5
Jupiter's notebook will be good.

Conclusion

Naive Bayes algorithm is based on probability and it is very good for the labeling of data.