Titanic Survival Analysis With Azure Machine Learning

We are going to design one Machine Learning model, using a two-class classification decision jungle to find out, who survives and who doesn’t. It’s a two class problem since we are only determining whether a passenger survives or not. Here in this model, we will process the data, clean our data, split our data in train module and testing module, cross validate the train data, use two-class decision jungle as classifier and score the trained model and test the module and at the end evaluate the overall performance of the model.


Figure: Azure Machine Learning Diagram of Titanic Survival Predictive Analysis

Azure Machine Learning Model Components

Titanic survival predictive analysis Machine Learning model has eight blocks (Figure -6). With the accuracy of 81.7%, it can detect if a passenger survives or not. Drag and drop each component, connect them according to Figure 6, change the values of Split data component, trained model and two-class classifier. Save the model and run it.

• Dataset
  We are using titanice.csv as the dataset of Machine Learning model. This dataset has 1310 rows and 14 columns of Titanic ship passenger information.

• Clean Missing Data
  Clean missing data component is used to eliminate missing and garbage data from the system.

• Splitting Data
  Splitting the data in two. First one is for training and second one is for testing purposes. Click Split data component and write 0.70 as the fraction of rows in the first output dataset. We are using 70% of our total data as training data, which is 917 passengers' information and 30% of the data as testing the data, which is 393 passengers' information. This is important because we want to create a model that guesses survival rate and then test it against some data, which was not used to generate the model. With such a small data set, how we split the data is important.

• Partition and Sample
  Partition and sample is important to cross validate the data and make the model robust and reliable. Select Assign to Folds as the Partition or sample mode while 10 specifies the number of Folds to split evenly. We are using 10 Fold cross validation to validate the training data.

• Two-Class Decision Jungle
  Two-Class classification decision jungle is a classification model, which we are using to classify the data. Here, the selected number of decision DAG’s are 8. Maximum depth of the decision DAG's is 32 and the width is 128. Number of optimization steps per decision DAG layer is 2048. As we have already determined that survival column is our output parameter, we select Single Parameter as the trainer mode. If we were not sure about the best parameter, then we should select Multiple Parameter.

• Train Model
  Train model classifies the 70% training data, using the untrained Two-Class decision jungle module and returns one trained model , which also represents ILearnDotNet class. Click Trained model component and select a column from the launch column selector. This column will be used as the output column of Machine Learning system.

• Score Model
  Score model takes an input from Trained model and 30% testing dataset and returns scored dataset of the problem. It calculates the score of the trained system with respect to the test dataset.

• Evaluate Model
  Evaluation model is taking an input of Scored dataset from Score model and is giving us the evaluation resultS. Select Evaluate model, right click on it and select Visualize. Azure Machine Learning Studio will show the detailed result of the Machine Learning system.

Model Prediction Analysis and Accuracy Calculation

To analyze the performance of the model, there are so many parameters visible like True Positive, False positive, true negative, false negative, Accuracy of algorithm, precision, recall, f1 score, lift, ROC graph, AUC – Area under cover, throughput. When you do not have an algorithm knowledge, you should compare various parameters algorithm-wise and decide the final algorithm to train your model in classification example.

True positive: Survived passenger correctly identified.

False positive: Non-Survived passenger incorrectly identified as survived.

True negative: Non-Survived passenger correctly identified.

False negative: Survived passenger incorrectly identified as Non-Survived.

In general, Positive = identified and negative = not identified.

From Designed model, 393 passenger information were used to test the system,

True Positive: 139, False Negative: 5, False Positive: 67, True Negative: 182. Total: 393 passengers.

Accuracy rate is, 0.817 or, 81.7%, AUC (Area under Curve) value is, 88.2% and, Threshold value is 0.38.


Figure: ROC Curve of the problem

The way to read this chart is to say for every probability of survival what percentage actually survived. From figure 7, we can see that the best possible model is represented by Green line and a totally random pick is represented by Orange line. We can also have a really bad model, where the graph will show Blue line in bottom or top right. The way to compare these models is by looking at the Area under the Curve, which will be somewhere between 0 (the worst) and 1 (the best). Our model is reasonably good and is given an AUC score of 0.882 or 88.2%. You need to set threshold value as well to get the maximum amount of accuracy. Here, we have used 0.38 as threshold value. Threshold value helps us to identify the class from the scored probabilities.

Conclusion

Two-class classifier algorithms are useful to solve and predict two-class classification problems. If our primary concern is accuracy and if we want to avoid overfitting problem, then two-class decision jungle is a Google model to design Machine Learning system, where it shows high accuracy and decent run time.