Load the dataset

First, import the packages required to continue.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

Read the dataset using Pandas

Read previously loaded data and store it in a variable named df, display the first few rows with head(), by default head() will return first 5 rows of the dataset, but you can specify any number of rows like head(10).

df = pd.read_csv('train.csv')
df.head()

Check the Dimension of the Dataset

Let's check for the basic information about the dataset by running simple commands.

#training_set.shape which returns a number of rows coloumns in dataset
df.shape

(891, 12)

#training_set.columns: It returns column headings.
df.columns

Index(['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp',
       'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked'],
      dtype='object')

#training_set.isnull().sum(): It returns a number of null values in each column.
df.isnull().sum()

PassengerId      0
Survived         0
Pclass           0
Name             0
Sex              0
Age            177
SibSp            0
Parch            0
Ticket           0
Fare             0
Cabin          687
Embarked         2
dtype: int64

Preparing the dataset

Perspectives

The target variable, "Survived," is what we will forecast after preprocessing our data. Therefore, we keep that column.

Only the "Age," "Cabin," and "Embarked" columns lack values.

PassengerId, Name, and Ticket don't really contribute much to forecasting the target variable.

We can create a new column called "Size of the family" because "ParCh" (Parent/Child) and "SibSp" (Siblings/Spouse) details are related to families.

The category data that needs to be encoded to numerical values are "Sex," "Cabin," and "Embarked."

Columns dropping

We will eliminate the columns with the lowest priority in this step. This category includes columns like "Ticket" and "PassengerId." To remove the columns, use drop().

#dropping ticket column
df.drop(['Ticket','PassengerId'], 1, inplace=True)

df.head()

Now, let’s run df.info(), and look at the status of our dataset.

df.info()

RangeIndex: 891 entries, 0 to 890
Data columns (total 10 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   Survived  891 non-null    int64  
 1   Pclass    891 non-null    int64  
 2   Name      891 non-null    object 
 3   Sex       891 non-null    object 
 4   Age       714 non-null    float64
 5   SibSp     891 non-null    int64  
 6   Parch     891 non-null    int64  
 7   Fare      891 non-null    float64
 8   Cabin     204 non-null    object 
 9   Embarked  889 non-null    object 
dtypes: float64(2), int64(4), object(4)
memory usage: 69.7+ KB

Which column has missing Value?

‘Cabin’, ‘Embarked’ and ‘Age’ column has missing values.

Creating new classes "Cabin"

Although the Cabin column has 687 missing values, it has a distinctive character that indicates the deck number at the beginning. As a result, we are going to establish a column named Deck to extract this information, which we may utilize in our forecast in the future.

# user defined function 
def assignDeckValue(CabinCode):
    if pd.isnull(CabinCode):
        category = 'Unknown'
    else:
        category = CabinCode[0]
    return category

Deck = np.array([assignDeckValue(cabin) for cabin in df['Cabin'].values])

df = df.assign(Deck = Deck)

df.head()

Derive a new column

‘ParCh’ and ‘SibSp’ are the details related to family size, so let’s derive a new column named ‘Size of the Family’.

df['FamilySize'] = df['SibSp'] + df['Parch'] + 1

‘Name’: Instead of dropping right away, from the Name of the Passenger, we need to get only their Title

 #Using expression pattern to extract the Title of the passenger
df['Title'] = df.Name.str.extract(' ([A-Za-z]+)\.', expand=False)

# Changing to common category
df['Title'] = df['Title'].replace(['Dr', 'Rev', 'Col', 'Major', 'Countess', 'Sir', 'Jonkheer', 'Lady', 'Capt', 'Don'], 'Others')
df['Title'] = df['Title'].replace('Ms', 'Miss')
df['Title'] = df['Title'].replace('Mme', 'Mrs')
df['Title'] = df['Title'].replace('Mlle', 'Miss')

df.head()

Now, let's drop Cabin, Name columns, we have extracted needed information from these two.

df.drop(['Cabin','Name'],1,inplace=True)

This is how our dataset looks like now.

df.head()

Managing missing values

‘Embarked’: For the Embarked column, only two rows are missing their data.
Embarked accepts categorical values such as C = Cherbourg, Q = Queenstown, and S = Southampton; in this instance, we may just substitute the most frequently occurring value, which in this example is 'S,' for the missing values.

# Returns count for each category
df['Embarked'].value_counts()

# Fills null values with 'S'-most common occurence
common = 'S'
df['Embarked']=df['Embarked'].fillna('S')

# Checking the no of null values now
df['Embarked'].isnull().sum()

0

df.isnull().sum()

Survived        0
Pclass          0
Sex             0
Age           177
SibSp           0
Parch           0
Fare            0
Embarked        0
Deck            0
FamilySize      0
Title           0
dtype: int64

Age: Using the mean value for each group, we will impute the missing values in the 'Age' column. Because there are many age ranges, taking the mean of the entire column can lead to conflicting results.

means = df.groupby('Title')['Age'].mean()

title_list = ['Master','Miss','Mr','Mrs','Others']
def age_missing_replace(means, dframe, title_list):
    for title in title_list:
        temp = dframe['Title'] == title 
        dframe.loc[temp, 'Age'] = dframe.loc[temp, 'Age'].fillna(means[title]) 

age_missing_replace(means, df, title_list)

df.head()

Coding categorical characteristics

Without being converted to numerical values, many machine learning algorithms are unable to accommodate categorical variables. Fortunately, there are numerous methods to manage this scenario provided by the sci-kit-learn and pandas python tools.

It's them,

• Finding and Replacing

• Labeling

• A single hot encode

• Individual Binary Encoding

• Using Sci-kit Learn's LabelEncoder

At first, we will only use map to convert the categorical values into numerical data ().

df['Embarked'] = df['Embarked'].map({'C':0, 'Q':1, 'S':2})
df['Sex'] = df['Sex'].map({'male':0, 'female':1})
df['Title'] = df['Title'].map({'Master':0,'Miss':1,'Mr':2,'Mrs':3,'Others':4})

df.head(20)

Manually replacing the categorical value is not the right choice if there are many categories.
Let’s do one conversion using LabelEncoder() provided by sklearn.preprocessing library.

from sklearn import preprocessing
le = preprocessing.LabelEncoder()
df['Deck'] = le.fit_transform(df['Deck'])

Dataset ready…

df.head()