#The voting dataset from Chapter 1 contained a bunch of missing values that we dealt with for you behind the scenes.
# Now, it's time for you to take care of these yourself!

#The unprocessed dataset has been loaded into a DataFrame df. Explore it in the IPython Shell with the .head() method.
# You will see that there are certain data points labeled with a '?'. These denote missing values. As you saw in the video,
# different datasets encode missing values in different ways. Sometimes it may be a '9999', other times a 0 - real-world
# data can be very messy! If you're lucky, the missing values will already be encoded as NaN. We use NaN because it is an
# efficient and simplified way of internally representing missing data, and it lets us take advantage of pandas methods such as .dropna()
# and .fillna(), as well as scikit-learn's Imputation transformer Imputer().

#In this exercise, your job is to convert the '?'s to NaNs,
# and then drop the rows that contain them from the DataFrame.

import pandas as pd
import numpy as np
from sklearn.metrics import classification_report

df = pd.read_csv('../../DataSets/house_votes_84/house_votes_84.csv')

#Convert '?' to NaN
df[df == '?'] = np.nan
df[df == 'n'] = 0
df[df == 'y'] = 1

# Print the number of NaNs
print(df.isnull().sum())

# Print shape of original DataFrame
print("Shape of Original DataFrame: {}".format(df.shape))

# Drop missing values and print shape of new DataFrame
df = df.dropna()

# Print shape of new DataFrame
print("Shape of DataFrame After Dropping All Rows with Missing Values: {}".format(df.shape))

#Imputing missing data in a ML Pipeline I
#As you've come to appreciate, there are many steps to building a model, from creating training and test sets,
# to fitting a classifier or regressor, to tuning its parameters, to evaluating its performance on new data.
# Imputation can be seen as the first step of this machine learning process, the entirety of which can be viewed within the
# context of a pipeline. Scikit-learn provides a pipeline constructor that allows you to piece together these steps into one process and thereby simplify your workflow.
#You'll now practice setting up a pipeline with two steps: the imputation step, followed by the instantiation of a classifier.
# You've seen three classifiers in this course so far: k-NN, logistic regression, and the decision tree.
# You will now be introduced to a fourth one - the Support Vector Machine, or SVM. For now, do not worry about how it works under the hood.
# It works exactly as you would expect of the scikit-learn estimators that you have worked with previously, in that it has the same .fit() and .predict() methods as before.

# Import the Imputer module
from sklearn.preprocessing import Imputer
from sklearn.svm import SVC

# Setup the Imputation transformer: imp
imp = Imputer(missing_values='most_frequent', strategy='mean', axis=0)

# Instantiate the SVC classifier: clf
clf = SVC()

# Setup the pipeline with the required steps: steps
steps = [('imputation', imp),
        ('SVM', clf)]


#Imputing missing data in a ML Pipeline II
#Having setup the steps of the pipeline in the previous exercise, you will now use
# it on the voting dataset to classify a Congressman's party affiliation.
# What makes pipelines so incredibly useful is the simple interface that they provide.
# You can use the .fit() and .predict() methods on pipelines just as you did with your classifiers and regressors!
# Practice this for yourself now and generate a classification report of your predictions.
# The steps of the pipeline have been set up for you, and the feature array X and target variable array y have been pre-loaded.
# Additionally, train_test_split and classification_report have been imported from sklearn.model_selection and sklearn.metrics respectively.
# Import necessary modules
from sklearn.preprocessing import Imputer
from sklearn.pipeline import Pipeline
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
df = pd.read_csv('../../DataSets/house_votes_84/house_votes_84.csv')

df[df == '?'] = np.nan
df[df == 'n'] = 0
df[df == 'y'] = 1

y = df['vote']
X = df.drop(columns='vote')


# Setup the pipeline steps: steps
steps = [('imputation', Imputer(missing_values='NaN', strategy='most_frequent', axis=0)),
        ('SVM', SVC())]

# Create the pipeline: pipeline
pipeline = Pipeline(steps)

# Create training and test sets
X_train, X_test, y_train, y_test =  train_test_split(X, y, test_size=0.3, random_state=42)

# Fit the pipeline to the train set
pipeline.fit(X_train, y_train)

# Predict the labels of the test set
y_pred = pipeline.predict(X_test)

# Compute metrics
print(classification_report(y_test, y_pred))

