#!/usr/bin/env python
# coding: utf-8

# # Starbucks Capstone Challenge
# 
# ### Introduction
# 
# This data set contains simulated data that mimics customer behavior on the Starbucks rewards mobile app. Once every few days, Starbucks sends out an offer to users of the mobile app. An offer can be merely an advertisement for a drink or an actual offer such as a discount or BOGO (buy one get one free). Some users might not receive any offer during certain weeks. 
# 
# Not all users receive the same offer, and that is the challenge to solve with this data set.
# 
# Your task is to combine transaction, demographic and offer data to determine which demographic groups respond best to which offer type. This data set is a simplified version of the real Starbucks app because the underlying simulator only has one product whereas Starbucks actually sells dozens of products.
# 
# Every offer has a validity period before the offer expires. As an example, a BOGO offer might be valid for only 5 days. You'll see in the data set that informational offers have a validity period even though these ads are merely providing information about a product; for example, if an informational offer has 7 days of validity, you can assume the customer is feeling the influence of the offer for 7 days after receiving the advertisement.
# 
# You'll be given transactional data showing user purchases made on the app including the timestamp of purchase and the amount of money spent on a purchase. This transactional data also has a record for each offer that a user receives as well as a record for when a user actually views the offer. There are also records for when a user completes an offer. 
# 
# Keep in mind as well that someone using the app might make a purchase through the app without having received an offer or seen an offer.
# 
# ### Example
# 
# To give an example, a user could receive a discount offer buy 10 dollars get 2 off on Monday. The offer is valid for 10 days from receipt. If the customer accumulates at least 10 dollars in purchases during the validity period, the customer completes the offer.
# 
# However, there are a few things to watch out for in this data set. Customers do not opt into the offers that they receive; in other words, a user can receive an offer, never actually view the offer, and still complete the offer. For example, a user might receive the "buy 10 dollars get 2 dollars off offer", but the user never opens the offer during the 10 day validity period. The customer spends 15 dollars during those ten days. There will be an offer completion record in the data set; however, the customer was not influenced by the offer because the customer never viewed the offer.
# 
# ### Cleaning
# 
# This makes data cleaning especially important and tricky.
# 
# You'll also want to take into account that some demographic groups will make purchases even if they don't receive an offer. From a business perspective, if a customer is going to make a 10 dollar purchase without an offer anyway, you wouldn't want to send a buy 10 dollars get 2 dollars off offer. You'll want to try to assess what a certain demographic group will buy when not receiving any offers.
# 
# ### Final Advice
# 
# Because this is a capstone project, you are free to analyze the data any way you see fit. For example, you could build a machine learning model that predicts how much someone will spend based on demographics and offer type. Or you could build a model that predicts whether or not someone will respond to an offer. Or, you don't need to build a machine learning model at all. You could develop a set of heuristics that determine what offer you should send to each customer (i.e., 75 percent of women customers who were 35 years old responded to offer A vs 40 percent from the same demographic to offer B, so send offer A).

# # Data Sets
# 
# The data is contained in three files:
# 
# * portfolio.json - containing offer ids and meta data about each offer (duration, type, etc.)
# * profile.json - demographic data for each customer
# * transcript.json - records for transactions, offers received, offers viewed, and offers completed
# 
# Here is the schema and explanation of each variable in the files:
# 
# **portfolio.json**
# * id (string) - offer id
# * offer_type (string) - type of offer ie BOGO, discount, informational
# * difficulty (int) - minimum required spend to complete an offer
# * reward (int) - reward given for completing an offer
# * duration (int) - time for offer to be open, in days
# * channels (list of strings)
# 
# **profile.json**
# * age (int) - age of the customer 
# * became_member_on (int) - date when customer created an app account
# * gender (str) - gender of the customer (note some entries contain 'O' for other rather than M or F)
# * id (str) - customer id
# * income (float) - customer's income
# 
# **transcript.json**
# * event (str) - record description (ie transaction, offer received, offer viewed, etc.)
# * person (str) - customer id
# * time (int) - time in hours since start of test. The data begins at time t=0
# * value - (dict of strings) - either an offer id or transaction amount depending on the record
# 
# **Note:** If you are using the workspace, you will need to go to the terminal and run the command `conda update pandas` before reading in the files. This is because the version of pandas in the workspace cannot read in the transcript.json file correctly, but the newest version of pandas can. You can access the termnal from the orange icon in the top left of this notebook.  
# 
# You can see how to access the terminal and how the install works using the two images below.  First you need to access the terminal:
# 
# <img src="pic1.png"/>
# 
# Then you will want to run the above command:
# 
# <img src="pic2.png"/>
# 
# Finally, when you enter back into the notebook (use the jupyter icon again), you should be able to run the below cell without any errors.

# In[1]:


import pandas as pd
import numpy as np
import math
import json
import matplotlib.pyplot as plt
get_ipython().run_line_magic('matplotlib', 'inline')
import datetime
from time import time

# read in the json files
portfolio = pd.read_json('data/portfolio.json', orient='records', lines=True)
profile = pd.read_json('data/profile.json', orient='records', lines=True)
transcript = pd.read_json('data/transcript.json', orient='records', lines=True)


# ## Data Understanding

# In[2]:


portfolio.head()


# In[3]:


portfolio.shape


# In[4]:


portfolio.isnull().sum()


# In[5]:


portfolio.offer_type.value_counts()


# In[6]:


portfolio.describe()


# In[7]:


profile.head()


# In[8]:


profile.shape


# In[9]:


profile.isnull().sum()


# In[10]:


profile.describe()


# In[11]:


transcript.head()


# In[12]:


transcript.shape


# In[13]:


transcript.isnull().sum()


# ## Data Cleaning/Wrangling

# In[14]:


portfolio.head()


# From the portfolio dataset, we can see that each channel can be in a seprate column instead of putting them all together.

# In[15]:


#extract all channels names 

channels_set = set()
for i in portfolio.channels:
    channels_set.update(set(i))
channels_set


# There are four unique channel names in the channel column.

# In[16]:


# Portfolio cleaning function 
def clean_portfolio(portfolio):
    '''
    Cleaning the portfolio dataset
    INPUT:
    portfolio - the portfolio dataframe to be cleaned
    
    OUTPUT:
    portfolio - the cleaned portfolio dataframe
    
    '''
    # Convert to datetime
    cleaned_portfolio = portfolio.copy()
    cleaned_portfolio['duration'] = cleaned_portfolio['duration'] * 24
    
    # apply one hot encoding to channels column
    cleaned_portfolio['web'] = cleaned_portfolio['channels'].apply(lambda x: 1 if 'web' in x else 0)
    cleaned_portfolio['email'] = cleaned_portfolio['channels'].apply(lambda x: 1 if 'email' in x else 0)
    cleaned_portfolio['mobile'] = cleaned_portfolio['channels'].apply(lambda x: 1 if 'mobile' in x else 0)
    cleaned_portfolio['social'] = cleaned_portfolio['channels'].apply(lambda x: 1 if 'social' in x else 0)
    
    # apply one hot encoding to offer_type column
    offer_type = pd.get_dummies(cleaned_portfolio['offer_type'])
    
    # drop the channels and offer_type column
    cleaned_portfolio.drop(['channels', 'offer_type'], axis=1, inplace=True)
    
    # combine the portfolio and offer_type dataframe to form a cleaned dataframe
    cleaned_portfolio = pd.concat([cleaned_portfolio, offer_type], axis=1, sort=False)
    
    return cleaned_portfolio


# In[17]:


# Portfolio dataset after cleaning
cleaned_portfolio = clean_portfolio(portfolio)
cleaned_portfolio.head()


# In[18]:


# Profile cleaning function 
def clean_profile(profile):
    '''
    data cleaning the profile dataframe
    
    INPUT:
    profile - the profile dataframe to be cleaned
    
    OUTPUT:
    profile - the cleaned profile dataframe
    
    '''
    
    # Convert to datetime
    profile.became_member_on = pd.to_datetime(profile.became_member_on, format = '%Y%m%d')
    
    # age: (numeric) missing value encoded as 118
    profile['age'] = profile['age'].apply(lambda x: np.nan if x == 118 else x)
    
    # drop all missing values
    profile.dropna(inplace=True)
    
    # the number of days since the user is a memeber of starbucks
    #profile['days'] = datetime.datetime.today().date() - pd.to_datetime(profile['became_member_on'], format='%Y%m%d').dt.date
    profile['year'] = profile['became_member_on'].apply(lambda x: x.year)
    
    
    return profile


# In[19]:


# Profile dataset after cleaning
cleaned_profile = clean_profile(profile)
cleaned_profile.head()


# In[20]:


# Iterate over transcript table, check value column and update it, put each key in seperated column.
transcript['offer_id'] = ''
transcript['amount'] = 0
transcript['reward'] = 0

for idx, row in transcript.iterrows():
    for k in row['value']:
        if k == 'offer_id' or k == 'offer id':
            transcript.at[idx, 'offer_id']= row['value'][k]
        if k == 'amount':
            transcript.at[idx, 'amount']= row['value'][k]
        if k == 'reward':
            transcript.at[idx, 'reward']= row['value'][k]


# In[21]:


clean_transcript = transcript.drop('value', axis=1)
clean_transcript.head()


# ## Exploratory Data Analysis

# In[22]:


import seaborn as sns


# In[23]:


def get_offer_type(offer_id):
    '''
    input: offer_id: id of offer
    
    output: offer type
    
    '''
    
    try:
        offer_type = portfolio[portfolio['id'] == offer_id]['offer_type'].values[0]
        return offer_type
    except:
        offer_type = 'NA'
        return offer_type

transcript['offer_type'] = transcript.apply(lambda x: get_offer_type(x['offer_id']), axis=1)


# In[24]:


sns.countplot(transcript[transcript['offer_type'] != 'NA']['offer_type'])
plt.title('Number of Completed Promotion for Type Offer')
plt.ylabel('Number of transactions')
plt.xlabel('Offer Type')
plt.xticks(rotation = 50);


# From the chart above, we can see that BOGO is the highest offer followed by discount.

# In[25]:


# display function 
def display_customer_profile():
    '''Display customer profile with histograms'''
    
    # Display Histogram of Customer Age
    user_age = cleaned_profile['age'].plot(kind='hist', bins=20, title='What is the average age of Starbucks customers?')
    user_age.set_xlabel("Customer Age")
       
    # Display Histogram of the days being member
    plt.figure()
    days = cleaned_profile['year'].plot(kind='hist', bins=50,color ='#086A87', title='Year Customers became a Member')
    days.set_xlabel("Year")
    
    # Display Histogram of User Income
    plt.figure()
    income = cleaned_profile['income'].plot(kind='hist', bins=20, title='What is the average income for Starbucks customers?')
    income.set_xlabel("Income")
    
    # Display Histogram of user gender
    plt.figure()
    plt.hist(cleaned_profile['gender'], bins = 10);
    plt.xlabel('Gender');
    plt.title('Gender of Customer');


# In[26]:


display_customer_profile()


# Doing an exploratory data analysis on customer attributes as Age, income, gender, can give us some answers to our first and second questions, that men consume Starbucks products more than women with an average age of 54 (it’s a normal distribution), and an average income of 65404 ( it’s a right-skewed distribution), The highest number of membership was  recorded in 2017.

# ## Data Preparation

# In[27]:


# from transcript dataset create dummies variable
transcript = pd.get_dummies(clean_transcript , columns=['event'] , prefix='' ,prefix_sep='')
transcript.head()


# In[28]:


# from new transcript dataset drop duplicates
transcript.drop_duplicates(inplace=True)


# In[29]:


transcript.duplicated().sum()


# In[30]:


# merge new transcript dataset with profile 
merged = pd.merge(transcript , profile ,left_on='person' , right_on='id',how='inner')
merged.drop(columns=['id'] , inplace=True)
# Fill nans with 0
merged.reward.fillna(0 , inplace=True)
merged.amount.fillna(0 , inplace=True)
merged.head()


# In[31]:


# sort values
merged.sort_values(by=['time'] , ascending=True , inplace=True)

list1 = ['transaction', 'amount']

for i in list1:
    merged[i] = merged.groupby('person')[i].cumsum()


# In[32]:


# drop missing values 
merged.dropna(subset=['offer_id'] ,inplace=True)


# In[33]:


# merge the cleaned merged with protfolio
merged = merged.merge(portfolio , left_on='offer_id' , right_on='id'  ,how='left')
merged.head()


# In[34]:


Model_data = merged.groupby(['person' , 'offer_type']).agg(
    {
         'offer completed':sum,   
         'offer received': sum, 
         'offer viewed': sum,  
        'age':'last',
        'became_member_on':'last',
        'gender':'last',
        'income':'last'
    }
).reset_index()
Model_data.head()


# In[35]:


def offers_transformation(df):
    '''
    Function: to transform the offers
    
    Returns: The transformed offers
    '''
    
    df['bogo_received'] = 0
    df['bogo_completed'] = 0

    df['discount_received'] = 0
    df['discount_completed'] = 0
    # Transform BOGO offer 
    for index , row in df.iterrows():
        if(row['offer_type'] == 'bogo'):
            df.loc[index , 'bogo_completed'] = row['offer completed']
            df.loc[index , 'bogo_received'] = row['offer received']
    # Transform dsicount offer
        elif(row['offer_type'] == 'discount'):
            df.loc[index , 'discount_completed'] = row['offer completed']
            df.loc[index , 'discount_received'] = row['offer received']
        
        
    # groupby by person    
    return df.groupby(['person']).agg(
    {
        'age':'last',
        'became_member_on':'last',
        'gender':'last',
        'income':'last', 
        'bogo_received':sum,
        'bogo_completed':sum,
        'discount_received':sum,
        'discount_completed':sum,        
    }
).reset_index()


# In[36]:


trans_data = offers_transformation(Model_data)


# In[37]:


trans_data.head()


# In[38]:


# create colunms with the precnetage of each offer
trans_data['bogo_perce'] = trans_data.bogo_completed.div(trans_data.bogo_received.where(trans_data.bogo_received != 0, np.nan))*100
trans_data['discount_perce'] = trans_data.discount_completed.div(trans_data.discount_received.where(trans_data.discount_received != 0, np.nan))*100


# In[39]:


trans_data.fillna(0 , inplace=True)


# In[40]:


trans_data.isna().sum()


# In[41]:


trans_data['bogo_offer'] = 0
trans_data['discount_offer'] = 0


for i ,row in trans_data.iterrows():

    if(row['bogo_perce'] >= 80):
        trans_data.loc[i , 'bogo_offer'] = 1
        
    if(row['discount_perce'] >= 80 ):
        trans_data.loc[i , 'discount_offer'] = 1


# In[42]:


trans_data.head()


# ## Data Modeling

# We will build a model that predict whether or not someone will respond to an offer, If the answer is yes, the model will choose discount or BOGO. 

# In[43]:


mdata = trans_data.copy()
# create year and month columns from became_member_on column
mdata['year'] = mdata.became_member_on.dt.year
mdata['month'] = mdata.became_member_on.dt.month


# In[44]:


# select necessary features 
mdata = mdata[['age' , 'gender' , 'income' ,
                           'year' , 'month' , 'discount_offer' ,
                           'bogo_offer']]
mdata.head()


# In[45]:


#dummies the gender column
new_mdata = pd.get_dummies(mdata , columns=['gender'])


# In[46]:


# select features and target 
X = new_mdata.drop(columns=['discount_offer' ,'bogo_offer'])
y_bogo = new_mdata.bogo_offer
y_discount = new_mdata.discount_offer
new_mdata.head()


# In[64]:


from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.metrics import accuracy_score,f1_score
from sklearn.model_selection import train_test_split,GridSearchCV


# In[65]:


# split data set into train and test 
X_train, X_test, y_bogo_train, y_bogo_test = train_test_split(X, y_bogo, random_state=42)
X_train, X_test, y_discount_train, y_discount_test = train_test_split(X, y_discount, random_state=42)


# ## Metrics
# Since we have a simple classification problem, I will use accuracy to evaluate my models because we want to see how well our model is,by seeing the number of correct predictions vs total number of predicitons and also anwser the question of : How many custemrs use Starbucks offers?.This is my opinion and reason why I chose to use the easiest metric (accuracy).
# 

# 1. AdaBoostClassifier Model

# In[66]:


ABC_bogo = AdaBoostClassifier()
ABC_discount = AdaBoostClassifier()
ABC_bogo.fit(X_train, y_bogo_train)
ABC_discount.fit(X_train, y_discount_train)


# In[67]:


print(f'Bogo offer Accuracy of AdaBoostClassifier on training set: {round(ABC_bogo.score(X_train,y_bogo_train)*100,2)}%.')
print(f'Bogo offer Accuracy of AdaBoostClassifier on test set: {round(ABC_bogo.score(X_test,y_bogo_test)*100,2)}%.')
print(f'Discount offer Accuracy of AdaBoostClassifier on training set: {round(ABC_discount.score(X_train,y_discount_train)*100,2)}%.')
print(f'Discount offer Accuracy of AdaBoostClassifier on test set: {round(ABC_discount.score(X_test,y_discount_test)*100,2)}%.')


# 2. KNeighboursClassifier Model

# In[68]:


KNN_bogo = KNeighborsClassifier()
KNN_discount = KNeighborsClassifier()
KNN_bogo.fit(X_train, y_bogo_train)
KNN_discount.fit(X_train, y_discount_train)


# In[69]:


print(f'Bogo offer Accuracy of KNeighborsClassifier on training set: {round(KNN_bogo.score(X_train,y_bogo_train)*100,2)}%.')
print(f'Bogo offer Accuracy of KNeighborsClassifier on test set: {round(KNN_bogo.score(X_test,y_bogo_test)*100,2)}%.')
print(f'Discount offer Accuracy of KNeighborsClassifier on training set: {round(KNN_discount.score(X_train,y_discount_train)*100,2)}%.')
print(f'Discount offer Accuracy of KNeighborsClassifier on test set: {round(KNN_discount.score(X_test,y_discount_test)*100,2)}%.')


# 3. Logistic Regression Model

# In[70]:


LR_bogo = LogisticRegression()
LR_discount = LogisticRegression()
LR_bogo.fit(X_train, y_bogo_train)
LR_discount.fit(X_train, y_discount_train)


# In[71]:


print(f'Bogo offer Accuracy of LogisticRegression on training set: {round(LR_bogo.score(X_train,y_bogo_train)*100,2)}%.')
print(f'Bogo offer Accuracy of LogisticRegression on test set: {round(LR_bogo.score(X_test,y_bogo_test)*100,2)}%.')
print(f'Discount offer Accuracy of LogisticRegression on training set: {round(LR_discount.score(X_train,y_discount_train)*100,2)}%.')
print(f'Discount offer Accuracy of LogisticRegression on test set: {round(LR_discount.score(X_test,y_discount_test)*100,2)}%.')


# 4. DecissionTreeClassifier Model

# In[72]:


DT_bogo = DecisionTreeClassifier()
DT_discount = DecisionTreeClassifier()
DT_bogo.fit(X_train, y_bogo_train)
DT_discount.fit(X_train, y_discount_train)


# In[73]:


print(f'Bogo offer Accuracy of DecisionTreeClassifier on training set: {round(DT_bogo.score(X_train,y_bogo_train)*100,2)}%.')
print(f'Bogo offer Accuracy of DecisionTreeClassifier on test set: {round(DT_bogo.score(X_test,y_bogo_test)*100,2)}%.')
print(f'Discount offer Accuracy of DecisionTreeClassifier on training set: {round(DT_discount.score(X_train,y_discount_train)*100,2)}%.')
print(f'Discount offer Accuracy of DecisionTreeClassifier on test set: {round(DT_discount.score(X_test,y_discount_test)*100,2)}%.')


# ## Model Results

# In[74]:


models_bogo = [ABC_bogo, KNN_bogo, LR_bogo, DT_bogo ]
models_discount = [ABC_discount, KNN_discount, LR_discount, DT_discount]
model_names = [type(n).__name__ for n in models_bogo]
bogo_tr_accuracy = [x1.score(X_train, y_bogo_train)*100 for x1 in models_bogo]
discount_tr_accuracy = [x2.score(X_train, y_discount_train)*100 for x2 in models_discount]
bogo_test_accuracy = [y1.score(X_test,y_bogo_test)*100 for y1 in models_bogo]
discount_test_accuracy = [y2.score(X_test,y_discount_test)*100 for y2 in models_discount]


# In[75]:


results = [bogo_tr_accuracy,bogo_test_accuracy,discount_tr_accuracy,discount_test_accuracy]
results_df = pd.DataFrame(results, columns = model_names,index=['Bogo Training Accuracy', 'Bogo Test Accuracy','Discount Training Accuracy', 'Discount Test Accuracy']) 
results_df


# My analysis suggests that the resulting DecisionTreeClassifier model has an bogo training data accuracy of 99 and bogo test accuracy of 58 . discount training accuracy of 99 and discount test accuracy of 57 suggests that the DecisionTreeClassifier model I constructed did not overfit the training data.

# ## Model Improvement
# I will use Grid Search with DecisionTree to Improve the model to get better results.

# In[76]:


# Tuning to get better accuracy
tree_para = {'criterion':['gini','entropy'],
             'max_depth':[4,5,6,7,8,9,10,11,12,15,20,30,40,50,70,90,120,150]}

# create and fit a DecisionTree model

DT_bogo = DecisionTreeClassifier()
grid_bogo = GridSearchCV(estimator=DT_bogo, param_grid=tree_para,cv=2,n_jobs = -1)
grid_bogo_result = grid_bogo.fit(X_train, y_bogo_train)
DT_discount = DecisionTreeClassifier()
grid_discount = GridSearchCV(estimator=DT_discount, param_grid=tree_para,cv=2,n_jobs = -1)
grid_discount_result = grid_discount.fit(X_train, y_discount_train)

# summarize the results of the random parameter search

print(f'Best bogo model Score: {grid_bogo_result.best_score_}')
print(f'Best bogo model params: {grid_bogo_result.best_params_}')
print(f'Best discount model Score: {grid_discount_result.best_score_}')
print(f'Best discount model params: {grid_discount_result.best_params_}')


# ## Conclusion

# In this Project,I analyzed and built a model that predicts whether a customer will respond to an offer. If yes,the model will choose the best offer type (discount or bogo). The Strategy I used for solving this problem has two steps. Firstly,I merged offer portfolio, customer profile, and transaction data. Secondly, I built models for BOGO offer and Discount offer.The analysis suggests that Decision Tree model has the best training data accuracy.
# My Conclusion is that,the Company should focus more on BOGO and Discount offers since they are the one that tend to make customers buy more.
# 
# 

# ## Improvements
# I think I got to a point where I had good results and a good understanding of the data. But to make our results even better, It would be better to improve the data collection and fix issues with NaN values and also try to get more data like location, and when the transaction was completed, which branch and what time of the day. All these data can help us know when and where to give our offers.

# In[ ]:




