In Python, the Grid Search technique evaluates various combinations of hyperparameters and identifies the combination that yields the lowest error score. This approach is particularly beneficial when there are only a limited number of hyperparameters to optimize. However, as the complexity of the Machine Learning model increases, this method tends to be surpassed by other probabilistic search techniques that utilize weighted randomness.
So let us begin by understanding Grid Search.
Understanding Grid Search
Grid Search is a technique for optimization that enables us to identify the most effective parameters to enhance a problem from a predefined set of parameter options, thereby automating the traditional 'trial-and-error' approach. While it can be applied to various optimization challenges, it is primarily recognized for its application in the field of machine learning, where it is used to determine the parameter values that yield the highest accuracy for models.
Let us consider that the model accepts the below three parameters in the form of input:
- Number of hidden layers [2, 4]
- Number of neurons in every layer [5, 10]
- Number of epochs [10, 50]
In order to explore two alternatives for each parameter input (as indicated in the square brackets above), it evaluates various combinations. For example, one potential combination could be [2, 5, 10]. Manually identifying these combinations would be quite cumbersome.
Imagine a scenario where we have ten distinct parameters as inputs, and we intend to explore five potential values for each individual parameter. This would require the programmer to manually input the values whenever there is a need to change a parameter, rerun the code, and meticulously document the results for every possible combination of these parameters.
Grid Search streamlines this procedure by taking in the potential values for each parameter and running the code to evaluate every conceivable combination. It generates results for these combinations and identifies the one that yields the highest accuracy.
Installing the Required Libraries
Prior to delving into the implementation of Grid Search using the Python programming language, it is important to first review some essential libraries and frameworks that must be installed on your system.
These libraries and frameworks are as follows:
- Python 3
- NumPy
- Pandas
- Keras
- Scikit-Learn
The installation process for these libraries is fairly straightforward. We can utilize the pip package manager to install them, as demonstrated in the following example:
$ pip install numpy tensorflow pandas scikit-learn keras
Note: If any issues arise while executing any package, try reinstalling and referring to each package's official documentation.
Let's commence the implementation of Grid Search using Python.
Implementation of Grid Search in Python
In the upcoming section, we will explore the implementation of Grid Search within a real-world application. Our focus will be on executing the code and providing a thorough discussion about the specific part where Grid Search is utilized, rather than delving into the topics of Machine Learning and Data Pre-processing.
So, let's get started.
We will utilize the Diet Dataset, which includes information about the height and weight of various individuals, categorized by attributes such as gender, age, and diet type. The data can be seamlessly imported from an online source using the read_csv function provided by the Pandas library.
However, prior to proceeding, it is essential to import the necessary packages:
File: mygrid.py
import sys
import pandas as pd
import numpy as np
from sklearn.model_selection import GridSearchCV, KFold
from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.wrappers.scikit_learn import KerasClassifier
from keras.optimizers import Adam
Explanation:
In the code snippet provided above, we have brought in the essential packages and libraries needed for the project. Additionally, you can save the program file and run it to verify that the libraries and packages have been correctly installed and imported.
After successfully importing the required packages, the next step involves utilizing the following code snippet to load the dataset and display the initial five rows.
File: mygrid.py
# importing the dataset
mydf = pd.read_csv("Diet_Dataset.csv")
# printing the first five lines of dataset
print(mydf.head())
Output:
Person gender Age Height pre.weight Diet weight6weeks
0 25 41 171 60 2 60.0
1 26 32 174 103 2 103.0
2 1 0 22 159 58 1 54.2
3 2 0 46 192 60 1 54.0
4 3 0 55 170 64 1 63.3
Explanation:
In the code snippet provided above, we utilized the read_csv function from the pandas library to import the dataset and assigned it to the variable mydf. Subsequently, we displayed the initial five rows of the dataset by applying the head method in conjunction with the mydf variable.
At this point, we will separate the data into two distinct sets: the feature set and the label set. This division is essential for us to implement standard scaling on the dataset effectively.
The snippet of code for the same is shown below:
File: mygrid.py
# converting dataframe into numpy array
mydataset = mydf.values
X = mydataset[:, 0:6]
Y = mydataset[:, 6].astype(int)
# Normalizing the data using sklearn StandardScaler
from sklearn.preprocessing import StandardScaler
myscaler = StandardScaler().fit(X)
# Transforming and displaying the training data
X_stdized = myscaler.transform(X)
mydata = pd.DataFrame(X_stdized)
Explanation:
In the code snippet provided, we have transformed the pandas DataFrame into a NumPy array. Subsequently, we imported the StandardScaler module from the sklearn library and applied the function to standardize the dataset. Finally, we utilized the transform function to transform and showcase the training data.
At this point, let us examine the subsequent code snippet to construct a basic deep learning model.
File: mygrid.py
# defining the function to create model
def create_my_model(learnRate, dropoutRate):
# Creating model
mymodel = Sequential()
mymodel.add(Dense(6, input_dim = 6, kernel_initializer = 'normal', activation = 'relu'))
mymodel.add(Dropout(dropoutRate))
mymodel.add(Dense(3, input_dim = 6, kernel_initializer = 'normal', activation = 'relu'))
mymodel.add(Dropout(dropoutRate))
mymodel.add(Dense(1, activation = 'sigmoid'))
# Compiling the model
my_Adam = Adam(learning_rate = learnRate)
mymodel.compile(loss = 'binary_crossentropy', optimizer = my_Adam, metrics = ['accuracy'])
return mymodel
Explanation:
The code snippet below illustrates the definition of a function named createmymodel that takes two arguments: learnRate and dropoutRate. Subsequently, we initialize the model as mymodel by utilizing the Sequential function. Additionally, the add method is employed in conjunction with the Dense and Dropout functions. Finally, the model is compiled using the compile method.
Consequently, when we run the code, it will result in the loading of the dataset, its preprocessing, and the development of a machine learning model. Given that our primary focus is to comprehend how Grid Search operates, we have not conducted a train/test split, and we have fitted the model using the entire dataset.
In the upcoming section, we will explore how Grid Search simplifies the programmer's workflow by enhancing the optimization of parameters.
Training the Model without Grid Search
In the code example provided below, we will construct a model utilizing parameter values that we have chosen either randomly or through our intuition, and we will evaluate the performance of our model:
File: mygrid.py
# Declaring the values of the parameter
dropoutRate = 0.1
epochs = 1
batchSize = 20
learnRate = 0.001
# Creating the model object by calling the create_my_model function we created above
mymodel = create_my_model(learnRate, dropoutRate)
# Fitting the model onto the training data
mymodel.fit(X_stdized, Y, batch_size = batchSize, epochs = epochs, verbose = 1)
Output:
4/4 [==============================] - 41s 14ms/step - loss: 0.9364 - accuracy: 0.0000e+00
Explanation:
In the previously mentioned code segment, we have specified the values for the parameters, which include dropoutRate, epochs, batchSize, and learnRate. Subsequently, we instantiated the model object by invoking the createmymodel function. Following that, we proceeded to train the model using the training dataset.
As a result, the accuracy we got is 0.0000e+00.
Optimizing Hyper-parameters using Grid Search
In instances where we are not employing the Grid Search technique, we can simply invoke the fit function on the model that we established earlier. However, to leverage the Grid Search method effectively, it is necessary to supply several arguments to the createmymodel function. Additionally, we must define a grid that encompasses various alternatives to test for each parameter. Let us carry out that process in separate stages.
To begin with, we will attempt to alter the createmymodel function so that it can receive parameters from the invoking function, as illustrated in the following example:
File: mygrid.py
def create_my_model(learnRate, dropoutRate):
# Creating the model
mymodel = Sequential()
mymodel.add(Dense(6, input_dim = 6, kernel_initializer = 'normal', activation = 'relu'))
mymodel.add(Dropout(dropoutRate))
mymodel.add(Dense(3, input_dim = 6, kernel_initializer = 'normal', activation = 'relu'))
mymodel.add(Dropout(dropoutRate))
mymodel.add(Dense(1, activation = 'sigmoid'))
# Compile the model
myadam = Adam(learning_rate = learnRate)
mymodel.compile(loss = 'binary_crossentropy', optimizer = myadam, metrics = ['accuracy'])
return mymodel
# Creating the model object
mymodel = KerasClassifier(build_fn = create_my_model, verbose = 1)
Explanation:
In the code snippet provided above, we have implemented modifications to the earlier createmymodel function and utilized the KerasClassifier to generate the model instance.
Next, we will execute the Grid Search algorithm and apply it to the dataset.
File: mygrid.py
# Defining the arguments that we want to use in Grid Search along
# with the list of values that we want to try out
learnRate = [0.001, 0.02, 0.2]
dropoutRate = [0.0, 0.2, 0.4]
batchSize = [10, 20, 30]
epochs = [1, 5, 10]
# Making a dictionary of the grid search parameters
paramgrid = dict(learnRate = learnRate, dropoutRate = dropoutRate, batch_size = batchSize, epochs = epochs )
# Building and fitting the GridSearchCV
mygrid = GridSearchCV(estimator = mymodel, param_grid = paramgrid, cv = KFold(random_state = None), verbose = 10)
gridresults = mygrid.fit(X_stdized, Y)
# Summarizing the results in a readable format
print("Best: " + gridresults.best_score_ + " using " + gridresults.best_params_)
means = gridresults.cv_results_['mean_test_score']
stds = gridresults.cv_results_['std_test_score']
params = gridresults.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
print(mean + "(" + stdev + ")" + " with: " + param)
Output:
Best: 0.00347268912077, using {batch_size=10, dropoutRate=0.4, epochs=5, learnRate=0.2}
Explanation:
The output displayed above indicates the combination of parameters that produces the highest level of accuracy.
In conclusion, it is evident that Grid Search is straightforward to implement within the Python programming language and has significantly reduced the time spent on manual efforts. We can enumerate all the parameters we wish to optimize, specify the values for testing, run the code, and then set it aside. This method is so user-friendly and efficient that it demands minimal input from the programmer. After identifying the optimal combination of parameters, we can apply it to the final model.