3-3-2-15. Mini-Project: Mean Normalization and Data Separation

Mean Normalization

In machine learning we use large amounts of data to train our models. Some machine learning algorithms may require that the data is normalized in order to work correctly. The idea of normalization, also known as feature scaling, is to ensure that all the data is on a similar scale, i.e. that all the data takes on a similar range of values. For example, we might have a dataset that has values between 0 and 5,000. By normalizing the data we can make the range of values be between 0 and 1.

In this lab, you will be performing a different kind of feature scaling known as mean normalization. Mean normalization will scale the data, but instead of making the values be between 0 and 1, it will distribute the values evenly in some small interval around zero. For example, if we have a dataset that has values between 0 and 5,000, after mean normalization the range of values will be distributed in some small range around 0, for example between -3 to 3. Because the range of values are distributed evenly around zero, this guarantees that the average (mean) of all elements will be zero. Therefore, when you perform mean normalization your data will not only be scaled but it will also have an average of zero.

To Do:

You will start by importing NumPy and creating a rank 2 ndarray of random integers between 0 and 5,000 (inclusive) with 1000 rows and 20 columns. This array will simulate a dataset with a wide range of values. Fill in the code below

# import NumPy into Python


# Create a 1000 x 20 ndarray with random integers in the half-open interval [0, 5001).
X = 

# print the shape of X
# Average of the values in each column of X
ave_cols =

# Standard Deviation of the values in each column of X
std_cols =
# Print the shape of ave_cols


# Print the shape of std_cols

Solution

You can refer the the Solution.ipynb available in the notebook server. See the snapshots below to navigate to the Solution file.

# import NumPy into Python
import numpy as np

# Create a 1000 x 20 ndarray with random integers in the half-open interval [0, 5001).
X = np.random.randint(0,5001,size=(1000, 20))

# print the shape of X
print("Shape of X is: ", X.shape)

# Average of the values in each column of X
ave_cols =np.mean(X, axis=0)

# Standard Deviation of the values in each column of X
std_cols = np.std(X, axis=0)

# Print the shape of ave_cols
print("The shape of ave_cols is: ", ave_cols.shape)

# Print the shape of std_cols
print("The shape of std_cols is: ", std_cols.shape)

# Mean normalize X
X_norm = (X - ave_cols) / std_cols

# Print the average of all the values of X_norm
# You can use either the function or a method. So, there are multiple ways to solve. 
print("The average of all the values of X_norm is: ")
print(np.mean(X_norm))
print(X_norm.mean())

# Print the average of the minimum value in each column of X_norm
print("The average of the minimum value in each column of X_norm is: ")
print(X_norm.min(axis = 0).mean())
print(np.mean(np.sort(X_norm, axis=0)[0]))

# Print the average of the maximum value in each column of X_norm
print("The average of the maximum value in each column of X_norm is: ")
print(np.mean(np.sort(X_norm, axis=0)[-1]))
print(X_norm.max(axis = 0).mean())

# We create a random permutation of integers 0 to 4
np.random.permutation(5)

# Create a rank 1 ndarray that contains a random permutation of the row indices of `X_norm`
row_indices = np.random.permutation(X_norm.shape[0])

# Make any necessary calculations.
# You can save your calculations into variables to use later.

# You have to extract the number of rows in each set using row_indices.
# Note that the row_indices are random integers in a 1-D array. 
# Hence, if you use row_indices for slicing, it will NOT give the correct result.

# Let's get the count of 60% rows. Since, len(X_norm) has a lenght 1000, therefore, 60% = 600
sixty = int(len(X_norm) * 0.6)

# Let's get the count of 80% rows
eighty = int(len(X_norm) * 0.8)

# Create a Training Set
# Here row_indices[:sixty] will give you first 600 values, e.g., [93 255 976 505 281 292 977,.....]
# Those 600 values will will be random, because row_indices is a 1-D array of random integers.
# Next, extract all rows represented by these 600 indices, as X_norm[row_indices[:sixty], :]
X_train = X_norm[row_indices[:sixty], :]

# Create a Cross Validation Set
X_crossVal = X_norm[row_indices[sixty: eighty], :]

# Create a Test Set
X_test = X_norm[row_indices[eighty: ], :]

# Print the shape of X_train
print(X_train.shape)

# Print the shape of X_crossVal
print(X_crossVal.shape)

# Print the shape of X_test
print(X_test.shape)
%d 블로거가 이것을 좋아합니다: