Logistic Regression¶
# used for manipulating directory paths
import os
# Scientific and vector computation for python
import numpy as np
from matplotlib import pyplot
%matplotlib inline
# Optimization module in scipy
from scipy import optimize
import utils
1 Logistic Regression¶
In this part of the exercise, you will build a logistic regression model to predict whether a student gets admitted into a university. Suppose that you are the administrator of a university department and you want to determine each applicant’s chance of admission based on their results on two exams. You have historical data from previous applicants that you can use as a training set for logistic regression. For each training example, you have the applicant’s scores on two exams and the admissions decision. Your task is to build a classification model that estimates an applicant’s probability of admission based the scores from those two exams.
The following cell will load the data and corresponding labels:
# Load data
# The first two columns contains the exam scores and the third column
# contains the label.
data = np.loadtxt(os.path.join('Data', 'ex2data1.txt'), delimiter=',')
X, y = data[:, 0:2], data[:, 2]
1.1 Visualizing the data¶
def plotData(X, y):
"""
Plots the data points X and y into a new figure. Plots the data
points with * for the positive examples and o for the negative examples.
"""
fig = pyplot.figure()
pos = (y == 1)
neg = (y == 0)
# Plot Examples
pyplot.plot(X[pos, 0], X[pos, 1], 'k*', lw=2, ms=10)
pyplot.plot(X[neg, 0], X[neg, 1], 'ko', mfc='y', ms=8, mec='k', mew=1)
Now, we call the implemented function to display the loaded data:
plotData(X, y)
# add axes labels
pyplot.xlabel('Exam 1 score')
pyplot.ylabel('Exam 2 score')
pyplot.legend(['Admitted', 'Not admitted'])
pass
1.2 Implementation¶
1.2.1 Warmup exercise: sigmoid function¶
Before you start with the actual cost function, recall that the logistic regression hypothesis is defined as:
$$ h_\theta(x) = g(\theta^T x)$$where function $g$ is the sigmoid function. The sigmoid function is defined as:
$$g(z) = \frac{1}{1+e^{-z}}$$.
Your first step is to implement this function sigmoid so it can be
called by the rest of your program. When you are finished, try testing a few
values by calling sigmoid(x) in a new cell. For large positive values of x, the sigmoid should be close to 1, while for large negative values, the sigmoid should be close to 0. Evaluating sigmoid(0) should give you exactly 0.5. Your code should also work with vectors and matrices. For a matrix, your function should perform the sigmoid function on every element.
def sigmoid(z):
"""
Compute sigmoid function given the input z.
Parameters
----------
z : array_like
The input to the sigmoid function. This can be a 1-D vector
or a 2-D matrix.
Returns
-------
g : array_like
The computed sigmoid function. g has the same shape as z, since
the sigmoid is computed element-wise on z.
Instructions
------------
Compute the sigmoid of each value of z (z can be a matrix, vector or scalar).
"""
# convert input to a numpy array
z = np.array(z)
# You need to return the following variables correctly
g = np.zeros(z.shape)
# ====================== YOUR CODE HERE ======================
g = 1 / (1 + np.exp(-z))
# =============================================================
return g
The following cell evaluates the sigmoid function at z=0. You should get a value of 0.5. You can also try different values for z to experiment with the sigmoid function.
# Test the implementation of sigmoid function here
z = 0
g = sigmoid(z)
print('g(',z,') = ', g)
1.2.2 Cost function and gradient¶
Now you will implement the cost function and gradient for logistic regression. Before proceeding we add the intercept term to X.
# Setup the data matrix appropriately, and add ones for the intercept term
m, n = X.shape
# Add intercept term to X
X = np.concatenate([np.ones((m, 1)), X], axis=1)
Now, complete the code for the function costFunction to return the cost and gradient. Recall that the cost function in logistic regression is
and the gradient of the cost is a vector of the same length as $\theta$ where the $j^{th}$ element (for $j = 0, 1, \cdots , n$) is defined as follows:
$$ \frac{\partial J(\theta)}{\partial \theta_j} = \frac{1}{m} \sum_{i=1}^m \left( h_\theta \left( x^{(i)} \right) - y^{(i)} \right) x_j^{(i)} $$Note that while this gradient looks identical to the linear regression gradient, the formula is actually different because linear and logistic regression have different definitions of $h_\theta(x)$.
def costFunction(theta, X, y):
"""
Compute cost and gradient for logistic regression.
Parameters
----------
theta : array_like
The parameters for logistic regression. This a vector
of shape (n+1, ).
X : array_like
The input dataset of shape (m x n+1) where m is the total number
of data points and n is the number of features. We assume the
intercept has already been added to the input.
y : arra_like
Labels for the input. This is a vector of shape (m, ).
Returns
-------
J : float
The computed value for the cost function.
grad : array_like
A vector of shape (n+1, ) which is the gradient of the cost
function with respect to theta, at the current values of theta.
Instructions
------------
Compute the cost of a particular choice of theta. You should set J to
the cost. Compute the partial derivatives and set grad to the partial
derivatives of the cost w.r.t. each parameter in theta.
"""
# Initialize some useful values
m = y.size # number of training examples
# You need to return the following variables correctly
J = 0
grad = np.zeros(theta.shape)
# ====================== YOUR CODE HERE ======================
h = sigmoid(X.dot(theta.T))
J = (1 / m) * np.sum(-y.dot(np.log(h)) - (1 - y).dot(np.log(1 - h)))
grad = (1 / m) * (h - y).dot(X)
# =============================================================
return J, grad
Once you are done call your costFunction using two test cases for $\theta$ by executing the next cell.
# Initialize fitting parameters
initial_theta = np.zeros(n+1)
print(n)
cost, grad = costFunction(initial_theta, X, y)
print('Cost at initial theta (zeros): {:.3f}'.format(cost))
print('Expected cost (approx): 0.693\n')
print('Gradient at initial theta (zeros):')
print('\t[{:.4f}, {:.4f}, {:.4f}]'.format(*grad))
print('Expected gradients (approx):\n\t[-0.1000, -12.0092, -11.2628]\n')
# Compute and display cost and gradient with non-zero theta
test_theta = np.array([-24, 0.2, 0.2])
cost, grad = costFunction(test_theta, X, y)
print('Cost at test theta: {:.3f}'.format(cost))
print('Expected cost (approx): 0.218\n')
print('Gradient at test theta:')
print('\t[{:.3f}, {:.3f}, {:.3f}]'.format(*grad))
print('Expected gradients (approx):\n\t[0.043, 2.566, 2.647]')
1.2.3 Learning parameters using scipy.optimize¶
In the previous lab, you found the optimal parameters of a linear regression model using a closed form solution. This time, you will use the scipy.optimize module for gradient descent. SciPy is a numerical computing library for python. It provides an optimization module for root finding and minimization. As of scipy 1.0, the function scipy.optimize.minimize is the method to use for optimization problems(both constrained and unconstrained).
For logistic regression, you want to optimize the cost function $J(\theta)$ with parameters $\theta$.
Concretely, you are going to use optimize.minimize to find the best parameters $\theta$ for the logistic regression cost function, given a fixed dataset (of X and y values). You will pass to optimize.minimize the following inputs:
costFunction: A cost function that, when given the training set and a particular $\theta$, computes the logistic regression cost and gradient with respect to $\theta$ for the dataset (X, y). It is important to note that we only pass the name of the function without the parenthesis. This indicates that we are only providing a reference to this function, and not evaluating the result from this function.initial_theta: The initial values of the parameters we are trying to optimize.(X, y): These are additional arguments to the cost function.jac: Indication if the cost function returns the Jacobian (gradient) along with cost value. (True)method: Optimization method/algorithm to useoptions: Additional options which might be specific to the specific optimization method. In the following, we only tell the algorithm the maximum number of iterations before it terminates.
If you have completed the costFunction correctly, optimize.minimize will converge on the right optimization parameters and return the final values of the cost and $\theta$ in a class object. Notice that by using optimize.minimize, you did not have to write any loops yourself, or set a learning rate. This is all done by optimize.minimize: you only needed to provide a function calculating the cost and the gradient.
In the following, we already have code written to call optimize.minimize with the correct arguments.
# set options for optimize.minimize
options= {'maxiter': 400}
res = optimize.minimize(costFunction,
initial_theta,
(X, y),
jac=True,
method='TNC',
options=options)
# the fun property of `OptimizeResult` object returns
# the value of costFunction at optimized theta
cost = res.fun
# the optimized theta is in the x property
theta = res.x
# Print theta to screen
print('Cost at theta found by optimize.minimize: {:.3f}'.format(cost))
print('Expected cost (approx): 0.203\n');
print('theta:')
print('\t[{:.3f}, {:.3f}, {:.3f}]'.format(*theta))
print('Expected theta (approx):\n\t[-25.161, 0.206, 0.201]')
Once optimize.minimize completes, we want to use the final value for $\theta$ to visualize the decision boundary on the training data as shown in the figure below.
To do so, we have written a function plotDecisionBoundary for plotting the decision boundary on top of training data. You do not need to write any code for plotting the decision boundary, but we also encourage you to look at the code in plotDecisionBoundary to see how to plot such a boundary using the $\theta$ values. You can find this function in the utils.py file which comes with this assignment.
# Plot Boundary
utils.plotDecisionBoundary(plotData, theta, X, y)
1.2.4 Evaluating logistic regression¶
After learning the parameters, you can use the model to predict whether a particular student will be admitted. For a student with an Exam 1 score of 45 and an Exam 2 score of 85, you should expect to see an admission
probability of 0.776. Another way to evaluate the quality of the parameters we have found is to see how well the learned model predicts on our training set. In this part, your task is to complete the code in function predict. The predict function will produce “1” or “0” predictions given a dataset and a learned parameter vector $\theta$.
def predict(theta, X):
"""
Predict whether the label is 0 or 1 using learned logistic regression.
Computes the predictions for X using a threshold at 0.5
(i.e., if sigmoid(theta.T*x) >= 0.5, predict 1)
Parameters
----------
theta : array_like
Parameters for logistic regression. A vecotor of shape (n+1, ).
X : array_like
The data to use for computing predictions. The rows is the number
of points to compute predictions, and columns is the number of
features.
Returns
-------
p : array_like
Predictions and 0 or 1 for each row in X.
Instructions
------------
Complete the following code to make predictions using your learned
logistic regression parameters.You should set p to a vector of 0's and 1's
"""
m = X.shape[0] # Number of training examples
# You need to return the following variables correctly
p = np.zeros(m)
# ====================== YOUR CODE HERE ======================
p = np.round(sigmoid(X.dot(theta.T)))
# ============================================================
return p
After you have completed the code in predict, we proceed to report the training accuracy of your classifier by computing the percentage of examples it got correct.
# Predict probability for a student with score 45 on exam 1
# and score 85 on exam 2
prob = sigmoid(np.dot([1, 45, 85], theta))
print('For a student with scores 45 and 85,'
'we predict an admission probability of {:.3f}'.format(prob))
print('Expected value: 0.775 +/- 0.002\n')
# Compute accuracy on our training set
p = predict(theta, X)
print('Train Accuracy: {:.2f} %'.format(np.mean(p == y) * 100))
print('Expected accuracy (approx): 89.00 %')
2 Regularized logistic regression¶
In this part of the exercise, you will implement regularized logistic regression to predict whether microchips from a fabrication plant passes quality assurance (QA). During QA, each microchip goes through various tests to ensure it is functioning correctly. Suppose you are the product manager of the factory and you have the test results for some microchips on two different tests. From these two tests, you would like to determine whether the microchips should be accepted or rejected. To help you make the decision, you have a dataset of test results on past microchips, from which you can build a logistic regression model.
First, we load the data from a CSV file:
# Load Data
# The first two columns contains the X values and the third column
# contains the label (y).
data = np.loadtxt(os.path.join('Data', 'ex2data2.txt'), delimiter=',')
X = data[:, :2]
y = data[:, 2]
2.1 Visualize the data¶
Similar to the previous parts of this exercise, plotData is used to generate a figure, where the axes are the two test scores, and the positive (y = 1, accepted) and negative (y = 0, rejected) examples are shown with
different markers.
plotData(X, y)
# Labels and Legend
pyplot.xlabel('Microchip Test 1')
pyplot.ylabel('Microchip Test 2')
# Specified in plot order
pyplot.legend(['y = 1', 'y = 0'], loc='upper right')
pass
The above figure shows that our dataset cannot be separated into positive and negative examples by a straight-line through the plot. Therefore, a straight-forward application of logistic regression will not perform well on this dataset since logistic regression will only be able to find a linear decision boundary.
2.2 Feature mapping¶
One way to fit the data better is to create more features from each data point. In the function mapFeature defined in the file utils.py, we will map the features into all polynomial terms of $x_1$ and $x_2$ up to the sixth power.
As a result of this mapping, our vector of two features (the scores on two QA tests) has been transformed into a 28-dimensional vector. A logistic regression classifier trained on this higher-dimension feature vector will have a more complex decision boundary and will appear nonlinear when drawn in our 2-dimensional plot. While the feature mapping allows us to build a more expressive classifier, it also more susceptible to overfitting. In the next parts of the exercise, you will implement regularized logistic regression to fit the data and also see for yourself how regularization can help combat the overfitting problem.
# Note that mapFeature also adds a column of ones for us, so the intercept
# term is handled
X = utils.mapFeature(X[:, 0], X[:, 1])
2.3 Cost function and gradient¶
Now you will implement code to compute the cost function and gradient for regularized logistic regression. Complete the code for the function costFunctionReg below to return the cost and gradient.
Recall that the regularized cost function in logistic regression is
$$ J(\theta) = \frac{1}{m} \sum_{i=1}^m \left[ -y^{(i)}\log \left( h_\theta \left(x^{(i)} \right) \right) - \left( 1 - y^{(i)} \right) \log \left( 1 - h_\theta \left( x^{(i)} \right) \right) \right] + \frac{\lambda}{2m} \sum_{j=1}^n \theta_j^2 $$Note that you should not regularize the parameters $\theta_0$. The gradient of the cost function is a vector where the $j^{th}$ element is defined as follows:
$$ \frac{\partial J(\theta)}{\partial \theta_0} = \frac{1}{m} \sum_{i=1}^m \left( h_\theta \left(x^{(i)}\right) - y^{(i)} \right) x_j^{(i)} \qquad \text{for } j =0 $$$$ \frac{\partial J(\theta)}{\partial \theta_j} = \left( \frac{1}{m} \sum_{i=1}^m \left( h_\theta \left(x^{(i)}\right) - y^{(i)} \right) x_j^{(i)} \right) + \frac{\lambda}{m}\theta_j \qquad \text{for } j \ge 1 $$def costFunctionReg(theta, X, y, lambda_):
"""
Compute cost and gradient for logistic regression with regularization.
Parameters
----------
theta : array_like
Logistic regression parameters. A vector with shape (n, ). n is
the number of features including any intercept. If we have mapped
our initial features into polynomial features, then n is the total
number of polynomial features.
X : array_like
The data set with shape (m x n). m is the number of examples, and
n is the number of features (after feature mapping).
y : array_like
The data labels. A vector with shape (m, ).
lambda_ : float
The regularization parameter.
Returns
-------
J : float
The computed value for the regularized cost function.
grad : array_like
A vector of shape (n, ) which is the gradient of the cost
function with respect to theta, at the current values of theta.
Instructions
------------
Compute the cost `J` of a particular choice of theta.
Compute the partial derivatives and set `grad` to the partial
derivatives of the cost w.r.t. each parameter in theta.
"""
# Initialize some useful values
m = y.size # number of training examples
# You need to return the following variables correctly
J = 0
grad = np.zeros(theta.shape)
# ===================== YOUR CODE HERE ======================
h = sigmoid(X.dot(theta.T))
temp = theta
temp[0] = 0
J = (1/m) * np.sum(-y.dot(np.log(h))-(1-y).dot(np.log(1-h))) + (lambda_/(2*m))*np.sum(np.square(temp))
grad = (1 / m) * (h - y).dot(X)
grad = grad + (lambda_ / m) * temp
# =============================================================
return J, grad
Once you are done with the costFunctionReg, we call it below using the initial value of $\theta$ (initialized to all zeros), and also another test case where $\theta$ is all ones.
# Initialize fitting parameters
initial_theta = np.zeros(X.shape[1])
# Set regularization parameter lambda to 1
# DO NOT use `lambda` as a variable name in python
# because it is a python keyword
lambda_ = 1
# Compute and display initial cost and gradient for regularized logistic
# regression
cost, grad = costFunctionReg(initial_theta, X, y, lambda_)
print('Cost at initial theta (zeros): {:.3f}'.format(cost))
print('Expected cost (approx) : 0.693\n')
print('Gradient at initial theta (zeros) - first five values only:')
print('\t[{:.4f}, {:.4f}, {:.4f}, {:.4f}, {:.4f}]'.format(*grad[:5]))
print('Expected gradients (approx) - first five values only:')
print('\t[0.0085, 0.0188, 0.0001, 0.0503, 0.0115]\n')
# Compute and display cost and gradient
# with all-ones theta and lambda = 10
test_theta = np.ones(X.shape[1])
cost, grad = costFunctionReg(test_theta, X, y, 10)
print('------------------------------\n')
print('Cost at test theta : {:.2f}'.format(cost))
print('Expected cost (approx): 3.16\n')
print('Gradient at initial theta (zeros) - first five values only:')
print('\t[{:.4f}, {:.4f}, {:.4f}, {:.4f}, {:.4f}]'.format(*grad[:5]))
print('Expected gradients (approx) - first five values only:')
print('\t[0.3460, 0.1614, 0.1948, 0.2269, 0.0922]')
2.3.1 Learning parameters using scipy.optimize.minimize¶
Similar to the previous parts, you will use optimize.minimize to learn the optimal parameters $\theta$. If you have completed the cost and gradient for regularized logistic regression (costFunctionReg) correctly, you should be able to step through the next part of to learn the parameters $\theta$ using optimize.minimize.
2.4 Plotting the decision boundary¶
To help you visualize the model learned by this classifier, we have provided the function plotDecisionBoundary which plots the (non-linear) decision boundary that separates the positive and negative examples. In plotDecisionBoundary, we plot the non-linear decision boundary by computing the classifier’s predictions on an evenly spaced grid and then and draw a contour plot where the predictions change from y = 0 to y = 1.
2.5 Effect of regularization¶
Try out different regularization parameters for the dataset to understand how regularization prevents overfitting.
Notice the changes in the decision boundary as you vary $\lambda$. With a small $\lambda$, you should find that the classifier gets almost every training example correct, but draws a very complicated boundary, thus overfitting the data. See the following figures for the decision boundaries you should get for different values of $\lambda$.
No regularization (overfitting)
|
Decision boundary with regularization
|
Decision boundary with too much regularization
|
This is not a good decision boundary: for example, it predicts that a point at $x = (−0.25, 1.5)$ is accepted $(y = 1)$, which seems to be an incorrect decision given the training set. With a larger $\lambda$, you should see a plot that shows an simpler decision boundary which still separates the positives and negatives fairly well. However, if $\lambda$ is set to too high a value, you will not get a good fit and the decision boundary will not follow the data so well, thus underfitting the data.
# Initialize fitting parameters
initial_theta = np.zeros(X.shape[1])
# Set regularization parameter lambda to 1 (you should vary this)
lambda_ = 1
res = optimize.minimize(costFunctionReg,
initial_theta,
(X, y, lambda_),
jac=True,
method='BFGS',
options={"maxiter":100, "disp":False})
# the fun property of OptimizeResult object returns
# the value of costFunction at optimized theta
cost = res.fun
# the optimized theta is in the x property of the result
theta = res.x
utils.plotDecisionBoundary(plotData, theta, X, y)
pyplot.xlabel('Microchip Test 1')
pyplot.ylabel('Microchip Test 2')
pyplot.legend(['y = 1', 'y = 0'])
pyplot.grid(False)
pyplot.title('lambda = %0.2f' % lambda_)
# Compute accuracy on our training set
p = predict(theta, X)
print('Train Accuracy: %.1f %%' % (np.mean(p == y) * 100))
print('Expected accuracy (with lambda = 1): 83.1 % (approx)\n')
