logstic regression

Logistic regression makes use of the sigmoid function which outputs a probability between 0 and 1.

h(x(i),theta) = 1 / (1+e-theta*x(i))

input: x(i) & theta
output y_predict
cost function: y_predict vs y

# find theta that minimizes the cost function

(1) initialize theta                              theta
(2) calculate y_predict using initial theta       h=h(X,theta)
(3) calculate the gradient of cost function       gradient=(1/m)*X*(h-y)
(4) update theta                                  theta=theta-alpha*gradient
(5) calculate loss                                J(theta)
(6) max number of iterations or stop-parameter

********************************************************************************************************************************************************************

def sigmoid(z): 
    '''
    Input:
        z: is the input (can be a scalar or an array)
    Output:
        h: the sigmoid of z
    '''
    
    # calculate the sigmoid of z
    h = 1 / (1 + np.exp(-z))
    
    return h

---------------------------------------------------------------------------------------------------------------------------

def gradientDescent(x, y, theta, alpha, num_iters):
    '''
    Input:
        x: matrix of features which is (m,n+1)
        y: corresponding labels of the input matrix x, dimensions (m,1)
        theta: weight vector of dimension (n+1,1)
        alpha: learning rate
        num_iters: number of iterations you want to train your model for
    Output:
        J: the final cost
        theta: your final weight vector
    Hint: you might want to print the cost to make sure that it is going down.
    '''
    # get 'm', the number of rows in matrix x
    m = x.shape[0]
    
    for i in range(0, num_iters):
        
        # get z, the dot product of x and theta
        z = np.dot(x,theta)
        
        # get the sigmoid of h
        h = sigmoid(z)
        
        # calculate the cost function
        # note that we can use also np.array.transpose() instead of np.array.T
        # np.array.T just makes code a little more readable :)
        J = -1./m * (np.dot(y.T, np.log(h)) + np.dot((1-y).T,np.log(1-h)))                                                    

        # update the weights theta
        theta = theta - (alpha/m) * np.dot(x.T,(h-y))
        
    J = float(J)
    return J, theta
    
# train logstic regression model
To train the model:
- Stack the features for all training examples into a matrix X.
- Call gradientDescent, which you've implemented above.
  
# collect the features 'x' and stack them into a matrix 'X'
X = np.zeros((len(train_x), 3))
for i in range(len(train_x)):
    X[i, :]= extract_features(train_x[i], freqs)

# training labels corresponding to X
Y = train_y

# Apply gradient descent
J, theta = gradientDescent(X, Y, np.zeros((3, 1)), 1e-9, 1500)

# predict on new instance
y_pred = sigmoid(np.dot(x_new,theta))

# caculae accuracy
accuracy = (y_hat==np.squeeze(test_y)).sum()/len(test_x)
 
# error analysis
if np.abs(y - (y_hat > 0.5)) > 0: