Implement logistic regression from scratch in NumPy: forward pass, loss, and gradient descent.

TL;DR: Logistic regression models P(y=1) = sigmoid(Xw + b), trained by minimizing binary cross-entropy via gradient descent. The gradient simplifies beautifully: dL/dw = Xᵀ(ŷ - y) / n. Vectorize it, use a numerically stable sigmoid/loss, and you have ~15 clean lines. The signal is knowing the gradient by heart and why it is that simple.

How to approach it. State the model (sigmoid of a linear function), the loss (binary cross-entropy from maximum likelihood), and that the gradient has a clean closed form, then write it vectorized. Mention numerical stability (clip or use a stable formulation) since the interviewer often pushes on it.

A strong answer.

import numpy as np

def sigmoid(z):
    return np.where(z >= 0, 1 / (1 + np.exp(-z)),         # stable for both signs
                    np.exp(z) / (1 + np.exp(z)))

def train_logreg(X, y, lr=0.1, epochs=1000):
    n, d = X.shape
    w, b = np.zeros(d), 0.0
    for _ in range(epochs):
        z = X @ w + b
        p = sigmoid(z)                       # predicted P(y=1), shape (n,)
        grad_w = X.T @ (p - y) / n           # the clean gradient
        grad_b = (p - y).mean()
        w -= lr * grad_w
        b -= lr * grad_b
    return w, b

def predict_proba(X, w, b):
    return sigmoid(X @ w + b)

What a strong candidate explains:

• The gradient is the punchline. Binary cross-entropy L = -mean(y·log p + (1-y)·log(1-p)) with p = sigmoid(Xw+b) has gradient dL/dw = Xᵀ(p - y)/n. The sigmoid derivative and the log cancel so neatly that the gradient is just the feature matrix times the prediction error, which is why it is the same shape as the linear-regression gradient. Knowing this without rederiving each time signals fluency.
• Numerical stability. exp(-z) overflows for large negative z; the branchless np.where form (or clipping z) keeps the sigmoid stable. For the loss, prefer a log-sum-exp / log1p formulation over log(sigmoid(...)) to avoid log(0).
• Vectorization. No Python loops over examples; the whole batch is matrix ops, which is both correct and fast.
• Convexity. The loss is convex in (w, b), so gradient descent converges to the global optimum (unlike a neural net); worth stating.

The whole loop is four lines that map directly onto the math:

Key takeaways

• The gradient Xᵀ(p − y)/n is identical in shape to linear regression's; the sigmoid derivative and log cancel, so memorize it instead of rederiving.
• Use a sign-aware sigmoid (np.where) and a log1p/log-sum-exp loss to avoid exp overflow and log(0).
• The whole batch is matrix ops; never loop over examples in Python.
• The loss is convex in (w, b), so gradient descent reaches the global optimum given a sane learning rate.

What interviewers probe next.

• "Derive the gradient." Chain rule through cross-entropy and sigmoid; the sigmoid derivative p(1-p) cancels with the cross-entropy denominator, leaving (p - y).
• "Add L2 regularization." Add λw to grad_w (and (λ/2)‖w‖² to the loss); it shrinks weights and reduces overfitting.
• "Why not use accuracy as the loss?" Accuracy is non-differentiable and flat; cross-entropy is smooth and convex, giving usable gradients.
• "Multiclass?" Replace sigmoid with softmax and binary cross-entropy with categorical cross-entropy; the gradient generalizes to Xᵀ(softmax - onehot)/n.

Common mistakes.

• A naive 1/(1+exp(-z)) that overflows for large-magnitude z.
• Getting the gradient sign wrong, or not knowing it simplifies to Xᵀ(p - y)/n.
• Looping over examples instead of vectorizing.
• Computing the loss as log(sigmoid(...)) and hitting log(0) instead of a stable formulation.