Linear and Logistic Regression

TL;DR: Linear regression predicts a continuous value as a weighted sum of features and is fit by minimizing squared error (least squares). Logistic regression feeds that same linear score through a sigmoid to get a probability between 0 and 1, then fits the weights by minimizing cross-entropy. Both give coefficients you can read directly, and logistic regression remains the production default for binary classification when you need a well-calibrated probability rather than just a label.

Two models, one linear core

Both models compute the same thing first: a linear score z = w·x + b, a dot product of learned weights with the feature vector plus a bias. The difference is what happens to z.

• Linear regression uses z directly as the prediction of a continuous target (price, demand, latency).
• Logistic regression passes z through the sigmoid σ(z) = 1 / (1 + e^(-z)), mapping any real number to (0, 1), and treats the result as P(y=1 | x).

That shared linear core is why they are taught together and why the coefficient interpretation rhymes.

Least squares and what the coefficients mean

Linear regression minimizes the sum of squared residuals, Σ (yᵢ - ŷᵢ)². Squaring (rather than absolute error) makes the loss smooth and gives a closed-form solution, w = (XᵀX)⁻¹ Xᵀy, the normal equations. In practice you solve it with QR or SVD, not a literal matrix inverse, and on large or streaming data you use gradient descent instead.

A coefficient wⱼ reads as: holding the other features fixed, a one-unit increase in feature j changes the prediction by wⱼ. Concrete example: predicting house price with price = 50,000 + 120·sqft + 8,000·bedrooms. Each extra square foot adds 120 dollars, each bedroom adds 8,000, all else equal. That "all else equal" clause is exactly where collinearity bites (below).

The sigmoid and cross-entropy objective

For logistic regression you cannot use squared error: it makes the loss non-convex in the weights and punishes confident-correct predictions oddly. Instead you maximize the likelihood of the observed labels, which is equivalent to minimizing binary cross-entropy:

L = -(1/N) Σ [ yᵢ·log(pᵢ) + (1 - yᵢ)·log(1 - pᵢ) ]

This is convex, so gradient descent reaches the global optimum. The coefficients are interpreted in log-odds: wⱼ is the change in log(p/(1-p)) per unit of feature j. Exponentiate it and you get an odds ratio. A coefficient of 0.7 on a churn feature means e^0.7 ≈ 2.0, so a one-unit increase roughly doubles the odds of churn.

Why logistic regression is still the production default

For binary classification where you need a score, not just a yes/no, logistic regression is hard to beat as a baseline and often as the shipped model. Reasons that survive contact with production:

• Calibrated by construction. Minimizing cross-entropy directly optimizes for probabilities that match observed frequencies. A logistic model that says 0.3 tends to be right about 30 percent of the time. Tree ensembles and neural nets usually need a separate calibration step (Platt scaling, isotonic regression).
• Interpretable and auditable. You can hand a regulator or a fraud analyst the coefficients and odds ratios. This matters in credit, insurance, and healthcare.
• Cheap and stable. Microsecond inference, trivial to retrain, easy to monitor for drift one coefficient at a time.

The decision rule: reach for logistic regression first when the features are reasonably engineered and you need a trustworthy probability. Switch to gradient-boosted trees when nonlinear interactions dominate and you can afford a calibration layer.

Key assumptions and where they break

Collinearity is the one that surprises people: the predictions stay fine, but individual coefficients become meaningless because the model cannot tell two correlated features apart. Check variance inflation factors and drop or combine the offenders. Regularization (L2/ridge) stabilizes the coefficients; L1/lasso zeroes some out for sparsity.

Why interviewers probe this

This screens for whether you understand the model under the hood rather than calling .fit(). The strong-answer move is to state the shared linear core, then name the two different objectives (squared error vs cross-entropy) and why logistic needs cross-entropy (convexity, calibration). The held-back follow-up is usually "why not use squared error for classification?" or "interpret this coefficient", and a weaker candidate fumbles the log-odds interpretation. Another reserved probe: "your AUC is great but the model is useless in production, why?", fishing for calibration versus ranking.

Common misconceptions

• "Logistic regression is a regression model." It is a classification model; the name refers to the logit (log-odds) being modeled linearly.
• "You fit logistic regression with least squares." No. Squared error on probabilities is non-convex and poorly calibrated; you minimize cross-entropy.
• "A high coefficient means a feature is important." Only after scaling features and ruling out collinearity. Unscaled coefficients reflect units as much as effect.
• "Linear models can only fit straight lines." They are linear in the parameters. Add polynomial or interaction terms and they fit curves; the math stays the same.

Key takeaways

• Both models compute w·x + b; linear regression uses it directly (squared-error loss), logistic regression squashes it through a sigmoid (cross-entropy loss).
• Linear coefficients are per-unit changes in the target; logistic coefficients are changes in log-odds, so exponentiate them to get odds ratios.
• Logistic regression is the production default for binary classification because cross-entropy yields calibrated probabilities for free, plus interpretability and cheap inference.
• Watch collinearity: it wrecks coefficient interpretation while leaving predictions intact; regularize or combine correlated features.