Probability Distributions You Should Know

TL;DR: Match the distribution to the data-generating process: Bernoulli for a single yes/no, binomial for the count of yeses in n trials, normal for things that are sums or averages of many small effects (and measurement noise), Poisson for counts of rare events in a fixed window, and exponential for the waiting time until the next such event. The payoff is that each distribution's negative log-likelihood is a loss function you already use: Bernoulli is binary cross-entropy, normal is mean-squared error, so assuming a distribution is the same as choosing a loss.

The five and what they model

The fastest way to pick a distribution is to ask what the data physically is: a single binary outcome, a count, a duration, or a continuous measurement.

Bernoulli is the atom: one trial, outcome 0 or 1, probability p of a 1. A click, a conversion, a fraud label. Binomial is n independent Bernoulli trials summed: out of 1000 emails, how many bounce. Normal shows up by the central limit theorem: anything that is a sum or average of many independent small effects tends to look Gaussian, which is why measurement noise and aggregate metrics are modeled as normal. Poisson counts how many independent rare events land in a fixed interval (support tickets per hour, photons per pixel) with a single rate λ that is both the mean and the variance. Exponential is the flip side of Poisson: if events arrive at Poisson rate λ, the gap between consecutive events is exponential with mean 1/λ, and it is memoryless (the expected wait does not depend on how long you have already waited).

Counts versus durations is the cleanest dividing line. "How many in this window" is Poisson. "How long until the next one" is exponential. Same underlying process, two questions.

Worked example: a support queue

A help desk gets tickets at an average of λ = 12 per hour.

• Probability of exactly 8 tickets next hour (Poisson): P(8) = e^-12 · 12^8 / 8! ≈ 0.066.
• Expected wait for the next ticket (exponential): 1/λ = 1/12 hour, that is 5 minutes.
• Probability the next ticket takes more than 10 minutes (exponential survival): e^(-12 · 1/6) = e^-2 ≈ 0.135.

One rate parameter answers both the count question and the timing question, which is the practical reason these two travel together.

Distributions become losses

This is the part interviews care about. Maximum-likelihood estimation says: pick parameters that make the observed data most probable, which means minimizing the negative log-likelihood. Plug in a distribution and a loss falls out.

• Assume each label is Bernoulli with the model's predicted p: the negative log-likelihood is -[y log p + (1-y) log(1-p)], exactly binary cross-entropy. Multi-class generalizes to categorical and gives softmax cross-entropy.
• Assume the target is normal around the model's prediction with fixed variance: the negative log-likelihood reduces to (y - ŷ)² up to constants, exactly mean-squared error.
• Assume Poisson counts: you get Poisson (log-link) regression, the standard model for count targets.

So a classifier trained with cross-entropy is implicitly modeling labels as Bernoulli, and a regressor with MSE is assuming Gaussian noise. When the assumption is wrong (heavy tails, count data with MSE), the loss is misspecified and predictions are biased or poorly calibrated.

Why interviewers probe this

This is a fast filter for whether you understand what a model fits versus just which API to call. The interviewer screens for the link from distribution to loss: state that cross-entropy is the Bernoulli/categorical NLL and MSE is the Gaussian NLL, unprompted. The strong-answer move is to name the data-generating process and then derive the loss, instead of treating the loss as arbitrary. A held-back follow-up: "your target is a count that is mostly zeros with occasional spikes, why is MSE a bad fit and what would you use?" (heavy skew, MSE assumes symmetric Gaussian noise; reach for Poisson or negative-binomial regression, possibly zero-inflated).

Common misconceptions

• "Everything is normal if you have enough data." The CLT applies to sums and averages of many independent effects, not to raw skewed quantities like incomes, durations, or counts.
• "Poisson and binomial are unrelated." Binomial with large n and small p converges to Poisson with λ = np; Poisson is the rare-event limit.
• "Exponential waiting times have a typical scale you can wait out." Exponential is memoryless: having waited 20 minutes does not make the next event sooner, the expected remaining wait is still 1/λ.
• "The loss function is a free choice." It encodes a noise assumption. MSE assumes Gaussian errors, cross-entropy assumes Bernoulli/categorical labels; mismatched losses give biased, miscalibrated models.

Key takeaways

• Pick by data type: Bernoulli/binomial for yes-no and counts of successes, normal for sums and noise, Poisson for event counts per window, exponential for waiting times.
• Poisson and exponential are two views of one process: counts in a window versus time between events, sharing the rate λ.
• Cross-entropy is the Bernoulli/categorical negative log-likelihood; MSE is the Gaussian one. Choosing a distribution chooses a loss.
• A misspecified noise assumption (MSE on skewed counts) produces biased and poorly calibrated predictions.