TL;DR: Do not grid-search. Random search beats grid in high dimensions because only a few hyperparameters matter; Bayesian optimization is more sample-efficient but harder to parallelize; and early-stopping schemes (Hyperband/ASHA) give the biggest practical win by killing bad trials early and reallocating GPUs to promising ones. On a cluster, run an asynchronous scheme (ASHA) so fast workers never wait on slow trials.
How to approach it. Reject grid search first and say why, then walk the three families (random, Bayesian, early-stopping bandits) along two axes that actually decide the choice: sample efficiency and parallelizability. Close on the cluster-specific concern, asynchronous scheduling to avoid stragglers. The interviewer is screening for efficient use of expensive GPU time, not exhaustive coverage of a search space.
A strong answer. Why not grid search. Grid scales exponentially with the number of hyperparameters and burns most of its budget on dimensions that do not move the metric. Random search covers each important dimension more densely for the same number of trials, because it never wastes evaluations tiling a fine grid over an irrelevant parameter. It is the right cheap baseline.
Bayesian optimization. Fit a probabilistic surrogate of "config to validation metric" (a Gaussian process, or a tree-based model like TPE), then use an acquisition function to pick the next config that balances exploring uncertain regions against exploiting the best-so-far. Far more sample-efficient than random, which matters when one trial is an expensive multi-hour run. The catch: it is sequential by construction (each suggestion depends on prior results), so naive parallelism suggests near-duplicate configs. Batched and async variants exist but the efficiency edge softens at high parallelism.
Early-stopping bandits (the practical winner at scale). Hyperband and especially ASHA (Asynchronous Successive Halving) treat tuning as resource allocation, not search. Start many configs with a tiny budget (a few epochs), periodically promote the top fraction to more budget, and kill the rest. Most bad configs declare themselves quickly, so GPUs stop pouring time into them. ASHA removes the synchronization barrier: a free worker promotes or launches a config immediately instead of waiting for a generation to finish. That is exactly what prevents stragglers on a heterogeneous cluster, one slow trial never stalls the pool. Pair ASHA's stopping with Bayesian or random sampling of the configs themselves to get both levers.
| Family | Sample efficiency | Parallelizes cleanly | Best for |
|---|---|---|---|
| Random search | Low | Yes | Cheap baseline, high dimensions |
| Bayesian (GP / TPE) | High | Weakly (sequential) | Few expensive trials |
| Hyperband / ASHA | High per GPU-hour | Yes (ASHA, async) | Large GPU clusters |
Cluster practicalities. A tuning orchestrator (Ray Tune, Optuna, Vizier-style) distributes trials; asynchronous scheduling keeps workers busy; checkpoint every trial so a promotion resumes rather than restarts; cap the total budget up front. Tune on a representative subset or reduced epochs where the ranking of configs is preserved.
Key takeaways.
- Grid wastes compute exponentially; random search is the correct cheap baseline because few hyperparameters matter.
- Bayesian optimization buys sample efficiency at the cost of clean parallelism.
- ASHA's async successive halving is the biggest per-GPU-hour win and is straggler-proof on a heterogeneous cluster.
- Checkpoint trials so promoted configs resume, and never tune so hard that you overfit one validation split.
What interviewers probe next.
- "Why does random search beat grid?" Few hyperparameters matter; random samples each important dimension more densely for the same budget, while grid spends evaluations on irrelevant ones.
- "Bayesian opt parallelization problem?" It is sequential (next suggestion uses prior results), so running many in parallel suggests redundant configs; use batched/async variants or pair it with bandits.
- "What makes ASHA good on a cluster?" No sync barrier: workers promote good trials and kill bad ones independently, so a slow trial never stalls everyone.
- "How do you avoid overfitting the validation set?" Use proper CV and a held-out test untouched by tuning; thousands of trials against one split will fit its noise.
Common mistakes.
- Defaulting to grid search, which wastes compute exponentially.
- Ignoring early-stopping (Hyperband/ASHA), the single biggest efficiency lever.
- Running a synchronous scheme on a cluster, where one slow trial stalls every worker.
- Tuning so hard against one validation split that you overfit it.