TL;DR: Deep learning is mostly large matrix multiplications, which are massively parallel, and GPUs are throughput machines: thousands of simple cores plus high-bandwidth memory and dedicated matrix units (tensor cores). CPUs are latency-optimized for few, complex, branchy threads. TPUs are Google's ASICs built around a systolic array specialized for matrix multiply, even more specialized than GPUs. The recurring constraint across all three is the memory hierarchy: keep data in fast on-chip memory and off slow HBM.
How to approach it. Anchor on the workload (DL is big dense matmuls, embarrassingly parallel), then map each processor's design philosophy (latency vs throughput vs specialized ASIC) onto that workload, and bring in the memory hierarchy because that is what actually bounds performance.
A strong answer. Why GPUs fit DL. Training and inference are dominated by large matrix multiplications and elementwise ops over big tensors, which decompose into many independent multiply-accumulates. That is data parallelism at the arithmetic level, and GPUs are built for it: thousands of relatively simple cores executing the same instruction across many data elements (SIMT), organized into warps and streaming multiprocessors (SMs). Modern GPUs add tensor cores, hardware units that do a small matrix multiply-accumulate in one operation (precisely the DL primitive), plus high-bandwidth memory (HBM) to feed all those cores. So a GPU trades single-thread speed for enormous aggregate throughput.
CPU vs GPU vs TPU. The three sit on a spectrum from general and latency-optimized to specialized and throughput-optimized:
| Design point | Cores | Best at | DL role | |
|---|---|---|---|---|
| CPU | latency-optimized | few, complex | branchy serial logic | data loading, orchestration |
| GPU | throughput-optimized | thousands, simple (SIMT) | dense parallel matmul | training and inference workhorse |
| TPU | specialized ASIC | systolic MAC array | fixed matmul shapes | matmul-heavy models at scale |
A CPU has a few powerful cores with big caches, branch prediction, and out-of-order execution, designed to run complex sequential logic fast. A GPU has many simple cores that crunch huge regular workloads in parallel, tolerating latency with thousands of threads in flight. CPUs win on serial and control-heavy code; GPUs win on the dense parallel math of DL. (A GPU starved while it waits on CPU preprocessing is a common bottleneck.)
TPU. Google's Tensor Processing Unit is an ASIC purpose-built for neural-network math, organized around a systolic array: a grid of multiply-accumulate units through which data flows so matrix multiplications happen with minimal memory movement. It is more specialized than a GPU, paired with software like JAX and XLA. The tradeoff is flexibility vs efficiency: GPUs are general accelerators with a huge software ecosystem; TPUs squeeze more performance-per-watt for the specific shapes they target.
The unifying constraint: memory hierarchy. All of them have fast, small on-chip memory (registers, SRAM, shared memory) and large, slower off-chip memory (HBM). The performance battle is keeping data in fast memory and minimizing trips to HBM, which is exactly why kernel fusion, tiling, and IO-aware algorithms (FlashAttention) matter, and why so much DL inference is memory-bandwidth-bound rather than compute-bound.
Key takeaways
- CPU optimizes latency for a few branchy threads; GPU optimizes throughput across thousands of simple ones; TPU specializes further into a fixed matmul array.
- Tensor cores and the systolic array exist for the same reason: matrix multiply-accumulate is the DL primitive worth hardwiring.
- The binding constraint on all three is the memory hierarchy: fast SRAM is scarce, HBM trips are expensive.
- GPU vs TPU is flexibility and ecosystem vs performance-per-watt on the shapes the ASIC targets.
What interviewers probe next.
- "What is SIMT and a warp?" Single-Instruction-Multiple-Threads: threads execute in lockstep groups (warps of 32); divergent branches within a warp serialize (warp divergence), hurting throughput.
- "Why are tensor cores a big deal?" They do a small matrix multiply-accumulate per instruction (often in mixed precision), the core DL op, far faster than general FP units.
- "GPU vs TPU tradeoff?" GPU is flexible with a broad ecosystem; TPU is more specialized systolic-array efficiency for matmul-heavy models, less general.
- "Why is inference often memory-bound?" Decode reads weights and KV cache per token; you are limited by memory bandwidth, not FLOPs, which is why quantization (fewer bytes) speeds it up.
Common mistakes.
- "GPUs are just faster CPUs"; they are a different (throughput vs latency) design, not a faster version of the same thing.
- Ignoring the memory hierarchy, which is what actually bounds most DL kernels.
- Not knowing what tensor cores or the systolic array do (the DL-specific hardware).
- Forgetting the CPU's role (data loading, orchestration) and that it is a common bottleneck when it cannot feed the GPU.