Design the training infrastructure for RLHF/PPO. Why are there four model copies and how do you fit them?

RLHF with PPO is not one model training, it is four models in the loop at once, three of them on the GPU during every step. The memory math and the generation bottleneck are what trip people up.

Updated Aug 2026 · Grounded in real Applied AI Engineer interview loops and written to a senior-engineer editorial bar.