80How do you choose the data mixture for pretraining an LLM, and what does domain reweighting buy you?▼hardGoogle DeepMindMetaMistral1 replies◆ premiumThe ratio of web, code, books, and math in pretraining quietly decides downstream skills. The signal is knowing how mixtures are chosen, why upsampling helps, and how methods like DoReMi automate it.Open full answer →
81How do you do continued pretraining to adapt an LLM to a new domain without forgetting general ability?▼hardMetaDatabricksSnowflake1 replies◆ premiumContinued pretraining injects domain knowledge that fine-tuning cannot, but naively it wrecks general ability. The signal is the replay ratio, learning-rate rewarming, and how you measure forgetting.Open full answer →
