Оптимізаційні методи для навчання генеративних великих мовних моделей

Abstract

This comprehensive analysis explores advanced optimization methods for training Generative Large Language Models (LLMs) to overcome profound memory and computational bottlenecks. It reviews the evolution from foundational first-order techniques, like AdamW, to memory-efficient alternatives such as Adafactor and Lion. The text further examines second-order and preconditioned strategies (e.g., Sophia, SOAP, Muon) that adapt to landscape curvature, alongside low-rank subspace projection frameworks like GaLore that drastically reduce memory footprints. Additionally, it highlights temporal dynamics methods like MARS, Schedule-Free, and Grokfast for accelerated convergence.