Порівняльний аналіз архітектур Transformer та Mamba в задачах обробки природної мови

Abstract

This work is devoted to a comparative analysis of two prominent neural network architectures, Transformer and Mamba, in the context of natural language processing tasks. The study examines the fundamental limitations of the Transformer model, particularly its quadratic computational complexity O(L^2) regarding sequence length, which poses significant challenges for processing long-range dependencies. In contrast, the newly emerged Mamba architecture, based on Selective State Space Models (SSM), is analyzed for its ability to achieve linear scaling O(L) and superior inference speed. The research provides an empirical evaluation of both models across various benchmarks, highlighting their efficiency in memory consumption and text generation quality. Special attention is paid to the trade-offs between the established attention mechanisms and the innovative selective recurrence approach. The findings suggest that while Transformers remain a standard for complex reasoning, Mamba offers a promising and hardware-efficient alternative for future large-scale language modeling.