sapiСистемный анализ и прикладная информатика«System analysis and applied information science»2309-49232414-0481Belarusian National Technical University10.21122/2309-4923-2026-2-78-81sapi-817Research ArticleИнформационные технологииInformation technologiesЭволюция архитектур трансформеров: от самовнимания к эффективным Sparse-моделямEvolution of transformer architectures: from self-attention to efficient sparse modelsГуляевВ. О.GulyaevV. O.

Гуляев Владислав Олегович - Магистрант.г. СамараE-mail: vladislavgulaev03@gmail.com

Vladislav O. Gulyaev - Master's student.SamaraE-mail: vladislavgulaev03@gmail.com

vladislavgulaev03@gmail.comЗахароваО. И.ZakharovaO. I.

Захарова Оксана Игоревнаг. СамараE-mail: o.zaharova@psuti.ru

Oksna I. Zaharova SamaraE-mail: o.zaharova@psuti.ru

o.zaharova@psuti.ruПоволжский государственный университет телекоммуникаций и информатикиРоссияVolga State University of Telecommunications and InformaticsRussian Federation202617072026027881Copyright © Гуляев В.О., Захарова О.И., 20262026Гуляев В.О., Захарова О.И.Gulyaev V.O., Zakharova O.I.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://sapi.bntu.by/jour/article/view/817

Статья посвящена исследованию новых архитектур искусственных нейронных сетей, направленных на улучшение классических трансформеров путем уменьшения вычислительных затрат и увеличения эффективности. Рассматриваются основные этапы эволюции трансформеров, начиная с введения механизма самовнимания и заканчивая современными Sparse-моделями. Особое внимание уделяется методам оптимизации, таким как параметрически эффективный fine-tuning (PEFT), адаптивные слои (adapters), а также использование внешних хранилищ знаний и mem-векторов для расширения долговременной памяти моделей. Статья завершается обсуждением текущих достижений, ограничений и перспективных направлений будущих исследований в данной области.

The article is devoted to the study of new architectures of artificial neural networks aimed at improving classical transformers by reducing computational costs and increasing efficiency. The main stages of the evolution of transformers are considered, starting with the introduction of the mechanism of self-attention and ending with modern sparse models. Particular attention is paid to optimization techniques such as parametrically efficient fine-tuning (PEFT), adaptive layers (adapters), as well as the use of external knowledge repositories and memory vectors to expand the long-term memory of models. The article concludes with a discussion of current achievements, limitations, and promising areas for future research in this area.

трансформерымеханизм самовниманияsparse-моделиPEFTmem-векторыtransformersself-attention mechanismsparse modelsPEFTmemory vectorsReferencesVaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A.N., et al. Attention is all you need. arXiv [Preprint]. 2017. Available from: https://arxiv.org/abs/1706.03762 (accessed 12 Jun 2026).Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A.N., et al. Attention is all you need. arXiv [Preprint]. 2017. Available from: https://arxiv.org/abs/1706.03762 (accessed 12 Jun 2026).Child R., Gray S., Radford A., Sutskever I. Generating long sequences with sparse transformers. arXiv [Preprint]. 2019. Available from: https://arxiv.org/abs/1904.10509 (accessed 12 Jun 2026).Child R., Gray S., Radford A., Sutskever I. Generating long sequences with sparse transformers. arXiv [Preprint]. 2019. Available from: https://arxiv.org/abs/1904.10509 (accessed 12 Jun 2026).Hu E.J., Shen Y., Wallis P., Allen-Zhu Z., Li Y., Wang S., et al. LoRA: Low-rank adaptation of large language models. arXiv [Preprint]. 2021. Available from: https://arxiv.org/abs/2106.09685 (accessed 12 Jun 2026).Hu E.J., Shen Y., Wallis P., Allen-Zhu Z., Li Y., Wang S., et al. LoRA: Low-rank adaptation of large language models. arXiv [Preprint]. 2021. Available from: https://arxiv.org/abs/2106.09685 (accessed 12 Jun 2026).Bulatov A., Kuratov Y., Burtsev M.S. Recurrent memory transformer. arXiv [Preprint]. 2022. Available from: https://arxiv.org/abs/2207.06881 (accessed 12 Jun 2026).Bulatov A., Kuratov Y., Burtsev M.S. Recurrent memory transformer. arXiv [Preprint]. 2022. Available from: https://arxiv.org/abs/2207.06881 (accessed 12 Jun 2026).Ramesh A., Pavlov M., Goh G., Scott G., Voss C., Radford A., et al. Zero-shot text-to-image generation. arXiv [Preprint]. 2021. Available from: https://arxiv.org/abs/2102.12092 (accessed 12 Jun 2026).Ramesh A., Pavlov M., Goh G., Scott G., Voss C., Radford A., et al. Zero-shot text-to-image generation. arXiv [Preprint]. 2021. Available from: https://arxiv.org/abs/2102.12092 (accessed 12 Jun 2026).Dhariwal P., Jun H., Payne C., Kim J.W., Radford A., Sutskever I. Jukebox: A Generative model for music. arXiv [Preprint]. 2020. Available from: https://arxiv.org/abs/2005.00341 (accessed 12 Jun 2026).Dhariwal P., Jun H., Payne C., Kim J.W., Radford A., Sutskever I. Jukebox: A Generative model for music. arXiv [Preprint]. 2020. Available from: https://arxiv.org/abs/2005.00341 (accessed 12 Jun 2026).Brown T.B., Mann B., Ryder N., Subbiah M., Kaplan J., Dhariwal P., et al. Language models are few-shot learners. arXiv [Preprint]. 2020. Available from: https://arxiv.org/abs/2005.14165 (accessed 12 Jun 2026).Brown T.B., Mann B., Ryder N., Subbiah M., Kaplan J., Dhariwal P., et al. Language models are few-shot learners. arXiv [Preprint]. 2020. Available from: https://arxiv.org/abs/2005.14165 (accessed 12 Jun 2026).

The authors declare that there are no conflicts of interest present.