Article Details

Abstract

Long-Sequence modeling is a key problem in sequence learning, with the main challenge of balancing long range dependency modeling, computational efficiency, and inference scalability as sequence length increases. In recent years, Transformers have achieved remarkable success due to the self-attention mechanism, but their quadratic complexity limits their applicability in ultra-long context scenarios. To address this issue, recursive state updating has regained attention, promoting the development of recurrent models, recurrentized attention models, and deep state space models. This paper systematically reviews the recent progress of Long-Sequence state space modeling in the Transformer era, focusing on representative methods such as Linear Transformer, RWKV, RetNet, S4, S5, LRU, and Mamba. Their structural characteristics and performance differences are analyzed, and the major challenges in model design, training paradigms, and system implementation are summarized. The study shows that multi-mechanism integration and state space modeling will become important directions for future Long-Sequence learning.

Keywords

Long-Sequence modeling; state-space model; Transformer; recurrent neural network; Mamba; linear attention

References

1. [1] Chen Jiajun, Liu Bo, Lin Weiwei, etc Overview of Time Series Prediction Methods Based on Transformer [J]. Computer Science, 2025, 52 (6): 96-105
2. [2] Elman J L. Finding Structure in Time[J].Cognitive Science，1990，14（2）：179-211.
3. [3] Hochreiter S，Schmidhuber J. Long Short-Term Memory[J].Neural Computation，1997，9（8）：1735-1780.
4. [4] Werbos P J. Backpropagation through Time: What It Does and How to Do It[J].Proceedings of the IEEE，1990，78（10）：1550-1560.
5. [5] Vaswani A，Shazeer N，Parmar N，et al. Attention Is All You Need[C]//Advances in Neural Information Processing Systems 30.Red Hook: Curran Associates，2017：5998-6008.
6. [6] Katharopoulos A，Vyas A，Pappas N，et al. Transformers Are RNNs: Fast Autoregressive Transformers with Linear Attention[C]//Proceedings of the 37th International Conference on Machine Learning.Vienna：PMLR，2020：5156-5165.
7. [7] Patro B N，Agneeswaran V S. Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges[PP/OL].arXiv.（2024-04-24）.https://arxiv.org/abs/2404.16112.
8. [8] Gu A，Goel K，Ré C. Efficiently Modeling Long Sequences with Structured State Spaces[PP/OL].arXiv.（2022-08-05）.https://arxiv.org/abs/2111.00396.
9. [9] Somvanshi S，Sharma A，Mimi M S，et al. A Comprehensive Survey on Structured State Space Models[PP/OL].arXiv.（2026-08-10）.https://arxiv.org/abs/2503.18970.
10. [10] Dai Z，Yang Z，Yang Y，et al. Transformer-XL: Attentive Language Models beyond a Fixed-Length Context[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence：Association for Computational Linguistics，2019：2978-2988.
11. [11] Peng B，Alcaide E，Anthony Q，et al. RWKV: Reinventing RNNs for the Transformer Era[PP/OL].arXiv.（2023-12-11）.https://arxiv.org/abs/2305.13048.
12. [12] Sun Y，Dong L，Huang S，et al. Retentive Network: A Successor to Transformer for Large Language Models[PP/OL].arXiv.（2023-08-09）.https://arxiv.org/abs/2307.08621.
13. [13] Ahamed M A，Cheng Q. TimeMachine: A Time Series Is Worth 4 Mambas for Long-term Forecasting[C]//Proceedings of the 27th European Conference on Artificial Intelligence (ECAI 2024), 2024. DOI: 10.3233/FAIA240677.
14. [14] Smith J T H，Warrington A，Linderman S. Simplified State Space Layers for Sequence Modeling[C]//ICLR 2023.https://openreview.net/forum?id=LjLi240wM0o.
15. [15] Orvieto A，Smith S L，Gu A，et al. Resurrecting Recurrent Neural Networks for Long Sequences[C]//Proceedings of the 40th International Conference on Machine Learning. Honolulu：PMLR，2023：26670-26698.
16. [16] Gu A，Dao T. Mamba: Linear-Time Sequence Modeling with Selective State Spaces[PP/OL].arXiv.（2024-05-31）.https://arxiv.org/abs/2312.00752.
17. [17] Dao T，Gu A. Transformers Are SSMs: Generalized Models and Efficient Algorithms through Structured State Space Duality[PP/OL].arXiv.（2024-05-31）.https://arxiv.org/abs/2405.21060.
18. [18] Yang S，Wang B，Shen Y，et al. Gated Linear Attention Transformers with Hardware-Efficient Training[PP/OL].arXiv.（2024-08-27）.https://arxiv.org/abs/2312.06635.
19. [19] Dehghani M，Orvieto A，Goyal A，et al. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models[PP/OL].arXiv.（2024-02-29）.https://arxiv.org/abs/2402.19427.
20. [20] Beck M，Pöppel K，Spanring M，et al. xLSTM: Extended Long Short-Term Memory[PP/OL].arXiv.（2024-12-06）.https://arxiv.org/abs/2405.04517.
21. [21] Tay Y，Dehghani M，Abnar S，et al. Long Range Arena: A Benchmark for Efficient Transformers[PP/OL].arXiv.（2020-11-08）.https://arxiv.org/abs/2011.04006.
22. [22] Chen Y，Zhang X，Hu S，et al. Stuffed Mamba: State Collapse and State Capacity of RNN-Based Long-Context Modeling[PP/OL].arXiv.（2026-01-13）.https://arxiv.org/abs/2410.07145.
23. [23] Rusch T K，Rus D. Oscillatory State-Space Models[PP/OL].arXiv.（2025-06-18）.https://arxiv.org/abs/2410.03943.
24. Xia Xue, Yan Enlai, Li Xiwu Overview of the Application of Transformer in Time Series Prediction [J]. Information Technology and Informatization, 2024 (3): 124-128
25. Zou Maoyang, Wu Yulan, Gao Lin, etc Research progress of state space models in medical image processing [J]. Chinese Journal of Image and Graphics, 2025, 30 (9): 3111-3126
26. Xiao Jie, Fan Zihao, Li Dong, etc Non causal selective state space model for image restoration [J]. Chinese Journal of Image and Graphics, 2025, 30 (10): 3173-3186
27. [27] Zhang Yijun, Zhang Runqing, Zhou Huajian, etc Research Status and Development Trends of Visual Basic Models [J]. Chinese Journal of Image and Graphics, 2025, 30 (1): 1-24
28. [28] Jafari F R，Montavon G，Müller K R，et al. MambaLRP: Explaining Selective State Space Sequence Models[PP/OL].arXiv.（2025-01-15）.https://arxiv.org/abs/2406.07592.

About the authors

Yu Bowen (2000-), male, Han ethnicity, from Hangzhou, Zhejiang, master's degree, research interests include machine learning and artificial intelligence; Ju Li (2001-), female, Han ethnicity, from Shanghai, master's degree, research interests include machine learning and artificial intelligence.