← All research spotlights

Research spotlight · Unit 10

Sepp Hochreiter

Johannes Kepler University Linz

Research field
Recurrent neural networks, the LSTM, machine learning

Sepp Hochreiter diagnosed the vanishing gradient of recurrent networks in his diploma thesis of 1991 and, with Jürgen Schmidhuber, published the long short-term memory network in 1997, the architecture that ran translation, speech recognition, and text prediction until the transformer arrived. He leads the Institute for Machine Learning at the Johannes Kepler University Linz. In a 2024 paper with eight co-authors he asked how far an LSTM gets when it is scaled to billions of parameters with exponential gates and a matrix memory, and found that the resulting xLSTM matches or beats transformers and state space models of the same size in language modelling on 15 billion and 300 billion tokens. The unit presents it after the LSTM section, as the question whether the recurrent network has a future at today's scale.

In the masterclass

Section 4, after the LSTM cell and the worked example, presents his 2024 return to the architecture he co-invented.

The research is presented in Unit 10: Embeddings and Sequence Models, in a research spotlight box that explains the result at the level of the unit and links it to what you have just learned.

The paper

Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., and Hochreiter, S. (2024). xLSTM: Extended Long Short-Term Memory. Advances in Neural Information Processing Systems, 37, arXiv:2405.04517. https://doi.org/10.48550/arXiv.2405.04517

The scientists named here are not contributors to the AI Masterclass, are not affiliated with it, and have not endorsed it. The program presents and explains their published research, with references to the original papers.