The language model of Sber and SberDevices has become the best in the world for understanding Russian texts. This conclusion was made by specialists of Russian SuperGLUE, the main criterion of Russian-speaking. This was reported by the press service of the bank.
The conclusion regarding the advantage of Sber and SberDevices solution is made according to the results. tests for evaluating large text models. Experts, model FRED-T5 (Full Scale Russian Advanced Noise Cancellers T5) is second only to humans in terms of accuracy.
Sber notes that the company has been working with transformer models for a long time – in 2019 the Russian models ruBERT and ruGPT-2 were trained, and in 2020 ruGPT-3 was created using the Christofari supercomputer.
The company said that unlike the models of the GPT family, which only consists of transformer decoding blocks, the FRED-T5 model also includes encoding blocks that allow it to solve various tasks in the field of natural language processing much more efficiently.
As Sergey Markov, director of SberDevices’ Department of Experimental Machine Learning Systems, noted, leading research centers in the field of machine learning have been building much larger models of neural language in recent years.
According to him, the number of parameters of the largest monolithic neural networks has already exceeded 500 billion and continues to grow. These are computing projects unprecedented in human history.
“But the progress lies not only in the creation of larger neural network monsters, but also in the development of network architectures and the methods to train them. Thus, most modern models with the same number of parameters are intellectually superior to their predecessors. A good example of this effect is by today’s standards. It is the FRED-T5 neural network that has become the leader in understanding the Russian language with a relatively modest number of parameters,” adds Sergey Markov.
Leaderboard Russian SuperGLUE (General Language Comprehension Assessment) is the first rating of neural networks for the Russian language. The place in the rating depends on how well the neural network performs the tasks of logic, common sense, goal setting and understanding the meaning of the text. Sber explains that this is an open project used by data scientists working with Russian neural networks.
Source: Gazeta
Jackson Ruhl is a tech and sci-fi expert, who writes for “Social Bites”. He brings his readers the latest news and developments from the world of technology and science fiction.