The developments of Sber and SberDevices scientists will make it possible to create new architectural solutions in the process of training generative artificial intelligence models and reduce the computational costs required for this. Representatives of Sber spoke about this topic at the international conference in the field of computational linguistics EACL 2024, held in Malta.
Sber and SberDevices researchers made a presentation on two studies on artificial intelligence.
So, the report of the head of the FusionBrain scientific group – Sberbank partner – AIRI Institute – Andrey Kuznetsov and research assistant of the group Anton Razzhigaev was as follows: special Investigation of the properties of transformer architectures of the models.
The researchers examined how important properties of embeddings (numerical representations of data) vary in two types of large language model architectures frequently used in natural language processing tasks.
The results obtained in the next stage of research will help distillate language models, that is, reduce their size with minimal loss of quality (while controlling the error variation during distillation). This is necessary to create new architectural solutions during model training and reduce the computational costs required for this.
The co-author of the study is Denis Dimitrov, General Manager of Data Research at Sberbank.
Alena Fenogenova, head of the AGI NLP team at SberDevices R&D and NLP ML engineer at Sberbank Mark Baushenko presented His research on productive approaches to spelling correction.
While working on the project, the team created a proofreading methodology and open the library A family of generative models trained on SAGE as well as datasets and the spelling correction task.
Speakers reported that the best model was superior in quality to open solutions (HunSpell, JamSpell) and OpenAI models (gpt-3.5-turbo-0301, gpt-4-0314, text-davinci-003).