Pre-Trained AI Language Models – How LEAM is Bringing Them to SMEs

Text-based information systems help us in all areas where communication happens between people and between people and machines. Especially when it comes to reading and processing many texts by machine, automated AI language models come into play. 

What challenges do companies face when they want to integrate smart models into their business? And how can these challenges be solved from a scientific perspective? 

In this interview, eco’s Hanna Sissmann talks to Prof. Alexander Löser, who researches text-based information systems and machine learning at the Berlin University of Applied Sciences (BHT), about large AI language models and the European LEAM initiative.

Prof. Löser, you have been researching machine learning for almost ten years. In which areas can text-based information systems help us in particular?

Löser: The effectiveness and robustness of the models have increased significantly over the last ten years. Meanwhile, with language models, we can do much better in scenarios with few and rare data. This can be in very different areas. For example, in medicine, mechanical engineering, supply chain management or on social media platforms and in the entertainment sector.

What is the focus of science on large AI language models and in which areas do you think there is still a great need for knowledge?

Löser: Among other things, we deal with the adaptation of pre-trained language models to specific domains, such as health or supply chain management. Recently, Bloom, a language model with 176 billion parameters, was published. It is open source and allows companies to customise it on their own servers. However, we need many more of these pre-trained language models, especially in Germany and Europe, because Bloom, unfortunately, does not exist in German and other European languages. Multilingualism, multimodality, robustness and adaptivity are the most urgent aspects of large language models that we need to further research and improve in the coming years.

How can we better transfer knowledge from research into practice?

Löser: The LEAM Initiative promotes the availability of large AI language models for small and medium-sized enterprises (SMEs), in particular. To this end, the initiative is developing pre-trained multilingual language models with which companies can train their own models and adapt them to their respective domains with little effort. However, this adaptation to a subject domain is still a major challenge, both in the area of methods  and in accessing the often limited training data. The German economy is particularly strong in industrial sectors such as mechanical engineering, automotive and chemicals. Pre-trained AI language models need to be more readily available to businesses. Models like Bloom are often still aimed at the B2C sector, but there is still a clear need here for the typically European B2B domains.

What challenges do German companies face when they want to use and adapt pre-trained language models?

Löser: Many companies do not yet know exactly where and how they can effectively use machine learning methods to save costs and conquer markets. When companies produce pumps or measurement technology, for example, their core business is the hardware they produce. However, pre-trained language models are often found in surrounding business areas, such as purchasing, supplier selection, market monitoring or even service. However, these surrounding business areas are exactly what can bring them great competitive advantages, such as higher customer loyalty.

Doesn’t that mean high initial costs for companies?

Löser: Language models are, of course, a significant investment, the costs of which companies have to recoup. This willingness to invest is currently still low in many markets, especially among SMEs. Around 70 per cent of the main costs are incurred in data procurement and data cleansing. Error analysis, adaptation of the models and deployment in the robust application incur the rest. To reduce costs, companies should, therefore, procure data cheaply or develop better processes to cleanse their data and deploy models, so-called MLOPs. This is where the data-centric approach of the LEAM initiative comes into play. This includes, for example, pre-trained and easily adaptable models for training or easy-to-use infrastructures for training and deployment. LEAM’s cost reduction for data products requires fewer specialists and enables new digital business models in cost-sensitive niche markets.

What technical innovations will we see in the mechanical engineering and automotive sectors with the help of language models?

Löser: Language models can support us throughout the value chain: From the idea and design of a machine, to the procurement of parts, to sales and interaction with customers. Let’s take the example of the car. By machine-reading relevant customer opinions, a system could tell us which features of a car are popular and which car a manufacturer should build. It could analyse supply chain problems in advance and support us in the procurement of parts. In modern cars, drivers can already be alerted to offers via screen and loudspeaker or process business communication and emails. During repairs, less knowledgeable employees could also ask a chatbot questions, which would provide them with specialised knowledge or even make diagnoses.

Thank you very much for the interview, Prof. Löser!

Prof. Alexander Löser heads the Data Science Research Centre of the BHT in Berlin. In the research group DATEXIS, he investigates Natural Language Processing (NLP), Machine Learning and also supports the LEAM initiative.