ChatGPT is an artificial intelligence system based on a language model and can generate text automatically. Professor Dr. Henning Wachsmuth tells us how it works and what ChatGPT cannot yet do but may be able to do in the future. The computational linguist leads two subprojects in TRR 318 and researches natural language processing at Leibniz University Hannover.
What is technically special about ChatGPT?
Henning Wachsmuth: ChatGPT uses a so-called language model that has learned from billions of texts, which words typically occur one after the other. The previous text serves as a context for the following text. When a human enters a new text, ChatGPT responds with the most likely following words according to its language model; it simply continues the text. What makes ChatGPT unique, apart from the size and quality of the data it has learned from, is that it has been specifically trained to generate typical dialogue responses using all its knowledge of the previous context.
How does ChatGPT respond to user queries?
HW: This is where the use of the previous context comes into play. ChatGPT considers not only the request but also everything that was said earlier in the dialogue – up to a certain maximum length. This allows it to make cross-references between different parts of the dialogue. At the same time, it uses all the information it has seen when it learned to generate text in order to provide the most appropriate response. Even as a researcher in natural language processing, it is still astonishing to me how impressively this works – even if not every generated text is correct.
Would it be possible to develop chatbots that ask users questions themselves and thus provide answers tailored to their level of knowledge?
HW: In fact, ChatGPT has been predominantly trained to answer instructional questions. In principle, it is also possible to let chatbots systematically ask questions, for example, to clarify ambiguities, and occasionally ChatGPT does so already. Chatbots can also pursue own goals if they are developed for predefined tasks. An example is an AI called Duplex, which Google unveiled in 2018. The AI can make appointments independently, for instance, with hairdressers, and seems to recognize when all required information has been communicated. However, pursuing goals becomes much more complicated if the tasks are not predefined: Then, the chatbot would have to establish own goals, which at some point would require its own awareness. Whether this will ever be possible with AI is debatable, although it may become increasingly difficult to tell the difference in the future.
ChatGPT is still in the early stages of development. What impact will chatbots like ChatGPT have on human-AI interaction?
HW: Basically, we can expect to communicate with AI in more and more situations in life to solve tasks. Sometimes this may simply be to find information or to write text more efficiently. At other times, we will communicate information to the AI in a step-by-step dialogue so that it can perform tasks for us, or vice versa, the AI supports us in decision making. So far, AI is still failing in some critical areas; for example, ChatGPT was not designed to be factually correct at its core. I expect that many of the problems will be solved gradually so that we can solve more tasks faster and better. But of course, there are risks when it is no longer possible to distinguish between what comes from a human and what comes from an AI. From a technical point of view, this can only be prevented to a certain extent. That’s why I think it’s essential that, in the future, all people are trained to deal with AI, for example, by integrating the topic into school lessons or as part of professional training.
Language Model
A language model is a representation of the probabilities of word sequences. These probabilities are derived from large collections of text; this can also be done for new word sequences by comparing them with similar ones. For a given sequence of words (e.g., a sentence), a language model can then be used to determine the most likely subsequent words and thus generate new texts (e.g., a following sentence). The sentence is used as a context to determine which following sentence fits best.
Further information:
- ChatGPT from a sociological perspective: "Technologies like ChatGPT have enormous potential to change the world" with Nils Klowait
