DiscoverScroll article

A large language model (LLM) is an AI model (typically a neural network) trained on a vast amount of text for natural language processing tasks, especially language generation.

Generative pre-trained transformers (GPTs) are a type of LLM that is pre-trained to predict the next word. Benchmark evaluations for LLMs attempt to measure model reasoning, factual accuracy, alignment, and safety.

Before the emergence of transformer-based models in 2017, some language models were considered large relative to the computational and data constraints of their time. In the early 1990s, IBM's statistical models pioneered word alignment techniques for machine translation, laying the groundwork for corpus-based language modeling. During the 2000s, with the rise of widespread internet access, researchers began compiling massive text datasets from the web ("web as corpus") to train statistical language models. Moving beyond n-gram models, researchers started in 2000 to use neural networks as language models. Following the breakthrough of deep neural networks in image classification around 2012, similar architectures were adapted for language tasks. GPT-3 in 2020 went a step further and as of 2025 is available only via API with no offering of downloading the model to execute locally. In 2024, OpenAI released the reasoning model OpenAI o1, which generates long chains of thought before returning a final answer.

An LLM is a type of foundation model (large X model) trained on language.

The tendency towards larger models is visible in the list of large language models. The qualifier "large" in "large language model" is inherently vague, as there is no definitive threshold for the number of parameters required to qualify as "large".

This type of model became commercially available in 2022 with ChatGPT, a sibling model of InstructGPT fine-tuned to accept and produce dialog-formatted text based on GPT-3.5.

For example, Google PaLM model was fine-tuned into a multimodal model and applied to robotic control. As an example, the Flamingo model uses cross-attention layers to inject visual information into its pre-trained language model.

is the number of parameters in the model.

The matter of LLM's exhibiting intelligence or understanding has two main aspects—the first is how to model thought and language in a computer system, and the second is how to enable the computer system to generate human-like language. American linguist George Lakoff presented neural theory of language (NTL) as a computational basis for using language as a model of learning tasks and understanding. The NTL model outlines how specific neural structures of the human brain shape the nature of thought and language and in turn what are the computational properties of such neural systems that can be applied to model thought and language in a computer system. In his 2014 book titled The Language Myth: Why Language Is Not An Instinct, British cognitive linguist and digital communication technologist Vyvyan Evans mapped out the role of probabilistic context-free grammar (PCFG) in enabling NLP to model cognitive patterns and generate human-like language.

The canonical measure of the performance of any language model is its perplexity on a given text corpus. Perplexity measures how well a model predicts the contents of a dataset; the higher the likelihood the model assigns to the dataset, the lower the perplexity.

In 2023, Nature Biomedical Engineering wrote that "it is no longer possible to accurately distinguish" human-written text from text created by large language models, and that "It is all but certain that general-purpose large language models will rapidly proliferate...

Language model benchmark – Standardized AI performance test Small language model – Type of artificial intelligence model SGLang – Open-source framework for large language model inference and multimodal models vLLM – Open-source software for large language model inference Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, 3rd Edition draft, 2023.

Quick Facts

• An LLM is a type of foundation model (large X model) trained on language.
• The qualifier "large" in "large language model" is inherently vague, as there is no definitive threshold for the number of parameters required to qualify as "large".
• In 2023, Nature Biomedical Engineering wrote that "it is no longer possible to accurately distinguish" human-written text from text created by large language models, and that "It is all but certain that general-purpose large language models will rapidly proliferate...
• Before the emergence of transformer-based models in 2017, some language models were considered large relative to the computational and data constraints of their time.
• Small language model – Type of artificial intelligence model

Source material: Wikipedia - "Large language model". Adapted and summarized for DiscoverScroll. Original contributors are credited through the linked Wikipedia article. Read original on Wikipedia. CC BY-SA 4.0. Changes were made from the original.