
Large language models are neural network methods that learn to generate and predict human-like text from vast datasets.Modern large language models are neural networks with a 'transformer' architecture, featuring anywhere from hundreds of millions to hundreds of billions of parameters. Each parameter is a number that influences how the signal from one word is passed to another. The model doesn't store words but converts them into vectors of numbers—mathematical points in a high-dimensional space of meanings. Pre-training is carried out on an autocomplete task: the model fills in missing words or predicts the next word in a sequence. Afterwards, it can be fine-tuned on specialized examples to hold a dialogue, solve problems, or write code.
The history began with statistical n-gram models (1950s), then neural network language models (Bengio et al., 2000s). The turning point was the paper Attention Is All You Need (2017), which proposed the transformer architecture. In 2018, OpenAI released GPT-1 with 117 million parameters, and in 2020, GPT-3 with 175 billion parameters demonstrated that a model without explicit training could translate, program, and reason simply by receiving instructions in natural language. Since then, the race has focused on both scaling up models and improving efficiency (Mixture-of-Experts, Chinchilla scaling laws).
The model receives a context window of n tokens as input. Each token is mapped to an embedding and encoded by self-attention layers, creating contextualized representations. At the output, a linear classifier predicts the probability distribution of the next token. Training minimizes the cross-entropy between the prediction and the true token. During generation, text is produced by sequential sampling of tokens (usually with top-p or top-k nucleus sampling). Key techniques: pre-training on unlabeled text, then instruction-based or RLHF fine-tuning to align with usefulness and safety. The attention mechanism computes similarity weights between queries and keys for each pair of tokens, allowing it to focus on long-range dependencies.
Such models write news, compose poems, generate programming code, answer customer questions in chats, and help doctors draft discharge summaries from medical histories.
GPT-3 can write an essay indistinguishable from a human-written one, as judged by a committee, in 2 seconds—the time it takes a person just to open a text editor.















