Skip to content
STIMSMITH

Auto-regressive Language Model

Concept WIKI v1 · 6/14/2026

An auto-regressive language model is a neural language model that generates text by predicting each successive token conditioned only on the preceding tokens. Modern auto-regressive language models are typically built on stacked transformer decoder layers with multi-head self-attention and feed-forward networks, exemplified by the Generative Pre-trained Transformer (GPT) family, of which GPT-2 is a well-known instance.

Auto-regressive Language Model

Definition

An auto-regressive language model is a language model that generates a sequence by predicting each successive token conditioned only on the tokens that have come before it. In the formulation described in the GoldenFuzz paper, GPT-style auto-regressive language models "predict each successive token using only previous tokens, relying on stacked transformer decoder layers with multi-head self-attention and feed-forward networks." This unidirectional (left-to-right) generation is the defining property of the auto-regressive family of transformer language models [GoldenFuzz].

Architecture

Auto-regressive language models are most commonly implemented using transformer architectures. The transformer uses attention mechanisms to capture complex relationships across long text sequences. Within the auto-regressive family, the Generative Pre-trained Transformer (GPT) architecture is among the most influential; it is "built using a unidirectional approach" in which generation proceeds from left to right and each new token is conditioned only on the prior context [GoldenFuzz].

The core building blocks of GPT-style auto-regressive models are:

  • Stacked transformer decoder layers that process the input sequence in a left-to-right manner.
  • Multi-head self-attention sub-layers that allow each position to attend to all previous positions.
  • Feed-forward networks applied position-wise within each decoder block.

Relationship to Language Models in General

Language models are described as "one of the most advanced methods in Natural Language Processing (NLP)," supporting tasks such as translation, summarization, and conversational agents by predicting how words naturally follow one another. Auto-regressive language models are the dominant sub-family used for open-ended text generation within this broader class [GoldenFuzz].

Training and Fine-Tuning

Training a language model from scratch is "highly resource-intensive," so practitioners often start from pre-trained models and customize them with fine-tuning. Two commonly cited fine-tuning strategies are:

  • Reinforcement Learning from Human Feedback (RLHF), which aligns the model's outputs with human preferences or guidelines.
  • Direct Preference Optimization (DPO), which avoids the explicit reward-model step used in RLHF while still optimizing for preferred outputs.

Both RLHF and DPO are described as effective, low-cost learning strategies for adapting pre-trained language models to specific tasks [GoldenFuzz].

Applications

Auto-regressive language models have been applied to a wide range of tasks, including:

  • Code completion. ReACC proposes a retrieval-augmented code completion framework that "adopts a stage-wise training approach that combines a source code retriever and an auto-regressive language model for programming language," and evaluates the approach on Python and Java code completion, achieving state-of-the-art performance on the CodeXGLUE benchmark [ReACC].
  • Source code and natural language translation. PyMT5 is a multi-mode text-to-text transformer trained on a large parallel corpus of Python methods and docstrings; the authors report that "for docstring and method generation, PyMT5 outperforms similarly-sized auto-regressive language models (GPT2) which were English pre-trained or randomly initialized" [PyMT5].
  • Hardware test-case generation. GoldenFuzz employs an auto-regressive language model as a generator that "accurately produces assembly instructions by understanding inter-instruction semantics," integrating target feedback to enhance test-case generation coverage in hardware fuzzing [GoldenFuzz].

Representative Implementations

  • GPT-2 is a widely cited example of an auto-regressive language model. PyMT5 explicitly benchmarks against "similarly-sized auto-regressive language models (GPT2)" that were either English-pretrained or randomly initialized [PyMT5]. GPT-2 is therefore a concrete, documented implementation of the auto-regressive language modeling paradigm described above.

Citations

  1. Definition and architecture (unidirectional prediction, stacked transformer decoder layers, multi-head self-attention, feed-forward networks): GoldenFuzz paper, Section B "Language Model and Fine-Tuning."
  2. Training cost and fine-tuning strategies (RLHF, DPO): GoldenFuzz paper, Section B "Language Model and Fine-Tuning."
  3. Application to retrieval-augmented code completion (Python and Java, CodeXGLUE): ReACC (arXiv:2203.07722v1).
  4. Application to Python method/docstring translation and comparison with GPT-2: PyMT5 (arXiv:2010.03150v1).
  5. Application as an assembly-instruction generator for hardware fuzzing: GoldenFuzz paper.

LINKED ENTITIES

1 links
GPT-2 IMPLEMENTS

CITATIONS

5 sources
5 citations
[1] Auto-regressive (GPT-style) language models predict each successive token using only previous tokens, relying on stacked transformer decoder layers with multi-head self-attention and feed-forward networks, and are built using a unidirectional approach. GoldenFuzz: Generative Golden Reference Hardware Fuzzing
[2] Training a language model from scratch is highly resource-intensive; RLHF and DPO are common effective, low-cost fine-tuning strategies for adapting pre-trained language models to specific tasks. GoldenFuzz: Generative Golden Reference Hardware Fuzzing
[3] Language models are among the most advanced NLP methods, supporting tasks such as translation, summarization, and conversational agents by predicting how words naturally follow one another. GoldenFuzz: Generative Golden Reference Hardware Fuzzing
[4] An auto-regressive language model for programming language is combined with a source code retriever in a stage-wise training framework (ReACC), achieving state-of-the-art code completion performance on CodeXGLUE for Python and Java. ReACC: A Retrieval-Augmented Code Completion Framework
[5] GPT-2 is an example of an auto-regressive language model; PyMT5 reports outperforming similarly-sized auto-regressive language models (GPT2), whether English-pretrained or randomly initialized, on docstring and method generation tasks. PyMT5: multi-mode translation of natural language and Python code with transformers