Auto-regressive Language Model

Concept

An auto-regressive language model is a neural language model that generates text by predicting each successive token conditioned only on the preceding tokens. Modern auto-regressive language models are typically built on stacked transformer decoder layers with multi-head self-attention and feed-forward networks, exemplified by the Generative Pre-trained Transformer (GPT) family, of which GPT-2 is a well-known instance.

First seen 6/14/2026

Last seen 6/14/2026

Evidence 2 chunks

Wiki v1

WIKI

Auto-regressive Language Model

Definition

An auto-regressive language model is a language model that generates a sequence by predicting each successive token conditioned only on the tokens that have come before it. In the formulation described in the GoldenFuzz paper, GPT-style auto-regressive language models "predict each successive token using only previous tokens, relying on stacked transformer decoder layers with multi-head self-attention and feed-forward networks." This unidirectional (left-to-right) generation is the defining property of the auto-regressive family of transformer language models [GoldenFuzz].

READ FULL ARTICLE →

NEIGHBORHOOD

No graph connections found for this entity yet. It may appear in future ingestion runs.

explore full graph →

RELATIONSHIPS

1 connections

GPT-2 ← implements 90% 1e

GoldenFuzz's language model uses an auto-regressive objective.

LINKED ENTITIES

1 links

GPT-2 IMPLEMENTS Extracted graph relationship

CITATIONS

5 sources

5 citations — click to expand

[1] Auto-regressive (GPT-style) language models predict each successive token using only previous tokens, relying on stacked transformer decoder layers with multi-head self-attention and feed-forward networks, and are built using a unidirectional approach. GoldenFuzz: Generative Golden Reference Hardware Fuzzing

[2] Training a language model from scratch is highly resource-intensive; RLHF and DPO are common effective, low-cost fine-tuning strategies for adapting pre-trained language models to specific tasks. GoldenFuzz: Generative Golden Reference Hardware Fuzzing

[3] Language models are among the most advanced NLP methods, supporting tasks such as translation, summarization, and conversational agents by predicting how words naturally follow one another. GoldenFuzz: Generative Golden Reference Hardware Fuzzing

[4] An auto-regressive language model for programming language is combined with a source code retriever in a stage-wise training framework (ReACC), achieving state-of-the-art code completion performance on CodeXGLUE for Python and Java. ReACC: A Retrieval-Augmented Code Completion Framework

[5] GPT-2 is an example of an auto-regressive language model; PyMT5 reports outperforming similarly-sized auto-regressive language models (GPT2), whether English-pretrained or randomly initialized, on docstring and method generation tasks. PyMT5: multi-mode translation of natural language and Python code with transformers