Skip to content
STIMSMITH

Automatic Workload Generation

Concept WIKI v1 · 6/13/2026

Automatic Workload Generation is a methodology for systematically synthesizing representative training applications that cover a broad range of program behavior state-space, primarily used to train machine learning models for hardware performance and power prediction. By exposing low-level program characteristics as user-controllable knobs, it overcomes the limitations of standard benchmark suites, achieving over 11x higher state-space coverage than suites such as SPEC CPU2006, MiBench, MediaBench, and TPC-H, and improving the accuracy of ML-based prediction systems by 2.5x to 3.6x.

Automatic Workload Generation

Overview

Automatic Workload Generation is a methodology and framework for systematically generating synthetic training applications whose low-level program characteristics can be precisely controlled. It is designed to produce training sets that cover a wide region of the program behavior state-space, enabling the construction of more accurate and generalizable machine learning (ML) models for tasks such as performance and power prediction. The approach was introduced to overcome the limitations of relying on hand-picked standard benchmark suites, which were shown to leave large portions of the application state-space uncovered and to require tedious manual effort to extend.

Motivation

ML-based performance and power prediction systems typically follow a supervised learning paradigm consisting of a training phase and a prediction phase. During training, programs are executed to capture performance features and reference metrics (e.g., IPC, power), and a predictive model is constructed that associates application characteristics with those reference metrics.

The accuracy and usefulness of such models depend heavily on the training set:

  • A well-formed training set improves accuracy and applicability to a wide range of test applications.
  • An ill-formed training set can cause overfitting and lead to inaccuracies during prediction.

Standard suites such as SPEC CPU2006, MiBench, MediaBench, and TPC-H often exhibit clustered and incomplete coverage of the state-space, and hand-tuned micro-benchmarks — while useful for stressing specific properties — are tedious to create and yield too few instances for effective statistical training.

Core Methodology

Automatic Workload Generation is built upon a set of core low-level program metrics, each corresponding to a low-level program feature and exposed as a user-controllable programmable knob. The user fixes the values of some metrics to target a specific program behavior; for any remaining metrics, the framework systematically randomizes values within reasonable bounds to achieve well-rounded state-space coverage around the target behavior.

The combination of chosen metric values acts as a Workload-Specific Profile, which is fed into a code generator algorithm. The code generator produces a suite of synthetic programs whose behavior matches the profile.

Categories of Core Metrics

The profile is organized into three metric categories:

Instruction-level Characteristics

  1. Instruction mix — 5 categories
  2. Instruction count — 1
  3. Instruction cache miss rate (ICMR) — 1
  4. Instruction-level parallelism (ILP) — 32 bins

Control-flow Characteristics 5. Average basic block size — 1 6. Branch transition rate (BTR) — 1 7. Branch misprediction rate — 1 8. Data footprint — 1

Memory-access Characteristics 9. Regular/irregular behavior — 1 10. Spatial locality stride bins — 32 bins 11. Temporal locality bins — 8 bins 12. L1/L2 Data cache miss rates — 2

Example Metric Mechanics

  • Instruction mix (IMIX): Randomized within bounded ranges; the resulting IMIX fractions are used to generate the suite of training applications.
  • Instruction count (IC): Controls the static instruction footprint. IC can be supplied directly or estimated automatically from a target ICMR. An initial estimate assumes a default instruction cache (32 KB, 64-byte blocks, 2-way set-associativity) and is then further tuned to achieve the target ICMR.
  • Instruction-level parallelism (ILP): Tight producer-consumer chains limit ILP, making this an important determinant of performance.

State-Space Coverage Metric: SpreadRatio

To compare state-space coverage across different training sets, Automatic Workload Generation introduces a metric called SpreadRatio. SpreadRatio is based on the area of the convex hull envelope surrounding the program points in the metric space.

Demonstrated Benefits

Empirical evaluation using this methodology demonstrated:

  • >11× higher state-space coverage compared to popular standard benchmark suites such as SPEC CPU2006, MiBench, MediaBench, and TPC-H.
  • >2.5× improvement in the predictability/accuracy of an ML-based power prediction system.
  • >3.6× improvement in the predictability/accuracy of an ML-based performance prediction system.

Relation to Predictive Benchmarking

Automatic Workload Generation is a foundational component of Predictive Benchmarking workflows, where systematically generated, behaviorally diverse training programs are used to build accurate predictive models of system-level metrics such as performance and power.

Implementation

The methodology is implemented in Genesys, an automatic workload generation framework that operationalizes the metric-driven code generation pipeline described above.

LINKED ENTITIES

2 links
Genesys IMPLEMENTS
Predictive Benchmarking PART_OF

CITATIONS

10 sources
10 citations
[1] Automatic Workload Generation is a workload generation framework that enables systematic generation of representative training set applications, providing a wider coverage of program behavior state-space for effectively training machine learning models. Genesys: Automatically Generating Representative [Workloads] (SAMOS 2016)
[2] The methodology allows control of a set of key workload-specific characteristics using easy-to-use, programmable knobs, providing the ability to generate applications targeting specific program properties. Genesys: Automatically Generating Representative [Workloads] (SAMOS 2016)
[3] SpreadRatio is a novel metric defined to compare state-space coverage, based on the area of the convex hull envelope surrounding the program points. Genesys: Automatically Generating Representative [Workloads] (SAMOS 2016)
[4] Automatically generated training sets achieve over 11 times higher state-space coverage than popular standard benchmarks such as SPEC CPU2006, MiBench, MediaBench, and TPC-H. Genesys: Automatically Generating Representative [Workloads] (SAMOS 2016)
[5] Modeling using targeted synthetic training sets improves the predictability/accuracy of two machine-learning-based power and performance prediction systems by over 2.5x and 3.6x respectively. Genesys: Automatically Generating Representative [Workloads] (SAMOS 2016)
[6] Genesys is built upon a set of core workload-specific metrics (instruction-level, control-flow, and memory-access) that can be controlled systematically; the user fixes some values and the framework randomizes the rest within reasonable bounds to achieve well-rounded state-space coverage. Genesys: Automatically Generating Representative [Workloads] (SAMOS 2016)
[7] Core metrics include: Instruction mix (5 categories), Instruction count, Instruction cache miss rate (ICMR), Instruction-level parallelism (ILP, 32 bins), Average basic block size, Branch transition rate (BTR), Branch misprediction rate, Data footprint, Regular/irregular behavior, Spatial locality stride bins (32 bins), Temporal locality bins (8 bins), L1/L2 Data cache miss rates. Genesys: Automatically Generating Representative [Workloads] (SAMOS 2016)
[8] The user provides target metric values that form a Workload-Specific Profile, which is fed into a code generator that produces a suite of synthetic programs (Syn Prog 1, Syn Prog 2, ...). Genesys: Automatically Generating Representative [Workloads] (SAMOS 2016)
[9] ML-based modeling relies heavily on training sets; an ill-formed training set can affect the statistical model (e.g., cause overfitting), while a well-formed training set improves accuracy and applicability. Genesys: Automatically Generating Representative [Workloads] (SAMOS 2016)
[10] Hand-tuned micro-benchmarks are tedious to create, time-consuming, and produce too few program instances for statistical training; standard benchmark suites such as SPEC CPU2006 have been shown to exhibit similar low-level behavior across application domains, leaving much of the state-space uncovered. Genesys: Automatically Generating Representative [Workloads] (SAMOS 2016)