Analysis Filter

An analysis filter is a small, language-agnostic script used within the Mishegos differential-fuzzing framework to process cohorts of decoder results. Filters consume cohorts from standard input (stdin) and conditionally emit them to standard output (stdout), acting as composable building blocks in a pipeline of analysis logic.

Structure and Mechanics

Filters are intentionally small and focused, each performing a single transformation or test. They are typically written in scripting languages such as Ruby and process newline-delimited JSON records describing the candidate instruction and each decoder's output.

For example, the filter-ndecoded-different filter parses each incoming JSON line, extracts the ndecoded field from each decoder's output, and emits only those records where the decoders disagree on the number of bytes decoded:

require "json"

STDERR.puts "[+] pass: filter-ndecoded-different"

count = 0
STDIN.each_line do |line|
  result = JSON.parse line, symbolize_names: true
  outputs_ndecoded = result[:outputs].map { |o| o[:ndecoded] }

  if outputs_ndecoded.uniq.size > 1
    count += 1
    next
  end

  STDOUT.puts result.to_json
end

STDERR.puts "[+] pass: filter-ndecoded-different done: #{count} filtered"

Diagnostic messages are written to stderr to track which filter is running and how many records it filtered out.

Composition into Passes

Individual filters are collected into analysis passes, which are defined declaratively in ./src/analysis/passes.yml. The pass concept is modeled after LLVM passes. A pass can order its internal filters either:

• via a dependency graph declared between filters, or
• based on perceived performance requirements (for example, running the largest filters first to reduce the volume of data carried through the pipeline early).

An example pass named same-size-different-decodings composes the following filters in sequence:

• filter-any-failure
• filter-ndecoded-different
• filter-same-effects
• minimize-input
• normalize

Transforming Results and Cohorts

Beyond simple pass-through filtering, some filters modify individual results or entire cohorts to make downstream analysis easier:

• The minimize-input filter, an instance of input minimization, chops the instruction candidate down to the longest indicated ndecoded field, producing a smaller, more focused reproducer.
• The normalize filter removes extra whitespace from instruction candidates in preparation for additional analysis of individual disassemblies.

Detecting Discrepancy Classes

Filter pipelines are the primary mechanism by which Mishegos discovers discrepancy classes between competing decoders (such as libbfd, capstone, zydis, and xed). These classes include:

• Outright decoding failures, which can be exploited to prevent usage of a susceptible tool or platform entirely, or to smuggle malicious code past an analyst.
• Component discrepancies, in which decoders disagree on individual fields of a decoded instruction. These can mislead an automated analysis or human analyst into incorrectly interpreting program behavior, and severe enough discrepancies could mask the recovered control flow graph.

Execution and Visualization

Passes are run as a whole via the Mishegos analysis command-line interface. The analysis output can be visualized using mishmat, a lightweight HTML visualization tool, with an optional cap on the size of the HTML table (for example, mishmat -l 10000 < /tmp/mishegos.sd > /tmp/mishegos.sd.html). The resulting tables juxtapose the instruction candidate against the per-decoder outputs, annotating each cell with the bytes decoded versus the total assembled length (e.g., (8 / 29)) and marking outright failures with (bad).