Skip to content

Commit a94233d

Browse files
authored
Further expand differential fuzzing documentation (#11398)
* Further expand differential fuzzing documentation Split out the list of non-deterministic behaviors to its own separate section to both make it more easy to scan over and additionally have more fuzzing-related discussion such as generating programs that bake-in NaN canonicalization. I've also updated this to include WASIp1 APIs to discourage differential fuzzing of WASIp1 as the proposal is effectively not suitable for this. I've noted, though, that WASIp2-and-beyond should be suitable for differential fuzzing. * Be bold.
1 parent 69b4acb commit a94233d

File tree

1 file changed

+55
-5
lines changed

1 file changed

+55
-5
lines changed

docs/contributing-fuzzing.md

Lines changed: 55 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -54,11 +54,10 @@ The Wasmtime maintainers appreciate bug reports with the following:
5454
well as your rationale for *why* that behavior is expected. For example, just
5555
because another Wasm engine or an alternative Wasmtime execution strategy
5656
produces a different result from default Wasmtime, that is not necessarily a
57-
bug. Wasm has a variety of sources of
58-
[non-determinism](./examples-deterministic-wasm-execution.md) (e.g. `NaN` bit
59-
patterns, maximum call stack depth, and whether growing memory succeeds) and
60-
the two different results could both be valid and spec-conforming. Make sure
61-
to account for this in your rationale and analysis of the bug.
57+
bug. See the [documentation
58+
below](#divergent-webassembly-behavior-across-runtimes) for examples of known divergent
59+
behavior of one module in two runtimes. If applicable, make sure to account
60+
for this in your rationale and analysis of the bug.
6261

6362
* **Actual behavior:** A description of the actual, buggy behavior. This should
6463
include things various things like incorrect computation results, assertion
@@ -72,6 +71,57 @@ Including the above information is extra important for bugs discovered
7271
mechanically, whether by fuzzing or other means, since the associated test cases
7372
will often be pseudo-random or otherwise unintuitive to debug.
7473

74+
### Divergent WebAssembly behavior across runtimes
75+
76+
WebAssembly has a variety of sources of [non-determinism] which means that the
77+
exact same module is allowed to behave differently under the same inputs
78+
across multiple runtimes. These specifics don't often arise in "real world"
79+
modules but can quickly arise during fuzzing. Some example behaviors are:
80+
81+
* **NaN bit patterns** - floating-point operations which produce NaN as a result
82+
are allowed to produce any one of a set of patterns of NaN. This means that
83+
the exact bit-representation of the result of a floating-point operation may
84+
diverge across engines. When fuzzing you can update your source-generation to
85+
automatically canonicalize NaN values after all floating point operations.
86+
Wasmtime has built-in options to curb this [non-determinism] as well.
87+
88+
* **Relaxed SIMD** - the `relaxed-simd` proposal to WebAssembly explicitly has
89+
multiple allowed results for instructions given particular inputs. These
90+
instructions are inherently non-deterministic across implementations. When
91+
fuzzing you can avoid these instructions entirely, canonicalize the results,
92+
or use Wasmtime's built-in options to curb the [non-determinism].
93+
94+
* **Call stack exhaustion** - the WebAssembly specification requires that all
95+
function calls consume a nonzero-amount of some resource which can eventually
96+
be exhausted. This means that infinite recursion is not allowed in any
97+
WebAssembly engine. Bounded, but very large, recursion is allowed in
98+
WebAssembly but is not guaranteed to work across WebAssembly engines. One
99+
engine may have different stack settings than another engine and/or runtime
100+
parameters may tune how much stack space is taken (e.g. optimizations on/off).
101+
If one engine stack overflows and another doesn't then that's not necessarily
102+
a bug in either engine. Short of banning recursion there's no known great way
103+
to handle this apart from throwing out fuzz test cases that stack overflow.
104+
105+
* **Memory exhaustion** - the `memory.grow` and `table.grow` instructions in
106+
WebAssembly are not always guaranteed to either fail or succeed. This means
107+
that growth may succeed in one engine but fail in another depending on various
108+
settings. To handle this in fuzzing it's recommended to generate memories with
109+
a maximum size and ensure that each engine being fuzzed can grow memory all
110+
the way to the maximum size.
111+
112+
* **WASIp1 API behavior** - the initial specification of WASI, WASIp1 or
113+
`wasi_snapshot_preview1`, effectively is not suitable for differential fuzzing
114+
across engines. The APIs are not thoroughly specified enough nor is there a
115+
rigorous enough test suite to codify what exactly should happen in all
116+
situations on all platforms. This means that exactly what kind of error arises
117+
or various other edge cases may behave differently across engines. The lack of
118+
specificity of WASIp1 means that there is no great oracle as to whether an
119+
engine is right or wrong. Development of WASIp1 has ceased and the Component
120+
Model is being worked on instead (e.g. WASIp2 and beyond) which is more
121+
suitable for differential fuzzing.
122+
123+
[non-determinism]: ./examples-deterministic-wasm-execution.md
124+
75125
### Do Not Report the Same Bug Multiple Times
76126

77127
Fuzzers will often trigger the same bug multiple times in multiple different

0 commit comments

Comments
 (0)