Fuzz
This is a library of fuzzers you can use to supply values to your fuzz tests. You can typically pick out which ones you need according to their types.
A Fuzzer a
knows how to create values of type a
. It can create them randomly,
so that your test's expectations are run against many values. Fuzzers will often
generate edge cases likely to find bugs. If the fuzzer can make your test fail,
the test runner also knows how to "simplify" that failing input into more minimal
examples, some of which might also cause the tests to fail. In this way, fuzzers
can usually find the simplest input that reproduces a bug.
Fuzzers
The representation of fuzzers is opaque. Conceptually, a Fuzzer a
consists
of a way to randomly generate values of type a
in a way allowing the test runner
to simplify those values.
Generate a few example values from the fuzzer.
Useful in REPL:
> import Fuzz
> Fuzz.examples 20 (Fuzz.intRange 20 50)
[42,45,32,26,33,29,41,45,23,45,34,23,22,42,29,27,41,43,30,50]
: Array Int
Uses the first argument as the seed as well as the count of examples to generate.
Will return an empty array in case of rejection.
Show examples of values satisfying given classification predicates (see
also Test.reportDistribution
and
Test.expectDistribution
).
Generates a given number of values and classifies them based on the predicates.
Uses the first argument as the seed as well as the count of examples to generate.
This function will always return all the given "base" labels, even if no examples of them could be found:
Fuzz.labelExamples 100
[ ( "Lower boundary (1)", \n -> n == 1 )
, ( "Upper boundary (20)", \n -> n == 20 )
, ( "In the middle (2..19)", \n -> n > 1 && n < 20 )
, ( "Outside boundaries??", \n -> n < 1 || n > 20 )
]
(Fuzz.intRange 1 20)
-->
[ ( [ "Lower boundary (1)" ], Just 1 )
, ( [ "Upper boundary (20)" ], Just 20 )
, ( [ "In the middle (2..19)" ], Just 5 )
, ( [ "Outside boundaries??" ], Nothing )
]
In case of predicate overlap (eg. something is both green and big) this function will also return all the found combinations:
Fuzz.labelExamples 100
[ ( "fizz", \n -> (n |> Math.modBy 3) == 0 )
, ( "buzz", \n -> (n |> Math.modBy 5) == 0 )
]
(Fuzz.intRange 1 20)
-->
[ ( [ "fizz" ], Just 3 )
, ( [ "buzz" ], Just 10 )
, ( [ "fizz, buzz" ], Just 15 )
]
Number fuzzers
A fuzzer for int values. It will never produce NaN
, Infinity
, or
-Infinity
.
This fuzzer will generate values in the range Random.minInt .. Random.maxInt
.
- Simplifies towards 0
- Prefers positive values over negative ones
- Prefers smaller values over larger ones
A fuzzer for int values between a given minimum and maximum value, inclusive. Shrunk values will also be within the range.
Draw an integer between 0 and n inclusive.
Will simplify towards 0, but draws uniformly over the whole range.
Max supported value is 2^32 - 1.
A fuzzer that will generate values in range n..2^32-1.
A fuzzer that will generate values in range -(2^32-1)..n.
A fuzzer for float values.
Will prefer integer values, nice fractions and positive numbers over the rest.
Will occasionally try infinities and NaN. If you don't want to generate these,
use Fuzz.niceFloat
.
A fuzzer for float values.
Will prefer integer values, nice fractions and positive numbers over the rest.
Will never try infinities or NaN.
A fuzzer for percentage values. Generates random floats between 0.0
inclusive and 1.0
exclusive, in an uniform fashion.
Will occasionally try the boundaries.
Doesn't shrink to nice values like Fuzz.float
does; shrinks towards
zero.
A fuzzer for float values within between a given minimum and maximum (inclusive).
Shrunken values will also be within the range.
Fuzzer generating floats in range n..Infinity
.
The positive part of the range will shrink nicely, the negative part will shrink uniformly.
The fuzzer will occasionally try the minimum, 0 (if in range) and Infinity.
Fuzzer generating floats in range -Infinity..n
.
The negative part of the range will shrink nicely, the positive part will shrink uniformly.
The fuzzer will occasionally try the maximum, 0 (if in range) and -Infinity.
String-related fuzzers
A fuzzer for arbitrary Unicode char values.
Avoids surrogate pairs or their components (0xD800..0xDFFF
).
Will prefer ASCII characters, whitespace, and some examples known to cause trouble, like combining diacritics marks and emojis.
A fuzzer for simple ASCII char values (range 32..126). Skips control characters and the extended character set.
For more serious char fuzzing look at Fuzz.char
which generates the
whole Unicode range.
Generates random unicode strings of up to 10 characters.
Generates random unicode strings of a given length.
Note that some unicode characters have String.length
of 2. This fuzzer will
make sure the String.length
of the returned string is equal to the wanted
length, even if it will mean there are less characters. If you instead want it
to give N characters even if their String.length
will be above N, you can use
Fuzz.arrayOfLength n Fuzz.char
|> Fuzz.map String.fromArray
Generates random unicode strings of length between the given limits.
Note that some unicode characters have String.length
of 2. This fuzzer will
make sure the String.length
of the returned string is equal to the wanted
length, even if it will mean there are less characters. If you instead want it
to give between MIN and MAX characters even if their String.length
will be
above MAX, you can use
Fuzz.arrayOfLengthBetween min max Fuzz.char
|> Fuzz.map String.fromArray
Generates random ASCII strings of up to 10 characters.
Generates random ASCII strings of a given length.
Generates random ASCII strings of length between the given limits.
Collection fuzzers
Create a fuzzer of pairs from two fuzzers.
Create a fuzzer of triples from three fuzzers.
Given a fuzzer of a type, create a fuzzer of a array of that type. Generates random arrays of varying length, up to 32 elements.
Given a fuzzer of a type, create a fuzzer of a array of that type. Generates random arrays of exactly the specified length.
Given a fuzzer of a type, create a fuzzer of a array of that type. Generates random arrays of length between the two given integers.
A fuzzer that shuffles the given array.
Given a fuzzer of a type, create a fuzzer of a maybe for that type.
Given fuzzers for an error type and a success type, create a fuzzer for a result.
Other fuzzers
A fuzzer for boolean values. It's useful when building up fuzzers of complex types that contain a boolean somewhere.
We recommend against writing tests fuzzing over booleans. Write a unit test for the true and false cases explicitly.
Simplifies in order False < True
.
A fuzzer for the unit value. Unit is a type with only one value, commonly used as a placeholder.
A fuzzer for order values.
Simplifies in order LT < EQ < GT
.
A fuzzer for boolean values, generating True with the given probability (0.0 = always False, 1.0 = always True).
Probabilities outside the 0..1
range will be clamped to 0..1
.
Simplifies towards False (if not prevented to do that by using probability >= 1).
Choosing fuzzers
Choose one of the given fuzzers at random. Each fuzzer has an equal chance
of being chosen; to customize the probabilities, use Fuzz.frequency
.
This fuzzer will simplify towards the fuzzers earlier in the array (each of which will also apply its own way to simplify the values).
Fuzz.oneOf
[ Fuzz.intRange 0 3
, Fuzz.intRange 7 9
]
Choose one of the given values at random. Each value has an equal chance
of being chosen; to customize the probabilities, use
Fuzz.frequencyValues
.
This fuzzer will simplify towards the values earlier in the array.
Fuzz.oneOfValues
[ 999
, -42
]
Create a new Fuzzer
by providing a array of probabilistic weights to use
with other fuzzers.
For example, to create a Fuzzer
that has a 1/4 chance of generating an int
between -1 and -100, and a 3/4 chance of generating one between 1 and 100,
you could do this:
Fuzz.frequency
[ ( 1, Fuzz.intRange -100 -1 )
, ( 3, Fuzz.intRange 1 100 )
]
This fuzzer will simplify towards the fuzzers earlier in the array (each of which will also apply its own way to simplify the values).
There are a few circumstances in which this function will return an invalid fuzzer, which causes it to fail any test that uses it:
- If you provide an empty array of frequencies
- If any of the weights are less than 0
- If the weights sum to 0
Be careful recursively using this fuzzer in its arguments. Often using
Fuzz.map
is a better way to do what you want. If you are fuzzing a
tree-like data structure, you should include a depth limit so to avoid infinite
recursion, like so:
type Tree
= Leaf
| Branch Tree Tree
tree : Int -> Fuzzer Tree
tree i =
if i <= 0 then
Fuzz.constant Leaf
else
Fuzz.frequency
[ { weight = 1, fuzzer = Fuzz.constant Leaf }
, { weight = 2, fuzzer = Fuzz.map2 Branch (tree (i - 1)) (tree (i - 1)) }
]
Create a Fuzzer
by providing a array of probabilistic weights to use with
values.
For example, to create a Fuzzer
that has a 1/4 chance of generating a string
"foo", and a 3/4 chance of generating a string "bar", you could do this:
Fuzz.frequencyValues
[ { weight = 1, value = "foo" }
, { weight = 3, value = "bar" }
]
This fuzzer will simplify towards the values earlier in the array.
There are a few circumstances in which this function will return an invalid fuzzer, which causes it to fail any test that uses it:
- If you provide an empty array of frequencies
- If any of the weights are less than 0
- If the weights sum to 0
Working with Fuzzers
Create a fuzzer that only and always returns the value provided, and performs no simplifying. This is hardly random, and so this function is best used as a helper when creating more complicated fuzzers.
A fuzzer that is invalid for the provided reason. Any fuzzers built with it are also invalid. Any tests using an invalid fuzzer fail.
A fuzzer that only lets through values satisfying the given predicate function.
Warning: By using Fuzz.filter
you can get exceptionally unlucky and get 15
rejections in a row, in which case the test will fluke out and fail!
It's always preferable to get to your wanted values using Fuzz.map
,
as you don't run the risk of rejecting too may values and slowing down your
tests, for example using Fuzz.intRange 0 5 |> Fuzz.map (\x -> x * 2)
instead
of Fuzz.intRange 0 9 |> Fuzz.filter (\x -> Math.modBy 2 x == 0)
.
If you want to generate indefinitely until you find a satisfactory value (with a risk of infinite loop depending on the predicate), you can use this pattern:
goodItemFuzzer =
itemFuzzer
|> Fuzz.andThen
(\item ->
if isGood item then
Fuzz.constant item
else
goodItemFuzzer
)
Map a function over a fuzzer.
Map over two fuzzers.
Map over three fuzzers.
Map over four fuzzers.
Map over five fuzzers.
Map over six fuzzers.
Map over seven fuzzers.
Map over eight fuzzers.
Map over many fuzzers. This can act as mapN
for N > 8
.
The argument order is meant to accommodate chaining:
Fuzz.constant fn
|> Fuzz.andMap fuzzerA
|> Fuzz.andMap fuzzerB
|> Fuzz.andMap fuzzerC
Use a generated value to decide what fuzzer to use next.
For example, let's say you want to generate a array of given length.
One (not ideal) possible way to do that is first choosing how many elements
will there be (generating a number), andThen
generating a array with that many
items:
Fuzz.intRange 1 10
|> Fuzz.andThen
(\length ->
let
go : Int -> Array a -> Fuzzer (Array a)
go left acc =
if left <= 0 then
Fuzz.constant (Array.reverse acc)
else
itemFuzzer
|> Fuzz.andThen (\item -> go (length - 1) (item :: acc))
in
go length []
)
This will work! Different fuzzers will have different PRNG usage patterns
though and will shrink with varying success. The currently best known way to
fuzz a array of items is based on a "flip a coin, andThen
generate a value and
repeat or end" approach, and is implemented in the Fuzz.array
helpers
in this module. Use them instead of rolling your own array generator!
Think of andThen
as a generalization of Fuzz.map
. Inside
Fuzz.map
you don't have the option to fuzz another value based on
what you already have; inside andThen
you do.
A fuzzer that delays its execution. Handy for recursive types and preventing infinite recursion.
Executes every fuzzer in the array and collects their values into the returned array.
Rejections (eg. from Fuzz.filter
or Fuzz.invalid
)
bubble up instead of being discarded.
Runs the Fuzzer-returning function on every item in the array, executes them= and collects their values into the returned array.
Rejections (eg. from Fuzz.filter
or Fuzz.invalid
)
bubble up instead of being discarded.
Misc helpers
(Avoid this function if you can! It is only provided as an escape hatch.)
Convert a Random.Generator into a Fuzzer.
Works internally by generating a random seed and running Random.step
.
Note this will not shrink well (in fact it will shrink randomly, to smaller seeds), as Generators are black boxes from the perspective of Fuzzers. If you want meaningful shrinking, define fuzzers using the other functions in this module!