Replaying failed tests

This page discusses how Hypothesis saves and replays failing tests, and how to tell Hypothesis to try a specific test input every time.

Hypothesis saves failing examples

When a test fails, Hypothesis saves the failure to a database in the local .hypothesis directory. It will then retry this failure any time the test gets run in the future. For instance, the following will take a few seconds to fail from random generation:

import time

from hypothesis import strategies as st

@given(st.integers())
def f(n):
    assert n < 50
    time.sleep(0.1)

f()

As it fails, Hypothesis saves the failing example to the local database. As a result, the next time you run this same code, f will fail instantly, because Hypothesis retries the failing example first.

Disabling the local database

You can disable the local database with @settings:

import time

from hypothesis import settings, strategies as st

@given(st.integers())
@settings(database=None)
def f(n):
    assert n < 50
    time.sleep(0.1)

f()

How to always try a specific input

If you want Hypothesis to always run a specific input, you can use the @example decorator. @example adds an explicit input which Hypothesis will run every time, in addition to the randomly generated examples. You can think of explicit examples as combining unit-testing with property-based testing.

For instance, suppose we are testing something using integers, but want to make sure we try a few large prime numbers on every test run. We can add these primes with an explicit @example:

# mersenne primes
@example(2**17 - 1)
@example(2**19 - 1)
@given(st.integers())
def test_something_with_integers(n):
    pass

test_something_with_integers()

Hypothesis runs all explicit examples first, in the Phase.explicit phase, before generating additional random examples in the Phase.generate phase.

Explicit examples do not shrink

Unlike examples generated by Hypothesis, examples provided using @example do not shrink. We can see this by adding a failing assertion:

@example(2**17 - 1)
@given(st.integers())
def test_something_with_integers(n):
    assert n < 100

Hypothesis will print Falsifying explicit example: test_something_with_integers(n=131071), instead of shrinking to n=100.

Prefer @example over the database for correctness

While the database is useful for local development, we don’t recommend relying on it for the correctness of your tests. If a test fails with a particularly interesting input, and you want to make sure your test works for this input in the future, we recommend explicitly adding it to your test with @example.

The reason is that entries in the database may be invalidated at any time by Hypothesis, because the internal format is not necessarily stable across versions. Additionally, the database is generally not checked into version control, so failures are not shared between developers, unlike with an explicit @example.

Note

If you do want to share the database between developers, using a database like RedisExampleDatabase might be helpful.

Replaying examples from CI logs

If your test fails in a CI job, Hypothesis will automatically print an @reproduce_failure decorator containing an opaque blob, which you can add to your local test to reproduce the failure:

You can reproduce this example by temporarily adding @reproduce_failure('6.130.5', b'AELnLA==') as a decorator on your test case
Falsifying example: test(
    i=-6356,
)