Skipping falsifying examples in Hypothesis
<p><strong>The Story:</strong></p>

<p>I'm currently in the process of unit-testing a function using <a href="http://hypothesis.readthedocs.io/en/latest/" rel="nofollow"><code>hypothesis</code></a> and a <a href="http://stackoverflow.com/q/39561310/771848">custom generation strategy</a> trying to find a specific input to "break" my current solution. Here is how my test looks like:</p>

<pre><code>from solution import answer

# skipping mystrategy definition - not relevant

@given(mystrategy)
def test(l):
    assert answer(l) in {0, 1, 2}
</code></pre>

<p>Basically, I'm looking for possible inputs when <code>answer()</code> function does not return 0 or 1 or 2.</p>

<p>Here is how my <em>current workflow</em> looks like:</p>

<ul>
<li>run the test</li>
<li><p><code>hypothesis</code> finds an input that produces an <code>AssertionError</code>:</p>

<pre><code>$ pytest test.py 
=========================================== test session starts ============================================
...
------------------------------------------------ Hypothesis ------------------------------------------------
Falsifying example: test(l=[[0], [1]])
</code></pre></li>
<li><p>debug the function with this particular input trying to understand if this input/output is a legitimate one and the function worked correctly</p></li>
</ul>

<p><strong>The Question:</strong></p>

<p>How can I skip this falsifying generated example (<code>[[0], [1]]</code> in this case) and ask <code>hypothesis</code> to generate me a different one?</p>

<hr>

<p>The Question can also be interpreted: Can I ask <code>hypothesis</code> to not terminate if a falsifying example found and generate more falsifying examples instead?</p>

<p>There's at present no way to get Hypothesis to keep trying after it finds a failure (it might happen at some point, but it's not really clear what the right behaviour for this should be and it hasn't been a priority), but you can get it to ignore specific classes of failure using the <a href="https://hypothesis.readthedocs.io/en/latest/details.html#making-assumptions" rel="nofollow">assume</a> functionality.</p>

<p>e.g. you could skip this example with:</p>

<pre><code>@given(mystrategy)
def test(l):
    assume(l != [[0], [1]])
    assert answer(l) in {0, 1, 2}
</code></pre>

<p>Hypothesis will skip any examples where you call assume with a False argument and not count them towards your budget of examples it runs.</p>

<p>You'll probably find that this just results in trivial variations on the example, but you can pass more complex expressions to assume to ignore classes of examples.</p>

<p>What's your actual use case here? The normal intended usage pattern her would be to fix the bug that causes Hypothesis to fail and let it find new bugs that way. I know this isn't always practical, but I'm interested in why.</p>

python unit-testing testing property-based-testing python-hypothesis