Property-Based Testing

Overview

Property-based testing (PBT) generates random inputs and verifies that properties hold for all of them. Instead of testing specific examples, you test invariants.

When PBT beats example-based tests:

• Serialization pairs (encode/decode)
• Pure functions with clear contracts
• Validators and normalizers
• Data structure operations

Property Catalog

decode(encode(x)) == x

f(f(x)) == f(x)

f(a, b) == f(b, a)

f(f(a,b), c) == f(a, f(b,c))

f(x, identity) == x

f(g(x)) == x

new_impl(x) == reference(x)

is_sorted(sort(x))

Strength hierarchy (weakest to strongest):

No Exception -> Type Preservation -> Invariant -> Idempotence -> Roundtrip

Always aim for the strongest property that applies.

Pattern Detection

Use PBT when you see:

encode

decode

serialize

deserialize

toJSON

fromJSON

pack

unpack

normalize

sanitize

canonicalize

is_valid

validate

When NOT to Use

• Simple CRUD without transformation logic
• UI/presentation logic
• Integration tests requiring complex external setup
• Code with side effects that cannot be isolated
• Prototyping where requirements are fluid
• Tests where specific examples suffice and edge cases are understood

Library Quick Reference

from hypothesis import given, strategies as st

import fc from 'fast-check'

use proptest::prelude::*

import "pgregory.net/rapid"

@Property

import Test.QuickCheck

For library-specific syntax and patterns: Use

@ed3d-research-agents:internet-researcher

Input Strategy Best Practices

1. Constrain early: Build constraints INTO the strategy, not via

   assume()

   # GOOD
   st.integers(min_value=1, max_value=100)
   
   # BAD - high rejection rate
   st.integers().filter(lambda x: 1 <= x <= 100)
   

2. Size limits: Prevent slow tests

   st.lists(st.integers(), max_size=100)
   st.text(max_size=1000)
   

3. Realistic data: Match real-world constraints

   st.integers(min_value=0, max_value=150)  # Real ages, not arbitrary ints
   

4. Reuse strategies: Define once, use across tests

   valid_users = st.builds(User, ...)
   
   @given(valid_users)
   def test_one(user): ...
   
   @given(valid_users)
   def test_two(user): ...
   

Settings Guide

# Development (fast feedback)
@settings(max_examples=10)

# CI (thorough)
@settings(max_examples=200)

# Nightly/Release (exhaustive)
@settings(max_examples=1000, deadline=None)

Quality Checklist

Before committing PBT tests:

• Not tautological (assertion doesn't compare same expression)
• Strong assertion (not just "no crash")
• Not vacuous (inputs not over-filtered by

  assume()

  )
• Edge cases covered with explicit examples (

  @example

  )
• No reimplementation of function logic in assertion
• Strategy constraints are realistic
• Settings appropriate for context

Red Flags

• Tautological:

  assert sorted(xs) == sorted(xs)

  tests nothing
• Only "no crash": Always look for stronger properties
• Vacuous: Multiple

  assume()

  calls filter out most inputs
• Reimplementation:

  assert add(a, b) == a + b

  if that's how add is implemented
• Missing edge cases: No

  @example([])

  ,

  @example([1])

  decorators
• Overly constrained: Many

  assume()

  calls means redesign the strategy

Common Mistakes