Deterministic Simulation Testing (Rust)

When to Use This Skill

Read this when testing distributed systems that need:

• Reproducible failure scenarios
• Network partition testing
• Consensus protocol verification
• Fast simulation of time-dependent behavior
• Reproducing intermittent bugs

Key insight: Control all non-determinism (time, randomness, network, I/O) with a seed = identical behavior every run.

---

The Four Pillars of Determinism

Pillar	Problem	Solution
Execution	Thread scheduling is non-deterministic	Single-threaded async executor
Entropy	RNGs use system entropy	Seeded PRNG throughout
Time	Real clocks advance unpredictably	Simulated clock, manual advancement
I/O	Network has variable latency/loss	In-memory simulated network

---

Key Libraries

Library	Purpose	Use When
turmoil	Network simulation for Tokio	Standard TCP/UDP patterns
madsim	Full libc interception	Need to catch all entropy/time sources
proptest	Property-based input generation	Generating test scenarios

---

Quick Start

1. Add Dependencies

[features]
default = []
simulation = ["turmoil"]

[dependencies]
tokio = { version = "1", features = ["full"] }
rand = "0.8"

[dev-dependencies]
turmoil = "0.6"

2. Basic Turmoil Test

use turmoil::Builder;
use std::time::Duration;

#[test]
fn test_echo_server() {
    let mut sim = Builder::new()
        .simulation_duration(Duration::from_secs(60))
        .build();

    // Server
    sim.host("server", || async {
        let listener = turmoil::net::TcpListener::bind("0.0.0.0:8080").await?;
        let (mut socket, _) = listener.accept().await?;

        let mut buf = [0u8; 1024];
        loop {
            let n = socket.read(&mut buf).await?;
            if n == 0 { break; }
            socket.write_all(&buf[..n]).await?;
        }
        Ok(())
    });

    // Client
    sim.client("client", async {
        let mut socket = turmoil::net::TcpStream::connect("server:8080").await?;
        socket.write_all(b"hello").await?;

        let mut buf = [0u8; 1024];
        let n = socket.read(&mut buf).await?;
        assert_eq!(&buf[..n], b"hello");
        Ok(())
    });

    sim.run().unwrap();
}

3. Seed Management

fn run_with_seed<F>(test_fn: F)
where
    F: FnOnce(u64) + std::panic::UnwindSafe
{
    let seed: u64 = std::env::var("TEST_SEED")
        .ok()
        .and_then(|s| s.parse().ok())
        .unwrap_or_else(|| rand::random());

    println!("TEST_SEED={}", seed);

    let result = std::panic::catch_unwind(|| test_fn(seed));

    if result.is_err() {
        eprintln!("Reproduce with: TEST_SEED={} cargo test", seed);
        std::panic::resume_unwind(result.unwrap_err());
    }
}

---

Common Patterns

Pattern 1: Deterministic HashMap

use std::collections::HashMap;
use std::hash::{BuildHasher, Hasher};

#[derive(Clone, Default)]
pub struct DeterministicHasher {
    state: u64,
}

impl Hasher for DeterministicHasher {
    fn finish(&self) -> u64 {
        self.state
    }

    fn write(&mut self, bytes: &[u8]) {
        const FNV_PRIME: u64 = 1099511628211;
        for &byte in bytes {
            self.state ^= byte as u64;
            self.state = self.state.wrapping_mul(FNV_PRIME);
        }
    }
}

#[derive(Clone, Default)]
pub struct DeterministicBuildHasher;

impl BuildHasher for DeterministicBuildHasher {
    type Hasher = DeterministicHasher;
    fn build_hasher(&self) -> Self::Hasher {
        DeterministicHasher { state: 14695981039346656037 }
    }
}

type DetHashMap<K, V> = HashMap<K, V, DeterministicBuildHasher>;

Pattern 2: Simulated Time (Tokio-only)

#[tokio::test(flavor = "current_thread", start_paused = true)]
async fn test_timeout() {
    let start = tokio::time::Instant::now();

    // This doesn't actually wait - time is simulated
    tokio::time::sleep(Duration::from_secs(3600)).await;

    // But an hour has "passed"
    assert_eq!(start.elapsed(), Duration::from_secs(3600));
}

Pattern 3: Abstract I/O Layer

#[cfg(not(feature = "simulation"))]
pub use tokio::net::{TcpListener, TcpStream};

#[cfg(feature = "simulation")]
pub use turmoil::net::{TcpListener, TcpStream};

Pattern 4: Fault Injection

let mut sim = Builder::new()
    .min_message_latency(Duration::from_millis(100))
    .max_message_latency(Duration::from_millis(500))
    .build();

// Network partition
sim.partition("node-a", "node-b");

// Run during partition
sim.run_until(Duration::from_secs(30));

// Heal partition
sim.repair("node-a", "node-b");

sim.run().unwrap();

---

Pitfalls to Avoid

Pitfall 1: Using std::time

// BAD ❌ - Bypasses simulated time
std::time::Instant::now()
std::thread::sleep(duration)

// GOOD ✅ - Uses simulated time
tokio::time::Instant::now()
tokio::time::sleep(duration).await

Pitfall 2: HashMap Iteration Order

// BAD ❌ - Order is non-deterministic
for (k, v) in hashmap.iter() {
    process_in_order(k, v); // Order affects result!
}

// GOOD ✅ - Use BTreeMap or sort first
let mut items: Vec<_> = hashmap.iter().collect();
items.sort_by_key(|(k, _)| *k);

Pitfall 3: Third-Party Crate Entropy

// BAD ❌ - uuid uses system entropy
let id = uuid::Uuid::new_v4();

// GOOD ✅ - Generate from seeded RNG
let id = uuid::Uuid::from_u128(rng.gen());

---

Examples

Network Partition Test

#[test]
fn test_survives_partition() {
    let seed = 42u64;
    let mut rng = StdRng::seed_from_u64(seed);

    let mut sim = Builder::new()
        .simulation_duration(Duration::from_secs(120))
        .build();

    // Create 3-node cluster
    for i in 0..3 {
        let node_name = format!("node-{}", i);
        sim.host(&node_name, || async move {
            // Consensus participant logic
            Ok(())
        });
    }

    // Run normally
    sim.run_until(Duration::from_secs(10));

    // Random partition
    let victim = rng.gen_range(0..3);
    sim.partition(
        &format!("node-{}", victim),
        &format!("node-{}", (victim + 1) % 3)
    );

    // Run during partition
    sim.run_until(Duration::from_secs(30));

    // Heal
    sim.repair(
        &format!("node-{}", victim),
        &format!("node-{}", (victim + 1) % 3)
    );

    // Verify recovery
    sim.run().unwrap();
}

Determinism Verification

#[test]
fn verify_determinism() {
    let seed = 12345u64;

    let result1 = run_and_capture(seed);
    let result2 = run_and_capture(seed);

    assert_eq!(result1, result2,
        "Non-determinism detected! Check for unseeded RNG or real time.");
}

fn run_and_capture(seed: u64) -> Vec<Event> {
    // Run simulation, capture events
    // Return events for comparison
}

---

Running Simulation Tests

# With random seed
cargo test --features simulation

# With specific seed for reproduction
TEST_SEED=12345 cargo test --features simulation

# Run specific test
TEST_SEED=42 cargo test test_survives_partition --features simulation

---

References

• TigerBeetle Blog: A Descent Into the Vortex
• S2.dev: Deterministic simulation testing for async Rust
• turmoil GitHub
• madsim GitHub
• FoundationDB Testing Talk

Examples

See

examples/

directory for working code:

• basic-dst-setup.md

  - Working DST test setup guide

• basic-dst-setup.rs

  - Complete DST test example code

See

docs/

directory for comprehensive guides:

• dst-rust-guide.md

  - Complete DST concepts and turmoil/madsim usage

• dst-internals-guide.md

  - Deep dive into DST implementation details

• README.md

  - DST documentation overview

Related Skills

• Rust with Async Code - For async patterns in production code
• Rust Testing Excellence - For general testing patterns

---

Last Updated: 2026-01-28 Version: 2.1-approved