Project Engines Benchmark Skill

======= description: Run and interpret basin-engines benchmarks (Steel, ember, shale) model: haiku

Basin Engines Benchmark Skill

origin/main

Run benchmarks for Steel, ember, and shale engines.

CRITICAL: Read Before Benchmarking

<<<<<<< HEAD ALWAYS read first:

~/p/benchmark-suite/docs/BENCHMARK_FAIRNESS.md

ALWAYS read first:

~/p/basin-bench/docs/BENCHMARK_FAIRNESS.md

origin/main

This document contains hard-won lessons about benchmark fairness. Ignoring it leads to misleading claims.

Pre-Benchmark Checklist

cat ~/p/benchmark-suite/docs/BENCHMARK_FAIRNESS.md

cat ~/p/basin-bench/docs/BENCHMARK_FAIRNESS.md

origin/main | Use

--batched

for LMDB/redb | 7-24x improvement with proper config | Add

--batched --batch-size 1000

| | Scale sled cache | Undersized cache = 17x slower | Add

--cache-mb 2048

for 1M+ records | | Check dataset vs RAM | If data fits in RAM, you're measuring memory | Use larger datasets for I/O testing |

Note: Steel uses verify-once checksums (like RocksDB/WiredTiger) - verify on first read from disk, then trust page cache. Use

FileLayoutConfig::fast()

Quick Commands

Steel (Oak engine)

# Build
<<<<<<< HEAD
cd ~/p/benchmark-suite && graft build --release -p ycsb-steel
=======
cd ~/p/basin-bench && graft build --release -p ycsb-steel
>>>>>>> origin/main

# Single-threaded
ycsb-steel --fast --data-dir /tmp/bench --workload a --records 50000 --ops 200000

# Multi-threaded with sharding
ycsb-steel --fast --shards 64 --threads 4 --data-dir /tmp/bench --workload a --records 50000 --ops 200000

# Ultimate adversarial benchmark (vs sled)
<<<<<<< HEAD
cd ~/p/engine-stack/engines/steel
=======
cd ~/p/basin-engines/engines/steel
>>>>>>> origin/main
graft run --release --example ultimate_adversarial

Fair 4-Engine Comparison

# Use the fair comparison script (includes proper batching for all engines)
<<<<<<< HEAD
RECORDS=50000 OPS=200000 ~/p/benchmark-suite/scripts/steel-fair-compare.sh
=======
RECORDS=50000 OPS=200000 ~/p/basin-bench/scripts/steel-fair-compare.sh
>>>>>>> origin/main

Individual Engine Commands (Fair Config)

# Steel
ycsb-steel --fast --workload a --records 50000 --ops 200000 --data-dir /tmp/bench

# sled (scaled cache)
ycsb-sled --high-throughput --cache-mb 256 --workload a --records 50000 --ops 200000 --data-dir /tmp/bench

# LMDB (batched + nosync)
ycsb-lmdb --batched --nosync --batch-size 1000 --workload a --records 50000 --ops 200000 --data-dir /tmp/bench

# redb (batched)
ycsb-redb --batched --batch-size 1000 --workload a --records 50000 --ops 200000 --data-dir /tmp/bench

Steel Results (2025-12-23) - Steel Wins All

Steel now beats LMDB on ALL workloads!

Optimizations That Closed the Gap

Implemented (see

docs/STEEL_OPTIMIZATIONS.md

• get_ref()

  +8.4% - zero-copy reads (KEY WIN)

• get_cached_epoch()

  +1% - thread-local epoch

• get_fast()

  - seqlock skip (no gain, kept for API)

Gap closed! Previous 43% gap on Workload B eliminated via zero-copy optimization.

Where Steel Actually Wins

Sharded Write Performance (2025-12-25)

With 64 shards, Steel beats sled by 2.3x:

Where Steel Does NOT Win

Common Mistakes (Avoid These)

--batched

--batched --batch-size 1000

--batched

--batched --batch-size 1000

Key Files

~/p/benchmark-suite/engines/ycsb-steel/

~/p/benchmark-suite/scripts/steel-fair-compare.sh

~/p/benchmark-suite/docs/BENCHMARK_FAIRNESS.md

~/p/engine-stack/engines/steel/BENCHMARKS.md

~/p/engine-stack/engines/steel/ROADMAP_BEST_KV.md

~/p/engine-stack/engines/steel/examples/ultimate_adversarial.rs

~/p/basin-bench/engines/ycsb-steel/

~/p/basin-bench/scripts/steel-fair-compare.sh

~/p/basin-bench/docs/BENCHMARK_FAIRNESS.md

~/p/basin-engines/engines/steel/BENCHMARKS.md

~/p/basin-engines/engines/steel/ROADMAP_BEST_KV.md

~/p/basin-engines/engines/steel/examples/ultimate_adversarial.rs

origin/main

Dialectical Improvement

When benchmarking, always ask:

1. "What would a competitor's maintainer criticize about this benchmark?"
2. "Am I using each engine's recommended configuration?"
3. "What am I NOT measuring that matters?"
4. "Is this result surprising? If so, investigate before publishing."