Schema E2E Validation

Self-Evolving Skill: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.

When to Use This Skill

Use this skill when:

• Validating schema changes before commit
• Verifying YAML schema matches live ClickHouse Cloud
• Regenerating Python types, DDL, or docs
• Running full schema workflow validation

Prerequisites

Docker Runtime (Required)

Earthly requires Docker. Start Colima before running:

colima start

Check if running:

docker ps  # Should not error

Doppler Access (For validation targets)

Required for

+test-schema-validate

+test-schema-e2e

doppler configure set token <token_from_1password>
doppler setup --project gapless-network-data --config prd

Earthly Installation

brew install earthly

Quick Commands

Generation only (no secrets)

cd ~/eon/gapless-network-data
colima start  # If not already running
earthly +test-schema-generate

Full E2E with validation (requires Doppler)

cd ~/eon/gapless-network-data
colima start  # If not already running
./scripts/earthly-with-doppler.sh +test-schema-e2e

All non-secret targets

cd ~/eon/gapless-network-data
earthly +all

Artifacts

After running

+test-schema-generate

+test-schema-e2e

./earthly-artifacts/

types/blocks.py

types/__init__.py

ddl/ethereum_mainnet.sql

docs/ethereum_mainnet.md

For E2E, artifacts are under

e2e/types/

e2e/ddl/

e2e/docs/

Earthfile Targets Reference

+deps

+build

+test-unit

+test-schema-generate

+test-schema-validate

+test-schema-e2e

+all

Troubleshooting

"could not determine buildkit address - is Docker or Podman running?"

Cause: Docker/Colima not running

Fix:

colima start
# Wait for "done" message, then retry
earthly +test-schema-generate

"unable to parse --secret-file argument"

Cause: Wrong flag name or malformed secrets file

Fix: The correct flag is

--secret-file-path

--secret-file

# WRONG
earthly --secret-file=/path/to/secrets +target

# CORRECT
earthly --secret-file-path=/path/to/secrets +target

Also ensure secrets file has no quotes around values:

# WRONG format
CLICKHOUSE_HOST="host.cloud"

# CORRECT format
CLICKHOUSE_HOST=host.cloud

"OSError: Readme file does not exist: README.md"

Cause: hatchling build backend requires README.md in container

Fix: Ensure Earthfile copies README.md in deps target:

deps:
    COPY pyproject.toml uv.lock README.md ./  # README.md required!

"missing secret" during validation

Cause: Doppler not configured or secrets not passed

Fix:

# Verify Doppler has the secrets
doppler secrets --project gapless-network-data --config prd | grep CLICKHOUSE

# Use the wrapper script (handles secret injection)
./scripts/earthly-with-doppler.sh +test-schema-validate

Cache Issues

Force rebuild without cache:

earthly --no-cache +test-schema-e2e

Implementation Details

Doppler Secret Injection

The wrapper script

scripts/earthly-with-doppler.sh

1. Downloads secrets from Doppler
2. Filters for

   CLICKHOUSE_*

   variables
3. Strips quotes (Doppler outputs

   KEY="value"

   , Earthly needs

   KEY=value

   )
4. Passes via

   --secret-file-path

   flag
5. Cleans up temp file on exit

Secrets Required

CLICKHOUSE_HOST_READONLY

CLICKHOUSE_USER_READONLY

CLICKHOUSE_PASSWORD_READONLY

Related Files

~/eon/gapless-network-data/Earthfile

~/eon/gapless-network-data/scripts/earthly-with-doppler.sh

~/eon/gapless-network-data/schema/clickhouse/ethereum_mainnet.yaml

~/eon/gapless-network-data/docs/adr/2025-12-03-earthly-schema-e2e.md

Validation History

• 2025-12-03: Created and validated with full E2E run against ClickHouse Cloud
• Lessons Learned:

  • --secret-file-path

    not

    --secret-file

    (Earthly v0.8.16)
  • Doppler

    --format env

    outputs quotes, must strip with

    sed 's/"//g'

  • README.md must be copied for hatchling build backend
  • Colima must be started before Earthly runs

Design Authority

This skill validates schemas but does not design them. For schema design guidance (ORDER BY, compression, partitioning), invoke

quality-tools:clickhouse-architect

Related Skills

quality-tools:clickhouse-architect

devops-tools:clickhouse-cloud-management

devops-tools:clickhouse-pydantic-config

Post-Execution Reflection

After this skill completes, check before closing:

1. Did the command succeed? — If not, fix the instruction or error table that caused the failure.
2. Did parameters or output change? — If the underlying tool's interface drifted, update Usage examples and Parameters table to match.
3. Was a workaround needed? — If you had to improvise (different flags, extra steps), update this SKILL.md so the next invocation doesn't need the same workaround.

Only update if the issue is real and reproducible — not speculative.