Bigconfig Generator

Composable: Works with metadata-manager (for schema/metadata generation) and bigquery-etl-core (for conventions) When to use: Creating/updating Bigeye configurations, data quality monitoring

Overview

Generate and manage Bigeye monitoring configurations for BigQuery tables in the Mozilla bigquery-etl repository. Bigeye is Mozilla's data quality monitoring platform that checks for freshness, volume anomalies, null values, uniqueness violations, and custom business logic validation.

This skill helps configure monitoring through:

1. metadata.yaml - High-level monitoring settings (freshness, volume, collections)
2. bigconfig.yml - Detailed metric definitions (auto-generated via bqetl CLI)
3. bigeye_custom_rules.sql - Custom SQL validation rules (optional, for complex business logic)

Official Documentation:

• bigConfig Reference: https://mozilla.github.io/bigquery-etl/reference/bigconfig/ (docs/reference/bigconfig.md)
• Bigeye Intro: https://mozilla.github.io/data-docs/cookbooks/data_monitoring/intro.html
• Bigeye Official Docs: https://docs.bigeye.com/docs/bigconfig

🚨 REQUIRED READING - Start Here

BEFORE creating monitoring configurations, READ these resources:

1. Existing Collections: READ

   references/existing_collections.md

   • Collections already in use across the repository
   • Notification channels by dataset/team
   • Helps maintain consistency and avoid creating duplicate collections

2. Monitoring Patterns: READ

   references/monitoring_patterns.md

   • Common monitoring scenarios
   • Freshness vs volume monitoring
   • When to use custom rules
   • Configuration workflow

📋 Templates - Copy These Structures

When adding monitoring to metadata.yaml, READ and COPY from these templates:

• Basic monitoring (most tables)? → READ

  assets/metadata_monitoring_basic.yaml

  • Standard freshness and volume checks
  • Collection assignment

• Critical table (high priority)? → READ

  assets/metadata_monitoring_critical.yaml

  • More aggressive monitoring settings
  • Faster alerting

• View (non-partitioned)? → READ

  assets/metadata_monitoring_view.yaml

  • Monitoring for views without partitions

For custom validation rules:

• Custom SQL checks? → READ

  assets/custom_rules_template.sql

  • Template for bigeye_custom_rules.sql
  • Shows how to write validation queries

When to Use This Skill

Use this skill when:

• Creating new tables and user wants to enable monitoring
• User explicitly requests "create a bigeye config for..."
• User asks about adding data quality monitoring
• Setting up freshness or volume checks
• Creating custom validation rules
• Troubleshooting monitoring configurations

Integration with metadata-manager: When metadata-manager creates new tables, it should ask the user: "Would you like to enable Bigeye monitoring for this table?" If yes, invoke this skill.

🚨 IMPORTANT: Deployment Safety

Manual deployment is BLOCKED for safety reasons.

If a user asks to run

./bqetl monitoring deploy

, warn them:

⚠️ Manual deployment can accidentally delete existing metrics. The recommended workflow is to commit your changes and let the

bqetl_artifact_deployment

DAG deploy automatically. Manual deployment is disabled in this environment.

If you need to manually deploy for testing purposes, you'll need to:

1. Ensure you have

   BIGEYE_API_KEY

   set
2. Understand that deploying only specific tables can remove metrics from other tables
3. Use

   --dry-run

   first to review changes
4. Contact Data Engineering if you're unsure

Proceed with caution - this can affect production monitoring.

The standard workflow (update → validate → commit → push) is safe and recommended.

Prerequisites

• Table must have metadata.yaml file
• Table must be deployed to BigQuery
• Understanding of table's update schedule (daily, hourly, etc.)
• For manual deployment (discouraged):

  BIGEYE_API_KEY

  environment variable must be set

Staying Current with Documentation

Always prefer official documentation over this skill's references:

1. For bigConfig syntax and structure: Read docs/reference/bigconfig.md or use WebFetch on https://mozilla.github.io/bigquery-etl/reference/bigconfig/
2. For available saved metrics: Check sql/bigconfig.yml in the repository (source of truth)
3. For Bigeye concepts: Use WebFetch on https://mozilla.github.io/data-docs/cookbooks/data_monitoring/intro.html
4. For bqetl CLI commands: Check

   ./bqetl monitoring --help

   or the monitoring.py source code

When to use WebFetch:

• User asks about specific bigConfig features not covered in this skill
• Need to verify current syntax or available options
• References in this skill seem outdated or incomplete
• Troubleshooting issues not covered in common patterns

This skill focuses on workflow and decision-making rather than being a comprehensive bigConfig reference.

Workflow

Step 1: Determine Monitoring Requirements

Ask the user what type of monitoring they need:

For new tables created by metadata-manager: "Would you like to enable Bigeye monitoring for this table? This can check for:

• Freshness (when data was last updated)
• Volume (row count anomalies)
• Column-level validation (nulls, uniqueness, formats)
• Custom business logic validation"

For existing tables: "What type of monitoring would you like to configure?

1. Basic (freshness + volume)
2. Critical (freshness + volume with blocking)
3. Column-level validation
4. Custom SQL rules
5. All of the above"

After determining monitoring type, check existing collections:

Before configuring metadata.yaml, READ

references/existing_collections.md

to:

• Find the dataset in "Collections by Dataset" section
• Check if there's an existing collection for this dataset/team
• Note the notification channels used by similar tables

Ask the user: "Based on existing configurations, would you like to use the [Collection Name] collection with [notification channels]? Or create a new collection?"

Step 2: Configure metadata.yaml

Add a

monitoring

section to metadata.yaml based on table type:

• Basic (most tables):

  assets/metadata_monitoring_basic.yaml

  - Freshness + volume, non-blocking
• Critical (production):

  assets/metadata_monitoring_critical.yaml

  - Blocking failures, collection assignment
• Views:

  assets/metadata_monitoring_view.yaml

  - Requires explicit partition_column

Key settings:

• blocking: true

  - Failures block deployments (use for critical tables)

• collection

  - Groups related tables, configures alerts

• partition_column

  - Required for views (or null if non-partitioned)

Step 3: Generate bigconfig.yml

Use the bqetl CLI to auto-generate bigconfig.yml from metadata.yaml:

./bqetl monitoring update <dataset>.<table>

This command:

• Reads monitoring settings from metadata.yaml
• Generates appropriate metric definitions in bigconfig.yml
• Adds freshness/volume checks based on configuration
• Uses saved metrics from sql/bigconfig.yml

What gets generated:

• If

  freshness.enabled: true

  → Adds freshness metric
• If

  volume.enabled: true

  → Adds volume metric
• If

  blocking: true

  → Uses

  freshness_fail

  /

  volume_fail

  variants
• If

  collection

  specified → Groups under that collection

Step 4: Customize bigconfig.yml (Optional)

Manually edit the generated bigconfig.yml for advanced use cases:

Column-level validation: Add

tag_deployments

section with

column_selectors

and metrics (is_not_null, is_unique, is_valid_client_id, etc.). See

sql/bigconfig.yml

for all available saved metrics.

Lookback windows: Adjust how far back Bigeye scans data (0=latest partition, 7=last 7 days, 28=last 28 days). Use longer lookback for tables with sporadic updates.

When to customize: Column-specific validation, custom thresholds, infrequent updates, different notification channels per metric.

See

references/monitoring_patterns.md

for examples.

Step 5: Add Custom SQL Rules (Optional)

For complex business logic validation (cross-column checks, format validation, business rules), create

bigeye_custom_rules.sql

in the table directory.

Use template:

assets/custom_rules_template.sql

contains structure, JSON configuration block, and examples.

Key points:

• Query returns percentage (0-100) or count
• JSON comment block configures name, range, collections, owner, schedule
• Supports Jinja variables:

  {{ project_id }}

  ,

  {{ dataset_id }}

  ,

  {{ table_name }}

Step 6: Validate Configuration

Validate bigconfig.yml syntax and configuration:

./bqetl monitoring validate <dataset>.<table>

What it checks:

• Valid YAML syntax
• No duplicate metric deployments
• Saved metric IDs exist
• For views: partition_column is explicitly set in metadata.yaml

Common validation errors:

• "Duplicate deployments" → Consolidate metrics under single deployment
• "Invalid metric" → Check saved_metric_id exists in sql/bigconfig.yml
• "Partition column needs to be configured" → Set

  partition_column

  and

  partition_column_set: true

  for views

Step 7: Deploy to Bigeye

Recommended approach: Automatic deployment via Airflow DAG

After validation passes, commit and push your changes to the main branch:

git add sql/<project>/<dataset>/<table>/
git commit -m "Add Bigeye monitoring for <dataset>.<table>"
git push origin main

What happens automatically:

1. The

   bqetl_artifact_deployment

   DAG detects bigconfig.yml changes
2. The

   publish_bigeye_monitors

   task deploys all bigConfig files
3. Bigeye metrics are created/updated based on your configuration
4. Custom SQL rules are deployed (if bigeye_custom_rules.sql exists)

This approach is recommended because:

• Ensures all bigconfig.yml files are deployed together (prevents accidental deletions)
• No need to manage

  BIGEYE_API_KEY

  locally
• Consistent with Mozilla's deployment practices
• Deployment history tracked in git

Alternative: Manual deployment (discouraged)

⚠️ CAUTION: Avoid running

./bqetl monitoring deploy

locally unless absolutely necessary. Local deployment can accidentally delete metrics if config files are not included. See docs/reference/bigconfig.md for details.

If you must deploy manually (e.g., for testing in non-production):

./bqetl monitoring deploy <dataset>.<table> --dry-run  # Review changes first
./bqetl monitoring deploy <dataset>.<table>            # Requires BIGEYE_API_KEY

Step 8: Test Monitoring (Optional)

After deployment, you can manually trigger monitoring checks to verify configuration:

./bqetl monitoring run <dataset>.<table>  # Requires BIGEYE_API_KEY

What it does:

• Triggers all metric checks for the table
• Runs custom SQL rules
• Returns success/failure status
• Provides links to Bigeye UI for details

When to test:

• After automatic deployment via DAG completes
• After modifying monitoring configuration
• Debugging false positives/negatives

Alternative: Wait for Bigeye's scheduled runs or check results in the Bigeye UI

Common Monitoring Patterns

Standard workflow for all patterns:

1. Add/update

   monitoring

   section in metadata.yaml
2. Run:

   ./bqetl monitoring update <dataset>.<table>

3. Run:

   ./bqetl monitoring validate <dataset>.<table>

4. Commit and push to main branch (automatic deployment)

Pattern 1: Basic Daily Table

Use

assets/metadata_monitoring_basic.yaml

template. Enables freshness and volume checks, non-blocking.

Pattern 2: Critical Production Table

Use

assets/metadata_monitoring_critical.yaml

template. Sets

blocking: true

and assigns to "Operational Checks" collection.

Pattern 3: View with Monitoring

Use

assets/metadata_monitoring_view.yaml

template. Must set

partition_column

and

partition_column_set: true

.

Pattern 4: Column-Level Validation

After generating basic bigconfig.yml, manually edit to add column-specific metrics. See

sql/bigconfig.yml

for available saved metrics (is_not_null, is_unique, is_valid_client_id, etc.).

Pattern 5: Custom Business Logic

Create

bigeye_custom_rules.sql

using

assets/custom_rules_template.sql

. Query must return percentage (0-100) or count. Configure via JSON comment block.

Integration with Other Skills

Works with metadata-manager

When metadata-manager creates new tables:

• metadata-manager should ask: "Would you like to enable Bigeye monitoring?"
• If yes, metadata-manager invokes bigconfig-generator skill
• bigconfig-generator adds monitoring configuration to metadata.yaml
• Generates bigconfig.yml via bqetl CLI

Workflow:

1. metadata-manager creates schema.yaml, metadata.yaml
2. metadata-manager asks about monitoring
3. If yes → invoke bigconfig-generator
4. bigconfig-generator adds monitoring section to metadata.yaml
5. bigconfig-generator runs

   ./bqetl monitoring update

6. User validates, commits, and pushes to main (automatic deployment via DAG)

Works with bigquery-etl-core

• Uses project structure conventions
• Follows naming patterns (dataset.table)
• References common partitioning strategies (submission_date)

Troubleshooting

Deployment Errors

Deployment delays:

• Deployment happens automatically after merge to main via

  bqetl_artifact_deployment

  DAG
• Check DAG status in Airflow UI if deployment seems delayed
• Typical deployment time: within 1 hour of merge

"Table does not exist in Bigeye"

• Table not yet ingested by Bigeye
• Wait for next schema sync or manually sync in Bigeye UI
• Check with Data Engineering if table is not appearing

"Partition column does not exist"

• Verify

  partition_column

  matches actual column in schema.yaml
• Check for typos in column name

Manual deployment errors (if using ./bqetl monitoring deploy): "Bigeye API token needs to be set"

• Set

  BIGEYE_API_KEY

  environment variable
• Note: Manual deployment is discouraged; prefer automatic DAG deployment

Validation Errors

"Duplicate deployments"

• Same column selector appears multiple times
• Consolidate metrics under single deployment

"Invalid metric"

• Referencing non-existent saved_metric_id
• Check sql/bigconfig.yml for available metrics

"Partition column needs to be configured"

• For views with monitoring enabled
• Add

  partition_column

  and

  partition_column_set: true

  to metadata.yaml

False Positives

Freshness checks failing:

• Verify table actually updated (query BigQuery)
• Check partition_column is correct
• Verify Bigeye's schedule aligns with table update schedule
• Consider longer lookback window

Volume checks failing:

• Normal for tables with varying row counts
• Consider disabling volume checks
• Use longer lookback window
• Adjust thresholds in bigconfig.yml

Best Practices

When to Enable Monitoring

Always enable:

• Production tables in dashboards/reports
• Tables with SLAs or freshness requirements
• Critical pipeline outputs

Consider enabling:

• Development/staging tables (for testing configs)
• Tables with known data quality issues

Skip monitoring:

• Temporary/scratch tables
• One-time analysis tables
• Tables with no consumers

Blocking vs Non-Blocking

Use

blocking: true

when:

• Failures must halt deployments
• Table is production-critical
• False positives are rare and quickly resolved

Use

blocking: false

when:

• Failures should alert but not block
• Table is still stabilizing
• False positives are expected

Collections

Use consistent naming:

• Group related tables by team/product
• Configure notification channels once per collection
• Makes alert management easier

Common collections:

• Team: "Subscription Platform", "Ads Team", "Growth Team"
• Function: "Operational Checks", "Data Quality"
• Environment: "Test", "Staging"

Custom Rules

Best practices:

• Return percentage (0-100) for "value" alert_conditions
• Return count for "count" alert_conditions
• Use descriptive rule names
• Set appropriate min/max ranges
• Document rule purpose in comments
• Test rules manually before deploying

Reference Documentation

Official Documentation (Always Preferred):

• docs/reference/bigconfig.md - Canonical reference for bigConfig in this repository
• sql/bigconfig.yml - Source of truth for available saved metrics
• https://mozilla.github.io/bigquery-etl/reference/bigconfig/ - Published docs
• https://mozilla.github.io/data-docs/cookbooks/data_monitoring/intro.html - Bigeye intro
• https://docs.bigeye.com/docs/bigconfig - Bigeye official documentation

Quick Reference (This Skill):

• references/monitoring_patterns.md

  - Workflow guidance and common patterns (may be outdated)

• assets/metadata_monitoring_basic.yaml

  - Basic monitoring config template

• assets/metadata_monitoring_critical.yaml

  - Critical table config template

• assets/metadata_monitoring_view.yaml

  - View monitoring config template

• assets/custom_rules_template.sql

  - Custom SQL rule template

Priority: When in doubt, read docs/reference/bigconfig.md or use WebFetch on the online docs.

Quick Reference: bqetl Monitoring Commands

# Refresh the collections reference file (run periodically to stay current)
python3 .claude/skills/bigconfig-generator/scripts/extract_collections.py

# Generate/update bigconfig.yml from metadata.yaml
./bqetl monitoring update <dataset>.<table>

# Validate bigconfig.yml syntax and configuration
./bqetl monitoring validate <dataset>.<table>

# ⚠️ DISCOURAGED: Manual deployment (prefer automatic DAG deployment)
./bqetl monitoring deploy <dataset>.<table> --dry-run  # Requires BIGEYE_API_KEY
./bqetl monitoring deploy <dataset>.<table>            # Requires BIGEYE_API_KEY

# Manually trigger monitoring checks (requires BIGEYE_API_KEY)
./bqetl monitoring run <dataset>.<table>

# Delete deployed monitoring (requires BIGEYE_API_KEY)
./bqetl monitoring delete <dataset>.<table> --metrics --custom-sql

Recommended workflow:

1. Check

   references/existing_collections.md

   for appropriate collection/channels
2. Update/create bigconfig.yml using

   monitoring update

3. Validate using

   monitoring validate

4. Commit and push to main branch

5. bqetl_artifact_deployment

   DAG automatically deploys changes