Data Quality Monitoring

Skill Profile

(Select at least one profile to enable specific modules)

• DevOps
• Backend
• Frontend
• AI-RAG
• Security Critical

Overview

Data Quality (DQ) Monitoring is the continuous process of validating data against predefined rules and expectations. In a modern data stack, monitoring must happen at every stage: Ingestion, Transformation, and Serving.

Core Principle: "Garbage in, garbage out. If data is wrong, analytics, AI, and decisions will be wrong too."

This skill provides comprehensive guidance on implementing data quality monitoring across the data pipeline, from automated testing to real-time anomaly detection.

Why This Matters

• <Benefit>: <short explanation>
• <Benefit>: <short explanation>
• <Benefit>: <short explanation>

Core Concepts & Rules

1. Core Principles

• Follow established patterns and conventions
• Maintain consistency across codebase
• Document decisions and trade-offs

2. Implementation Guidelines

• Start with the simplest viable solution
• Iterate based on feedback and requirements
• Test thoroughly before deployment

Inputs / Outputs / Contracts

• Inputs:
  • <e.g., env vars, request payload, file paths, schema>
• Entry Conditions:
  • <Pre-requisites: e.g., Repo initialized, DB running, specific branch checked out>
• Outputs:
  • <e.g., artifacts (PR diff, docs, tests, dashboard JSON)>
• Artifacts Required (Deliverables):
  • <e.g., Code Diff, Unit Tests, Migration Script, API Docs>
• Acceptance Evidence:
  • <e.g., Test Report (screenshot/log), Benchmark Result, Security Scan Report>
• Success Criteria:
  • <e.g., p95 < 300ms, coverage ≥ 80%>

Skill Composition

• Depends on: None
• Compatible with: None
• Conflicts with: None
• Related Skills: None

Quick Start

Assumptions

• Data streams are accessible for monitoring
• Historical data is available for baselines
• Team has capacity to respond to alerts
• Monitoring tools are compatible with data stack

Compatibility

• Works with any SQL database
• Compatible with all major data platforms
• Framework-agnostic approach
• Adaptable to real-time and batch systems

Test Scenario Matrix

Scenario	Test Case	Expected Outcome
Null values	Monitor for nulls in required fields	Alert when threshold exceeded
Freshness	Monitor data latency	Alert if data is stale
Volume	Monitor row count	Alert on sudden drops or spikes
Distribution	Monitor statistical properties	Alert on significant shifts
Schema	Monitor schema changes	Alert on column additions/removals
Accuracy	Compare to source system	Alert on mismatches

Technical Guardrails & Security Threat Model

1. Security & Privacy (Threat Model)

• Top Threats: Injection attacks, authentication bypass, data exposure

• Data Handling: Sanitize all user inputs to prevent Injection attacks. Never log raw PII
• Secrets Management: No hardcoded API keys. Use Env Vars/Secrets Manager
• Authorization: Validate user permissions before state changes

2. Performance & Resources

• Execution Efficiency: Consider time complexity for algorithms
• Memory Management: Use streams/pagination for large data
• Resource Cleanup: Close DB connections/file handlers in finally blocks

3. Architecture & Scalability

• Design Pattern: Follow SOLID principles, use Dependency Injection
• Modularity: Decouple logic from UI/Frameworks

4. Observability & Reliability

• Logging Standards: Structured JSON, include trace IDs

  request_id

• Metrics: Track

  error_rate

  ,

  latency

  ,

  queue_depth

• Error Handling: Standardized error codes, no bare except
• Observability Artifacts:
  • Log Fields: timestamp, level, message, request_id
  • Metrics: request_count, error_count, response_time
  • Dashboards/Alerts: High Error Rate > 5%

Agent Directives

1. Always monitor - Never skip monitoring for production data
2. Alert on failures - All quality failures should trigger alerts
3. Block downstream - Prevent bad data from propagating
4. Track metrics - Monitor quality over time
5. Iterate on rules - Update rules based on incidents

Definition of Done

Data quality monitoring implementation is complete when:

• Monitoring rules defined for all critical tables
• Real-time checks implemented where needed
• Alerting configured for all quality failures
• Dashboard built and accessible to stakeholders
• Incident procedures documented
• Team trained on monitoring tools
• Regular review schedule established
• Integration with pipeline orchestration
• Performance optimization completed
• Continuous improvement process in place

Anti-patterns

1. Monitoring only at load - Monitor at every stage
2. Silent failures - All failures should alert
3. Over-monitoring - Too many checks cause alert fatigue
4. Ignoring alerts - Every alert needs attention
5. Static rules - Rules must evolve with data

Reference Links

Versioning

This skill follows semantic versioning (MAJOR.MINOR.PATCH):

• MAJOR: Breaking changes to procedures or standards
• MINOR: New monitoring methods or significant enhancements
• PATCH: Bug fixes or documentation updates