Celery Development Best Practices

Comprehensive background task processing framework with 60+ prioritized rules across 21 categories for building reliable, scalable distributed task systems with Celery.

When to Apply

Reference these guidelines when:

• Implementing background task processing
• Setting up distributed job queues
• Configuring periodic tasks with Celery Beat
• Optimizing task performance and reliability
• Monitoring and debugging Celery applications
• Implementing task routing and result handling
• Managing Celery workers and brokers
• Ensuring security in distributed task environments
• Designing complex workflows with Canvas primitives
• Handling serialization and data transmission
• Deploying Celery in production environments

Rule Categories by Priority

config-

task-

error-

canvas-

monitoring-

perf-

perf-

security-

result-

routing-

periodic-

serial-

worker-

signals-

remote-

testing-

daemonization-

debugging-

extensions-

advanced-

Quick Reference

1. Configuration & Setup (CRITICAL)

• config-broker-setup

  : Properly configure message broker (Redis/RabbitMQ)

• config-environment-separation

  : Separate configurations for different environments

2. Task Definition & Execution (CRITICAL)

• task-calling-methods

  : Choose appropriate calling methods (delay/apply_async/call)

• task-atomic-operations

  : Ensure tasks are atomic and idempotent

3. Error Handling & Reliability (HIGH)

• error-retry-strategy

  : Implement intelligent retry mechanisms

2. Task Definition & Execution (CRITICAL)

• task-calling-methods

  : Choose appropriate calling methods (delay/apply_async/call)

• task-atomic-operations

  : Ensure tasks are atomic and idempotent

• task-options-configuration

  : Configure task behavior and reliability options

3. Error Handling & Reliability (HIGH)

• error-retry-strategy

  : Implement intelligent retry mechanisms

4. Canvas: Designing Work-flows (HIGH)

• canvas-chain-workflows

  : Sequential task execution and error propagation

• canvas-group-parallel

  : Parallel task execution for performance

• canvas-chord-synchronization

  : Complex workflows with parallel execution and synchronization

• canvas-map-starmap

  : Efficient batch processing and data transformations

5. Monitoring & Logging (HIGH)

• monitoring-task-tracking

  : Track task execution and performance

• perf-monitoring-metrics

  : Performance optimization and issue detection

6. Performance & Scaling (MEDIUM-HIGH)

• perf-concurrency-tuning

  : Optimize worker concurrency settings

7. Security (MEDIUM-HIGH)

• security-task-authentication

  : Prevent unauthorized task execution

8. Result Backends (MEDIUM-HIGH)

• result-backend-selection

  : Choose appropriate result backend

• result-expiry-management

  : Manage result expiration policies

9. Routing & Queues (MEDIUM)

• routing-task-distribution

  : Intelligent task routing and queue management

10. Periodic Tasks (MEDIUM)

• periodic-celery-beat

  : Reliable scheduled task execution

11. Serialization (MEDIUM)

• serial-data-handling

  : Reliable data transmission and storage

12. Worker Management (MEDIUM)

• worker-lifecycle-management

  : Reliable worker operation and resource management

13. Signals and Events (MEDIUM)

• signals-events-system

  : Hook into Celery events for monitoring and extension

14. Remote Control and Inspection (MEDIUM)

• remote-control-inspection

  : Runtime management and troubleshooting of workers

15. Testing Celery Applications (MEDIUM-HIGH)

• testing-celery-applications

  : Comprehensive testing strategies for async applications

16. Daemonization and Process Management (MEDIUM)

• daemonization-process-management

  : Production deployment and service management

17. Debugging Celery Applications (HIGH)

• debugging-celery-applications

  : Effective troubleshooting and issue resolution

18. Advanced Performance Optimization (MEDIUM-HIGH)

• advanced-performance-optimization

  : High-throughput optimization techniques

19. Extensions and Bootsteps Customization (LOW)

• extensions-bootsteps-customization

  : Advanced customization and extension

20. Advanced Patterns (LOW)

• advanced-task-chaining

  : Complex workflow orchestration

How to Use

Read individual rule files for detailed explanations and code examples:

rules/config-broker-setup.md                 # Broker configuration
rules/task-calling-methods.md                # Task execution methods
rules/task-options-configuration.md          # Task behavior options
rules/canvas-chain-workflows.md              # Sequential workflows
rules/canvas-group-parallel.md               # Parallel execution
rules/canvas-chord-synchronization.md        # Complex synchronization
rules/canvas-map-starmap.md                  # Batch processing
rules/monitoring-task-tracking.md            # Task monitoring
rules/result-backend-selection.md            # Result storage
rules/routing-task-distribution.md           # Task routing
rules/periodic-celery-beat.md                # Scheduled tasks
rules/serial-data-handling.md                # Data serialization
rules/signals-events-system.md               # Event hooking
rules/remote-control-inspection.md           # Worker management
rules/testing-celery-applications.md         # Testing strategies
rules/security-task-authentication.md        # Task security
rules/daemonization-process-management.md    # Production deployment
rules/debugging-celery-applications.md       # Troubleshooting
rules/advanced-performance-optimization.md   # High-performance tuning
rules/extensions-bootsteps-customization.md  # Advanced customization

Each rule file contains:

• Impact level and category
• Problem description with consequences
• Solution with code examples
• Common mistakes to avoid
• When to apply the rule

Reference Documentation

Detailed guides for complex topics:

references/worker-management.md      # Worker deployment and scaling

Additional reference guides planned: monitoring setup, troubleshooting, Canvas patterns

Full Compiled Document

For the complete Celery best practices guide with all rules expanded:

AGENTS.md

Celery Version Support: Compatible with Celery 4.x, 5.x, and 6.x (latest)</content> <parameter name="filePath">skills/celery-skill/SKILL.md