Celery is a distributed task queue for UNIX systems. Celery is useful for background task processing and deferred execution in Django. Task queues are used to distribute work across workers.
Two common use cases of Celery:
- Celery workers are worker processes that run tasks independently from one another and outside the context of your main service.
- Celery beat is a scheduler that orchestrates when to run tasks. You can use it to schedule periodic tasks as well.
Celery’s components:
- message broker — component for exchange messages and tasks distribution between workers, popular implementations are RabbitMQ and Redis. To receive tasks from your program and send results to a back end, Celery requires a message broker for communication.
- worker — calculation unit, which executes the task.
- application — an application that requires background task processing or deferred execution.
- task — unit for asynchronous execution.
- result backends (result storage) — place where we can store results (database) after deferred execution or background task processing, popular implementation are SQLAlchemy/Django ORM, Memcached, Redis, AMQP (RabbitMQ), and MongoDB.
Note:
The idea behind result-backend is that keeping a record of all task results is often helpful, especially if you’re distributing tasks to multiple queues. To persist information about task results, you need a database back end.
Warning:
Backends use resources to store and transmit results. To ensure that resources are released, you must eventually call get() or forget() on EVERY AsyncResult instance returned after calling a task.
Input for celery is message broker, and output is result backends (if required).