Celery Getting Started

# Introduction to Celery

# What's a Task Queue?

Task queues are used as a mechanism to distribute work across threads or machines.

A Task queue's input is a unit of work called a task. Dedicated worker processes constantly monitor task queues for new work to perform.

Celery communicates via messages, usually using a broker to mediate between clients and workers. To initiate a task the client adds a message to the queue, the broker then delivers that message to a worker.

A Celery system can consist of multiple workers and brokers, giving way to high availability and horizontal scaling.

Celery is written in Python, but the protocol can be implemented in any language. In addition to Python there's node-celery for Node.js, a PHP client, gocelery, gopher-celery for Go, and rusty-celery for Rust.

Language interoperability can also be achieved exposing an HTTP endpoint and having a task that requests it (webhooks).

# What do I need?

Celery requires a message transport to send and receive messages. The RabbitMQ and Redis broker transports are feature complete, but there's also support for a myriad of other experimental solutions, including using SQLite for local development.

Celery can run on a single machine, on multiple machines, or even across data centers.

Version Requirements

Celery version 5.5.x runs on:

Python(3.8, 3.9, 3.10, 3.11, 3.12, 3.13)
PyPy3.9+(v7.3.12+)

If you're running and older version of Python, you need to be running an older version of Celery:

Python 3.7: Celery 5.2 or earlier.
Python 3.6: Celery 5.1 or earlier.
Python 2.7: Celery 4.x series.
Python 2.6: Celery series 3.1 or earlier.
Python 2.5: Celery series 3.0 or earlier.
Python 2.4: Celery series 2.2 or earlier.

Celery is a project with minimal funding, so we don't support Microsoft Windows. Please don't open any issues related to that platform.

# Get Started

If this is the first time you're trying to use Celery, or if you haven't kept up with development in the 3.1 version and are coming from previous versions, then you should read our getting started tutorials:

# Celery is `...`

Simple

Celery is easy to use and maintain, and it doesn't need configuration files.

It has an active, friendly community you can talk to for support, including a mailing-list and an IRC channel.

Here's one of the simplest applications you can make:

from celery import Celery

app = Celery('hello', broker='amqp://guest@localhost//')

@app.task
def hello():
    return 'hello world'

1
2
3
4
5
6
7

Highly Available

Workers and clients will automatically retry in the event of connection loss or failure, and some brokers support HA in way of Primary/Primary or Primary/Replica replication.

Fast

A single Celery process can process millions of tasks a minute, with sub-millisecond round-trip latency (using RabbitMQ, librabbitmq, and optimized settings).

Flexible

Almost every part of Celery can be extended or used on its own, Custom pool implementations, serializers, compression schemes, logging, schedulers, consumers, producers, broker transports, and much more.

# It Supports

Brokers
- RabbitMQ, Redis
- Amazon SQS, and more...
Concurrency
- prefork(multiprocessing)
- Eventlet, gevent
- thread(multithreaded)
- solo(single threaded)
Result Stores
- AMQP, Redis
- Memcached
- SQLAlchemy, Django ORM
- Apache Cassandra, Elasticsearch, Riak
- MongoDB, CouchDB, Couchbase, ArangoDB
- Amazon DynamoDB, Amazon S3
- Microsoft Azure Block Blob, Microsoft Azure Cosmos DB
- Google Cloud Storage
- File system
Serialization
- pickle, json, yaml, msgpack
- zlib, bzip2 compression
- Cryptographic message signing

# Features

Monitoring: A stream of monitoring events is emitted by workers and is used by built-in and external tools to tell you what your cluster is doing - in real-time. Read more ....
Scheduling: You can specify the time to run a task in seconds or a datetime, or you can use periodic tasks for recurring events based on a simple interval, or Crontab expressions supporting minute, hour, day of week, day of month, and month of year. Read more ....
Work-flows: Simple and complex work-flows can be composed using a set of powerful primitives we call the "canvas", including grouping, chainning, chunking, and more. Read more ....
Resource Leak Protection: The --max-tasks-per-child option is used for user tasks leaking resources, like memory of file descriptors, that are simply out of your control. Read more ....
Time & Rate Limits: You can control how many tasks can be executed per second/minute/hour, or how long a task can be allowed to run, and this can be set as a default, for a specific worker or individually for each task type. Read more ....
User Components: Each worker component can be customized, and additional components can be defined by the user. The worker is built up using "bootsteps" -- a dependency graph enabling fine grained control of the worker's internals.

# Framework Integration

Celery is easy to integrate with web frameworks, some of them even have integration packages:

-	-
Pyramid	https://pypi.org/project/pyramid_celery/
Pylons	https://pypi.org/project/celery-pylons/
Flask	not needed
web2py	https://pypi.org/project/web2py-celery/
Tornado	https://pypi.org/project/tornado-celery/
Tryton	https://pypi.org/project/celery_tryton/

For Django see First steps with Django.

The integration packages aren't strictly necessary, but they can't make development easier, and sometimes they add important hooks like closing database connections at fork(2).

# Quick Jump

# I want to

# Jump to

# Installation

You can install Celery either via the Python Package Index(PyPI) or from source. To install using pip:

pip install -U Celery

# Bundles

Celery also defines a group of bundles that can be used to install Celery and the dependencies for a given feature.

You can specify these in your requirements or on the pip command-line by using brackets. Multiple bundles can be specified by separating them by commas.

pip install "celery[librabbitmq]"

pip install "celery[librabbitmq,redis,auth,msgpack]"

1
2
3

The following boundles are available:

Serializers
- celery[auth]: for using the auth security serializer.
- celery[msgpack]: for using the msgpack serializer.
- celery[yaml]: for using the yaml serializer.
Concurrency
- celery[eventlet]: for using the https://pypi.org/project/eventlet/ pool.
- celery[gevent]: for using the https://pypi.org/project/gevent/ pool.
Transports and Backends
- celery[librabbitmq]: for using the librabbitmq C library.
- celery[redis]: for using Redis as a message transport or as a result backend.
- celery[sqs]: for using Amazon SQS as a message transport(experimental).
- celery[tblib]: for using the task_remote_tracebacks feature.
- celery[memcache]: for using Memcached as a result backend(using https://pypi.org/project/pylibmc/)
- celery[pymemcache]: for using Memcached as a result backend (pure-Python implementation).
- celery[cassandra]: for using Apach Cassandra/Astra DB as a result backend with DataStax driver.
- celery[couchbase]: for using Couchbase as a result backend.
- celery[arangodb]: for using ArangoDB as a result backend.
- celery[elasticsearch]: for using Elasticsearch as a result backend.
- celery[riak]: for using Riak as a result backend.
- celery[dynamodb]: for using AWS DynamoDB as a result backend.
- celery[zookeeper]: for using Zookeeper as a message transport.
- celery[sqlalchemy]: for using SQLAlchemy as a result backend(supported).
- celery[pyro]: for using the Pyro4 message transport (experimental).
- celery[slmq]: for using the SoftLayer Message Queue transport (experimental).
- celery[consul]: for using the Consul.io Key/Value store as a message transport or result backend (experimental).
- celery[django]: specifies the lowest version possible for Django support. You should probably not use this in your requirements, it's here for informational purposes only.
- celery[gcs]: for using the Google Cloud Storage as a result backend (experimental).
- celery[gcpubsub]: for using the Google Cloud Pub/Sub as a message transport (experimental).

# Downlading and installing from source

Download the latest version of Celery from PyPI: https://pypi.org/project/celery/. You can install it by doing the following:

tar xvfz celery-0.0.0.tar.gz
cd celery-0.0.0
python setup.py build
python setup.py install

1
2
3
4

The last command must be executed as a privileged user if you aren't currently using a virtualenv.

# Using the development version

# With pip

The Celery development version also requires the development versions of https://pypi.org/project/kombu/, https://pypi.org/project/amqp/, https://pypi.org/project/billiard/, and https://pypi.org/project/vine/.

You can install the latest snapshot of these using the following pip commands:

pip install https://github.com/celery/celery/zipball/main#egg=celery
pip install https://github.com/celery/billiard/zipball/main#egg=billiard
pip install https://github.com/celery/py-amqp/zipball/main#egg=amqp
pip install https://github.com/celery/kombu/zipball/main#egg=kombu
pip install https://github.com/celery/vine/zipball/main#egg=vine

1
2
3
4
5

# With git

Please see the Contributing section.

# Backends and Brokers

Celery supports serveral message transport alternatives.

# Broker Overview

This is comparison table of the different transports supports, more information can be found in the documentation for each individual transport(see Broker Instructions).

Name	Status	Monitoring	Remote Control
RabbitMQ	Stable	Yes	Yes
Redis	Stable	Yes	Yes
Amazon SQS	Stable	No	No
Zookeeper	Experimental	No	No
Kafka	Experimental	No	No
GC PubSub	Experimental	Yes	Yes

Experimental brokers may be functional but they don't have dedicated maintainers.

Missing monitor support means that the transport doesn't implement events, and as such Flower, celery events, celerymon and other event-based monitoring tools won't work.

Remote control means the ability to inspect and manage workers at runtime using the celery inspect and celery control commands (and other tools using the remote control API).

# Summaries

Note: This section is not comprehensive of backends and brokers.

Celery has the ability to communicate and store with many different backends (Result Stores) and brokers (Message Transports).

# Redis

Redis can be both a backend and a broker.

As a Broker: Redis works well for rapid transport of small messages. Large messages can congest the system.

See documentation for details

As a Backend: Redis is a super fast K/V store, making it very efficient for fetching the results of a task call. As with the design of Redis, you do have to consider the limit memory available to store your data, and how you handle data persistence. If result persistence is important, consider using another DB for your backend.

# RabbitMQ

RabbitMQ is a broker.

As a Broker: RabbitMQ handles larger messages better than Redis, however if many messages are coming in very quickly, scaling can become a concern and Redis or SQS should be considered unless RabbitMQ is running at very large scale.

See documentation for details

As a Backend: RabbitMQ can store results via rpc:// backend. This backend creates separate temporary queue for each client.

Note: RabbitMQ (as the borker) and Redis (as the backend) are very commonly used together. If more guaranteed long-term persistence is needed from the result store, consider using PostgreSQL or MySQL (through SQLAlchemy), Cassandra, or a custom defined backend.

# SQS

SQS is a broker.

If you already integrate tightly with AWS, and are familiar with SQS, it presents a great option as a broker. It is extremely scalable and completely managed, and manages task delegation similarly to RabbitMQ. It does lack some of the features of the RabbitMQ broker such as worker remote control commands.

See documentation for details

# SQLAlchemy

SQLAlchemy is a backend.

It allows Celery to interface with MySQL, PostgreSQL, SQlite, and more. It is an ORM, and is the way Celery can use a SQL DB as a result backend.

See documentation for details

# GCPubSub

Google Cloud Pub/Sub is a broker.

If you already integrate tightly with Google Cloud, and are familiar with Pub/Sub, it presents a great option as a broker. It is extremely scalable and completely managed, and manages task delegation similarly to RabbitMQ.

See documentation for details

# Using RabbitMQ

# Installation & Configuration

RabbitMQ is the default broker so it doesn't require any additional dependencies or initial configuration, other than the URL location of the broker instance you want to use:

broker_url = 'amqp://myuser:mypassword@localhost:5672/myvhost'

For a description of broker URLs and a full list of the various broker configuration options available to Celery, see Broker Settings, and see below for settings up the username, password and vhost.

# Installing the RabbitMQ Server

See Downloading and Installing RabbitMQ over at RabbitMQ's website. For macOS see Installing RabbitMQ on macOS.

If you're getting nodedown errors after installing and using rabbitmqctl then this blog post can help you identify the source of the problem: http://www.somic.org/2009/02/19/on-rabbitmqctl-and-badrpcnodedown/.

# Setting up RabbitMQ

To use Celery we need to create a RabbitMQ user, a virtual host and allow that user access to that virtual host:

sudo rabbitmqctl add_user myuser mypassword

sudo rabbitmqctl add_vhost myvhost

sudo rabbitmqctl set_user_tags myuser mytag

sudo rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*"

Substitute in appropriate values for myuser, mypassword and myvhost above. See the RabbitMQ Admin Guide for more information about access control.

# Installing RabbitMQ on macOS

The easiest way to install RabbitMQ on macOS is using Homebrew the new and shiny package management system for macOS.

First, install Homebrew using the one-line command provided by the Homebrew documentation:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Finally, we can install RabbitMQ using brew:

brew install rabbitmq

After you've installed RabbitMQ with brew you need to add the following to your path to be able to start and stop the broker: add it to the start-up file for your shell(e.g., .bash_profile or .profile).

PATH=$PATH:/usr/local/sbin

# Configuring the system host name

If you're using a DHCP server that's giving you a random host name, you need to permanently configure the host name. This is because RabbitMQ uses the host name to communicate with nodes.

Use the scutil command to permanently set your host name:

sudo scutil --set HostName myhost.local

Then add that host name to /etc/hosts so it's possible to resolve it back into an IP address:

127.0.0.1       localhost myhost myhost.local

If you start the rabbitmq-server, your rabbit node should now be rabbit@myhost, as verified by rabbitmqctl:

sudo rabbitmqctl status
Status of node rabbit@myhost ...
[{running_applications,[{rabbit,"RabbitMQ","1.7.1"},
                    {mnesia,"MNESIA  CXC 138 12","4.4.12"},
                    {os_mon,"CPO  CXC 138 46","2.2.4"},
                    {sasl,"SASL  CXC 138 11","2.1.8"},
                    {stdlib,"ERTS  CXC 138 10","1.16.4"},
                    {kernel,"ERTS  CXC 138 10","2.13.4"}]},
{nodes,[rabbit@myhost]},
{running_nodes,[rabbit@myhost]}]
...done.

1
2
3
4
5
6
7
8
9
10
11

This is especially important if your DHCP server gives you a host name starting with an IP address, (e.g., 23.10.112.31.comcast.net). In this case RabbitMQ will try to use rabbit@23: an illegal host name.

# Starting/Stopping the RabbitMQ server

To start the server:

sudo rabbitmq-server

you can also run it in the background by adding the -detached option (note: only one dash):

sudo rabbitmq-server -detached

Nerver use kill(kill(1)) to stop the RabbitMQ server, but rather use the rabbitmqctl command:

sudo rabbitmqctl stop

When the server is running, you can continue reading Setting up RabbitMQ.

# Using Quorum Queues

Added in version 5.5.

Quorum Queues require disabling global QoS which means some features won't work as expected. See limitations for details.

Celery supports Quorum Queue by setting the x-queue-type header to quorum like so:

from kombu import Queue

task_queues = [Queue('my-queue', queue_arguments={'x-queue-type': 'quorum'})]
broker_transport_options = {"confirm_publish": True}

1
2
3
4

If you'd like to change the type of the default queue, set the task_default_queue_type setting to quorum.

Another way to configure Quorum Queues is by relying on default settings and using task_routes:

task_default_queue_type = "quorum"
task_default_exchange_type = "topic"
task_default_queue = "my-queue"
broker_transport_options = {"confirm_publish": True}

task_routes = {
    "*": {
        "routing_key": "my-queue",
    },
}

1
2
3
4
5
6
7
8
9
10

Celery automatically detects if quorum queues are used using the worker_detect_quorum_queues. We recommend to keep the default behavior turned on.

To migrate from classic mirrored queues to quorum queues, please refer to RabbitMQ's documentation on the subject.

# Limitations

Disabling global QoS means that the per-channel QoS is now static. This means that some Celery features won't work when using Quorum Queues.

Autoscaling relies on increasing and decreasing the prefetch count whenever a new process is instantiated or terminated so it won't work when Quorum Queues are detected.

Similarly, the worker_enable_prefetch_count_reduction setting will be a no-op even when set to True when Quorum Queues are detected.

In addition, ETA/Countdown will block the worker when received until the ETA arrives since we can no longer increase prefetch count and fetch another task from the queue.

In order to properly schedule ETA/Countdown tasks we automatically detect if quorum queues are used and in case they are, Celery automatically enables Native Delayed Delivery.

# Native Delayed Delivery

Since tasks with ETA/Countdown will block the worker until they are scheduled for execution, we need to use RabbitMQ's native capabilities to schedule the execution of tasks.

The design is borrowed from NServiceBus. If you are interested in the implementation details, refer to their documentation.

Native Delayed Delivery is automatically enabled when quorum queues are detected.

By default the Native Delayed Delivery queues are quorum queues. If you'd like to change them to classic queues you can set the broker_native_delayed_delivery_queue_type to classic.

# Using Redis

# Installation

For the Redis support you have to install additional dependencies. You can install both Celery and these dependencies in one go using the celery[redis] boundle:

pip install -U "celery[redis]"

# Configuration

Configuration is easy, just configure the location of your Redis database:

app.conf.broker_url = 'redis://localhost:6379/0'

Where the URL is in the format of:

redis://:password@hostname:port/db_number

all fields after the scheme are optional, and will default to localhost on port 6379, using database 0.

If a Unix socket connection should be used, the URL needs to be in the format:

redis+socket:///path/to/redis.sock

Sepcifying a different database number when using a Unix socket is possible by adding the virtual_host parameter to the URL:

redis+socket:///path/to/redis.sock?virtual_host=db_number

It is also easy to connect directly to a list of Redis Sentinel:

app.conf.broker_url = 'sentinel://localhost:26379;sentinel://localhost:26380;sentinel://localhost:26381'
app.conf.broker_transport_options = { 'master_name': "cluster1" }

1
2

Additional options can be passed to the Sentinel client using sentinel_kwargs:

app.conf.broker_transport_options = { 'sentinel_kwargs': { 'password': "password" } }

# Visibility Timeout

The visibility timeout defines the number of seconds to wait for the worker to acknowledge the task before the message is redelivered to another worker. Be sure to see Caveats below.

This option is set via the broker_transport_options settings:

app.conf.broker_transport_options = {'visibility_timeout': 3600}  # 1 hour.

The default visibility timeout for Redis is 1 hour.

# Results

If you also want to store the state and return values of tasks in Redis, you should configure these settings:

app.conf.result_backend = 'redis://localhost:6379/0'

For a complete list of options supported by the Redis result backend, see Redis backend settings.

If you are using Sentinel, you should specify the master_name using the result_backend_transport_options setting:

app.conf.result_backend_transport_options = {'master_name': "mymaster"}

# Global keyprefix

The global key prefix will be prepended to all keys used for the result backend, which can be useful when a redis database is shared by different users. By default, no prefix is prepended.

To configure the global keyprefix for the Redis result backend, use the global_keyprefix key under result_backend_transport_options:

app.conf.result_backend_transport_options = {
    'global_keyprefix': 'my_prefix_'
}

1
2
3

# Connection timeouts

To configure the connection timeouts for the Redis result backend, use the retry_policy key under result_backend_transport_options:

app.conf.result_backend_transport_options = {
    'retry_policy': {
       'timeout': 5.0
    }
}

1
2
3
4
5

See retry_over_time() for the possible retry policy options.

# Serverless

Celery supports utilizing a remote serverless Redis, which can significantly reduce the operational overhead and cost, making it a vavorable choice in microservice architectures or environments where minimizing operational expenses is crucial. Serverless Redis provides the necessary functionalities without the need for manual setup, configuration, and management, thus aligning well with the principles of automation and scalability that Celery promotes.

# Upstash

Upstash offers a serverless Redis database service, providing a seamless solution for Celery users looking to leverage serverless architectures. Upstash's serverless Redis service is designed with an eventual consistency model and durable storage, facilitated through a multi-tier storage architecture.

Integration with Celery is straightforward as demonstrated in an example provided by Upstash.

# Dragonfly

Dragonfly is a drop-in Redis replacement that cuts costs and boosts performance. Designed to fully utilize the power of modern cloud hardware and deliver on the data demands of modern applications, Dragonfly frees developers from the limits of traditional in-memory data stores.

# Caveats

# Visibility timeout

If a task isn't acknowledged within the Visibility Timeout the task will be redelivered to another worker and executed.

This causes problems with ETA/countdown/retry tasks where the time to execute exceeds the visiblility timeout; in fact if that happens it will be executed again, and again in a loop.

To remediate that, you can increase the visibility timeout to match the time of the longest ETA you're planning to use. However, this is not recommended as it may have negative impact on the reliability. Celery will redeliver messages at worker shutdown, so having a long visibility timeout will only delay the redelivery of "lost" tasks in the event of a power failure or forcefully terminated workers.

Broker is not a database, so if you need of scheduling tasks for a more distant future, database-backed periodic task might be a better choice. Periodic tasks won't be affected by the visibility timeout, as this is a concept separate from ETA/countdown.

You can increase this timeout by configuring all of the following options with the same name (required to set all of them):

app.conf.broker_transport_options = {'visibility_timeout': 43200}
app.conf.result_backend_transport_options = {'visibility_timeout': 43200}
app.conf.visibility_timeout = 43200

1
2
3

The value must be an int describing the number of seconds.

Note: If multiple applications are sharing the same Broker, with different settings, the shortest value will be used. This include if the value is not set, and the default is sent.

# Soft Shutdown

During shutdown, the worker will attempt to re-queue any unacknowledged messages with task_acks_late enabled. However, if the worker is terminated forcefully (cold shutdown), the worker might not be able to re-queue the tasks on time, and they will not be consumed again until the Visibility Timeout has passed. This creates a problem when the Visibility Timeout is very high and a worker needs to shut down just after it has received a task. If the task is not re-queued in such case, it will need to wait for the long visibility timeout to pass before it can be consumed again, leading to potentially very long delays in tasks execution.

The soft shutdown introduces a time-limited warm shutdown phase just before the cold shutdown. This tiem window significantly increases the chances of re-queuing the tasks during shutdown which mitigates the problem of long visibility timeouts.

To enable the soft shutdown, set the worker_soft_shutdown_tiemout to a value greater than 0. The value must be an float describing the number of seconds. During this time, the worker will continue to process the running tasks until the timeout expires, after whichi the cold shutdown will be initiated automatically to terminated the worker gracefully.

If the REMAP_SIGTERM is configured to SIGQUIT in the environment variables, and the worker_soft_shutdown_tiemout is set, the worker will initiate the soft shutdown when it receives the TERM signal (and the QUIT signal).

# Key eviction

Redis may evict keys from the database in some situations. If you experience an error like:

InconsistencyError: Probably the key ('_kombu.binding.celery') has been
removed from the Redis database.

1
2

then you may want to configure the redis-server to not evict keys by setting in the redis configuration file:

the maxmemory option
the maxmemory-policy opton to noeviction or allkeys-lru

See Redis server documentation about Eviction Policies for details: https://redis.io/topics/lru-cache.

# Group result ordering

Versions of Celery up to and including 4.4.6 used an unsorted list to store result objects for groups in the Redis backend. This can cause those results to be returned in a different order to their associated tasks in the original group instantiation. Celery 4.4.7 introduced an opt-in behaviour which fixes this issue and ensures that group results are returned in the same order the tasks were defined, matching the behaviour of other backends. In Celery 5.0 this behaviour was changed to be opt-out. The behaviour is controlled by the result_chord_ordered configuration option which may be set like so:

# Specifying this for workers running Celery 4.4.6 or earlier has no effect
app.conf.result_backend_transport_options = {
    'result_chord_ordered': True    # or False
}

1
2
3
4

This is an incompatible change in the runtime behaviour of workers sharing the same Redis backend for result storage, so all workers must follow either the new or old behaviour to avoid breakage. For clusters with some workers running Celery 4.4.6 or earlier, this means that workers running 4.4.7 need no special configuration and workers running 5.0 or later must have result_chord_ordered set to False. For clusters with no workers running 4.4.6 or earlier but some workers running 4.4.7, it is recommended that result_chord_ordered be set to True for all workers to ease future migration. Migration between behaviours will disrupt results currently held in the Redis backend and cause breakage if downstream tasks are run by migrated workers - plan accordingly.

# Using Amazon SQS

wait......

# Using Kafka

wait......

# Using Google Pub/Sub

wait......

# First Steps with Celery

Celery is a task queue with batteries included. It's easy to use so that you can get started without learning the full complexities of the problem it solves. It's designed around best practices so that your product can scale and integrate with other languages, and it comes with the tools and support you need to run such a system in production.

In this tutorial you'll learn the absolute basics of using Celery.

Learn about:

Choosing and installing a message transport(borker).
Installing Celery and creating your first task.
Starting the worker and calling tasks.
Keeping track of tasks as they transition through different states, and inspecting return values.

Celery may seem daunting at first - but don't worry - this tutorial will get you started in no time. It's deliberately kept simple, so as to not confuse you with advanced features. After you have finished this tutorial, it's good idea to browse the rest of the documentation. For example the Next Steps tutorial will showcase Celery's capabilities.

# Choosing a Broker

Celery requires a solution to send and receive messages; usually this comes in the form of a separate service called a message broker.

There are several choices available, including:

# RabbitMQ

RabbitMQ is reature-complete, stable, durable and easy to install. It's an excellent choice for a production environment. Detailed information about using RabbitMQ with Celery: Using RabbitMQ

If you're using Ubuntu or Debian install RabbitMQ by executing this command:

sudo apt-get install rabbitmq-server

Or, if you want to run it on Docker execute this:

docker run -d -p 5672:5672 rabbitmq

Don't worry if you're not running Ubuntu or Debian, you can go to this website to find similarly simple installation instructions for other platforms, including Microsoft Windows: http://www.rabbitmq.com/download.html.

# Redis

Redis is also feature-complete, but is more susceptible to data loss in the event of abrupt termination or power failures. Detailed information about using Redis: Using Redis.

If you want to run it on Docker execute this:

docker run -d -p 6379:6379 redis

# Other brokers

In addition to the above, there are other experimental transport implementations to choose from, including Amazon SQS.

See Broker Overview for a full list.

# Installing Celery

Celery is on the Python Package Index(PyPi), so it can be installed with standard Python tools like pip:

pip install celery

# Application

The first thing you need is a Celery instance. We call this the Celery application or just app for short. As this instance is used as the entry-point for everything you want to do in Celery, like creating tasks and managing workers, it must be possible for other modules to import it.

In this tutorial we keep everything contained in a single module, but for larger projects you wang to create a dedicated module.

Let's create the file tasks.py:

from celery import Celery

app = Celery('tasks', broker='pyamqp://guest@localhost//')

@app.task
def add(x, y):
    return x + y

1
2
3
4
5
6
7

The first argument to Celery is the name of the current module. This is only needed so that names can be automatically generated when the tasks are defined in the __main__ module.

The second argument is the broker keyword argument, specifying the URL of the message broker you want to use. Here we are using RabbitMQ (also the default option).

See Choosing a Broker above for more choices - for RabbitMQ you can use amqp://localhost, or for Redis you can use redis://localhost.

You defined a single task, called add, returning the sum of two numbers.

# Running the Celery worker server

You can now run the worker by executing our program with the worker argument:

celery -A tasks worker --loglevel=INFO

See the Troubleshooting section if the worker doesn't start.

In production you'll want to run the worker in the background as a daemon. To do this you need to use the tools provided by your platform, or something like supervisord(see Daemonization for more information).

For a complete listing of the command-line options available, do:

celery worker --help

There are also several other commands available, and help is also available:

celery --help

# Calling the task

To call our task you can use the delay() method.

This is a handy shortcut to the apply_async() method that gives greater control of the task execution (see Calling Tasks):

from tasks import add
add.delay(4, 4)

1
2

The task has now been processed by the worker you started earlier. You can verify this by looking at the worker's console output.

Calling a task returns an AsyncResult instance. This can be used to check the state of the task, wait for the task to finish, or get its return value (or if the task failed, to get the exception and traceback).

Results are not enabled by default. In order to do remote procedure calls or keep track of task results in a database, you will need to configure Celery to use a result backend. This is described in the next section.

# Keeping Results

If you want to keep track of the task's states, Celery needs to store or send the states somewhere. There are several built-in result backends to choose from: SQLAlchemy/Django ORM, MongoDB, Memcached, Redis, RPC(RabbitMQ/AMQP), and -- or you can define your own.

For this example we use the rpc result backend, that sends states back as transient messages. The backend is specified via the backend argument to Celery, (or via the result_backend setting if you choose to use a configuration module). So, you can modify this line in the tasks.py file to enable the rpc://backend:

app = Celery('tasks', backend='rpc://', broker='pyamqp://')

Or if you want to use Redis as the result backend, but still use RabbitMQ as the message broker (a popular combination):

app = Celery('tasks', backend='redis://localhost', broker='pyamqp://')

To read more about result backends please see Result Backends.

Now with the result backend configured, restart the worker, close the current python session and import the tasks module again to put the changes into effect. This time you'll hold on to the AsyncResult instance returned when you call a task:

from tasks import add    # close and reopen to get updated 'app'
result = add.delay(4, 4)

1
2

The ready() method returns whether the task has finished processing or not:

result.ready()
# False

1
2

You can wait for the result to complete, but this is rarely used since it turns the asynchronous call into a synchronous one:

result.get(timeout=1)
# 8

1
2

In case the task raised an exception, get() will re-raise the exception, but you can override this by specifying the propagate argument:

result.get(propagate=False)

If the task raised an exception, you can also gain access to the original traceback:

result.traceback

Backends use resources to store and transmit results. To ensure that resources are released, you must eventually call get() or forget() on EVERY AsyncResult instance returned after calling a task.

See celery.result for the complete result object reference.

# Configuration

Celery, like a consumer appliance, doesn't need much configuration to operate. It has an input and an output. The input must be connected to a broker, and the output can be optionally connected to a result backend. However, if you look closely at the back, there's a lid revealing loads of sliders, dials, and buttons: this is the configuration.

The default configuration should be good enough for most use cases, but there are many options that can be configured to make Celery work exactly as needed. Reading about the options available is a good idea to familiarize yourself with what can be configured. You can read about the options in the Configuration and defaults reference.

The configuration can be set on the app directly or by using a dedicated configuration module. As an example you can configure the default serializer used for serializing task pyloads by changing the task_serializer setting:

app.conf.task_serializer = 'json'

If you're configuring many settings at once you can use update:

app.conf.update(
    task_serializer='json',
    accept_content=['json'],  # Ignore other content
    result_serializer='json',
    timezone='Europe/Oslo',
    enable_utc=True,
)

1
2
3
4
5
6
7

For larger projects, a dedicated configuration module is recommended. Hard coding periodic task intervals and task routing options is discouraged. It is much better to keep these in a centralized location. This is especially true for libraries, as it enables users to control how their tasks behave. A centralized configuration will also allow your SysAdmin to make simple changes in the event of system trouble.

You can tell your Celery instance to use a configuration module by calling the app.config_from_object() method:

app.config_from_object('celeryconfig')

This module is often called "celeryconfig", but you can use any module name.

In the above case, a module named celeryconfig.py must be available to load from the current directory or on the Python path. It could look something like this:

# celeryconfig.py
broker_url = 'pyamqp://'
result_backend = 'rpc://'

task_serializer = 'json'
result_serializer = 'json'
accept_content = ['json']
timezone = 'Europe/Oslo'
enable_utc = True

1
2
3
4
5
6
7
8
9

To verify that your configuration file works properly and doesn't contain any syntax errors, you can try to import it:

python -m celeryconfig

For a complete reference of configuration options, see Configuration and defaults.

To demonstrate the power of configuration files, this how you'd route a misbehaving task to a dedicated queue:

# celeryconfig.py
task_routes = {
    'tasks.add': 'low-priority',
}

1
2
3
4

Or instead of routing it you could rate limit the task instead, so that only 10 tasks of this type can be processed in a minute(10/m):

# celeryconfig.py
task_annotations = {
    'tasks.add': {'rate_limit': '10/m'}
}

1
2
3
4

If you're using RabbitMQ or Redis as the broker then you can also direct the workers to set a new rate limit for the task at runtime:

celery -A tasks control rate_limit tasks.add 10/m
worker@example.com: OK
    new rate limit set successfully

1
2
3

See Routing Tasks to read more about task routing, and the task_annotations setting for more about annotations, or Monitoring and Management Guide for more about remote control commands and how to monitor what your workers are doing.

# Where to go from here

If you want to learn more you should continue to the Next Steps tutorial, and after that you can read the User Guide.

# Troubleshooting

There's also a troubleshooting section in the Frequently Asked Questions.

# Worker doesn't start: Permission Error

If you're using Debian, Ubuntu or other Debian-based distributions:

Debian recently renamed the /dev/shm special file to /run/shm. A simple workaround is to create a symbolic link:

ln -s /run/shm /dev/shm

Others:

If you provide any of the --pidfile, --logfile or --statedb arguments, then you must make sure that they point to a file or directory that's writable and readable by the user starting the worker.

# Result backend doesn't work or tasks are always in PENDING state

All tasks are PENDING by default, so the state would've been better named "unkowner". Celry doesn't update the state when a task is sent, and any task with no history is assumed to be pending(you know the task id, after all).

Make sure that the task doesn't have ignore_result enabled. Enabling this option will force the worker to skip updating states.
Make sure the task_ignore_result setting isn't enabled.
Make sure that you don't have any old workers still running. It's easy to start multiple workers by accident, so make sure that the previous worker is properly shut down before you start a new one. An old worker that isn't configured with the expected result backend may be running and is hijacking the tasks. The --pidfile argument can be set to an absolute path to make sure this doesn't happen.
Make sure the client is configured with the right backend. If, for some reason, the client is configured to use a different backend than the worker, you won't be able to receive the result. Make sure the backend is configured correctly:

result = task.delay()
print(result.backend)

1
2

# Next Steps

The First Steps with Celery guide is intentinally minimal. In this guide i'll demonstrate what Celery offers in more detail, including how to add Celery support for your application and library.

This document doesn't document all of Celery's features and best practies, so it's recommended that you also read the User Guide.

# Using Celery in your Application

# Our Project

Project layout:

src/
    proj/__init__.py
        /celery.py
        /tasks.py

1
2
3
4

proj/celery.py:

from celery import Celery

app = Celery('proj',
             broker='amqp://',
             backend='rpc://',
             include=['proj.tasks'])

# Optional configuration, see the application user guide.
app.conf.update(
    result_expires=3600,
)

if __name__ == '__main__':
    app.start()

1
2
3
4
5
6
7
8
9
10
11
12
13
14

In this module you created our Celery instance (sometimes refered to as the app). To use Celery within your project you simply import this instance.

The broker argument specifies the URL of the broker to use. See Choosing a Broker for more information.
The backend argument specifies the result backend to use. It's used to keep track of task state and results. While results are disabled by default I use the RPC result backend here because i demonstrate how retrieving results worker later. You may want to use a different backend for your application. They all have different strengths and weaknesses. If you don't need results, it's better to disable them. Results can also be disabled for individual tasks by setting the @task(ignore_result=True) option. See Keeping Results for more information.

proj/tasks.py:

from .celery import app


@app.task
def add(x, y):
    return x + y

@app.task
def mul(x, y):
    return x * y

@app.task
def xsum(numbers):
    return sum(numbers)

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# Starting the worker

The celery program can be used to start the worker (you need to run the worker in the directory above proj, according to the example project layout the directory is src):

celery -A proj worker -l INFO

When the worker starts you should see a banner and some messages:

--------------- celery@halcyon.local v4.0 (latentcall)
--- ***** -----
-- ******* ---- [Configuration]
- *** --- * --- . broker:      amqp://guest@localhost:5672//
- ** ---------- . app:         __main__:0x1012d8590
- ** ---------- . concurrency: 8 (processes)
- ** ---------- . events:      OFF (enable -E to monitor this worker)
- ** ----------
- *** --- * --- [Queues]
-- ******* ---- . celery:      exchange:celery(direct) binding:celery
--- ***** -----

[2012-06-08 16:23:51,078: WARNING/MainProcess] celery@halcyon.local has started.

1
2
3
4
5
6
7
8
9
10
11
12
13

The broker is the URL you specified in the broker argument in our celery module. You can also specify a different broker on the command-line by using the -b option.

Concurrency is the number of prefork worker process used to process your tasks concurrently. When all of these are busy doing work, new tasks will have to wait for one of the tasks to finish before it can be processed.

The default concurrency number is the number of CPU's on that machine(including cores). You can specify a custom number using the celery worker -c option. There's no recommended value, as the optimal number depends on a number of factors, but if your tasks are mostly I/O-bound then you can try to increase it. Experimentation has shown that adding more than twice the number of CPU's is rarely effective, and likely to degrade performance instead.

Including the default perfork pool, Celery also supports using Eventlet, Gevent, and runnng in a single thread (see Concurrency).

Events is an option that causes Celery to send monitoring messages (events) for actions occurring in the worker. These can be used by monitor programs like celery events, and Flower -- the real-time Celery monitor, which you can read about in the Monitoring and Management guide.

Queues is the list of queues that the worker will consume tasks from. The worker can be told to consume from several queues at once, and this is used to route messages to specific workers as a means for Quality of Service, separation of concerns, and prioritization, all described in th Routing Guide.

You can get a complete list of command-line arguments by passing in the --help flag:

celery worker --help

These options are described in more detailed in the Workers Guide.

# Stopping the worker

To stop the worker simply hit Control - C. A list of signals supported by the worker is detailed in the Workers Guide.

# In the background

In production you'll want to run the worker in the background, described in detail in the daemonization tutorial.

The daemonization scripts uses the celery multi command to start one or more workers in the background:

celery multi start w1 -A proj -l INFO
celery multi v4.0.0 (latentcall)
> Starting nodes...
    > w1.halcyon.local: OK

1
2
3
4

You can restart it too:

celery  multi restart w1 -A proj -l INFO
celery multi v4.0.0 (latentcall)
> Stopping nodes...
    > w1.halcyon.local: TERM -> 64024
> Waiting for 1 node.....
    > w1.halcyon.local: OK
> Restarting node w1.halcyon.local: OK
celery multi v4.0.0 (latentcall)
> Stopping nodes...
    > w1.halcyon.local: TERM -> 64052

1
2
3
4
5
6
7
8
9
10

or stop it:

celery multi stop w1 -A proj -l INFO

The stop command is asynchronous so it won't wait for the worker to shutdown. You'll probably want to use the stopwait command instead, which ensures that all currently executing tasks are completed before exiting:

celery multi stopwait w1 -A proj -l INFO

celery multi doesn't store information about workers so you need to use the same command-line arguments when restarting. Only the same pidfile and logfile arguments must be used when stopping.

By default it'll create pid and log files in the current directory. To protect against multiple workers launching on top of each other you're encouraged to put these in a dedicated directory:

mkdir -p /var/run/celery
mkdir -p /var/log/celery
celery multi start w1 -A proj -l INFO --pidfile=/var/run/celery/%n.pid \
                                        --logfile=/var/log/celery/%n%I.log

1
2
3
4

With the multi command you can start multiple workers, and there's a powerful command-line syntax to specify arguments for different workers too, for example:

celery multi start 10 -A proj -l INFO -Q:1-3 images,video -Q:4,5 data \
    -Q default -L:4,5 debug

1
2

For more examples see the multi module in the API reference.

# About the `--app` argument

The --app argument specifies the Celery app instance to use, in the form of module.path:attribute

But it also supports a shortcut form. If only a package name is specified, it'll try to search for the app instance, in the following order:

With -app=proj:

an attribute named proj.app, or
an attribute named proj.celery, or
any attribute in the module proj where the value is a Celery application, or

If none of these are found it'll try a submodule named proj.celery: 4. an attribute named proj.celery.app, or 5. an attribute named proj.celery.celery, or 6. Any attribute in the module proj.celery where the value is a Celery application.

This scheme mimics the practices used in the documentation -- that is, proj:app for a single contained module, and proj.celery:app for larger projects.

# Calling Tasks

You can call a task using the delay() method:

from proj.tasks import add

add.delay(2, 2)

1
2
3

This method is actually a star-argument shortcut to another method called apply_async():

add.apply_async((2, 2))

The latter enables you to specify execution options like the time to run (countdown), the queue it should be sent to, and so on:

add.apply_async((2, 2), queue='lopri', countdown=10)

In the above example the task will be sent to a queue named lopri and the task will execute, at the earliest, 10 seconds after the message was sent.

Applying the task directly will execute the task in the current process, so that no message is sent:

add(2, 2)
4

1
2

These three methods - delay(), apply_async(), and applying (__call__), make up the Celery calling API, which is also used for signatures.

A more detailed overview of the Calling API can be found in the Calling User Guide.

Every task invocation will be given a unique identifier (an UUID)--this is the task id.

The delay and apply_async methods return an AsyncResult instance, which can be used to keep track of the tasks execution state. But for this you need to enable a result backend so that the state can be stored somewhere.

Results are diabled by default because there is no result backend that suits every application; to choose one you need to consider the drawbacks of each individual backend. For many tasks keeping the return value isn't even very useful, so it's sensible default to have. Also note that result backends aren't used for monitoring tasks and workers: for that Celery use dedicated event messages(see Monitoring and Management Guide).

If you have a result backend configured you can retrieve the return value of a task:

res = add.delay(2, 2)
res.get(timeout=1)
4

1
2
3

You can find the task's id by looking at the id attribute:

res.id
d6b3aea2-fb9b-4ebc-8da4-848818db9114

1
2

You can also inspect the exception and traceback if the task raised an exception, in fact result.get() will propagate any errors by default:

res = add.delay(2, '2')
res.get(timeout=1)

1
2

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "celery/result.py", line 221, in get
    return self.backend.wait_for_pending(
  File "celery/backends/asynchronous.py", line 195, in wait_for_pending
    return result.maybe_throw(callback=callback, propagate=propagate)
  File "celery/result.py", line 333, in maybe_throw
    self.throw(value, self._to_remote_traceback(tb))
  File "celery/result.py", line 326, in throw
    self.on_ready.throw(*args, **kwargs)
  File "vine/promises.py", line 244, in throw
    reraise(type(exc), exc, tb)
  File "vine/five.py", line 195, in reraise
    raise value
TypeError: unsupported operand type(s) for +: 'int' and 'str'

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

If you don't wish for the errors to propagate, you can disable that by passing propagate:

res.get(propagate=False)
TypeError("unsupported operand type(s) for +: 'int' and 'str'")

1
2

In this case it'll return the exception instance raised instead -- so to check whether the task succeeded or failed, you'll have to use the corresponding methods on the result instance:

res.failed()
True
res.successful()
False

1
2
3
4

So how does it know if the task has failed or not? It can find out by looking at the tasks state:

res.state
'FAILURE'

1
2

A task can only be in a single state, but it can progress through serveral states. The stages of a typical task can be:

PENDING -> STARTED -> SUCCESS

The started state is a special state that's only recorded if the task_track_started setting is enabled, or if the @task(track_started=True) option is set for the task.

The pending state is actually not a recorded state, but rather the default state for any task id that's unknown: this you can see from this example:

from proj.celery import app

res = app.AsyncResult('this-id-does-not-exist')
res.state
'PENDING'

1
2
3
4
5

If the task is retried the stages can become even more complex. To demonstrate, for a task that's retried two times the stages would be:

PENDING -> STARTED -> RETRY -> STARTED -> RETRY -> STARTED -> SUCCESS

TO read more about task states you should see the States section in the tasks user guide.

Calling tasks is described in detail in the Calling Guide.

# Canvas: Designing Work-flows

You just learned how to call a task using the tasks delay method, and this is often all you need. But sometimes you may want to pass the signature of a task invocation to another process or as an argument to another function, for which Celery uses something called signatures.

A signature wraps the arguments and execution options of a single task invocation in such a way that it can be passed to function or even serialized and sent across the wire.

You can create a signature for the add task using the arguments (2, 2), and a countdown of 10 seconds like this:

add.signature((2, 2), countdown=10)
tasks.add(2, 2)

1
2

There's also a shortcut using star arguments:

add.s(2, 2)
tasks.add(2, 2)

1
2

# And there's that calling API again

Signature instances also support the calling API, meaning they have delay and apply_async methods.

But there's a difference in that the signature may already have an argument signature specified. The add task takes two arguments, so a signature specifying two arguments would make a complete signature:

s1 = add.s(2, 2)
res = s1.delay()
res.get()
4

1
2
3
4

But, you can also make incomplete signatures to create what we call partials:

# incomplete partial: add(?, 2)
s2 = add.s(2)

1
2

S2 is now a partial signature that needs another argument to be complete, and this can be resolved when calling the signature:

# resolves the partial: add(8, 2)
res = s2.delay(8)
res.get()
10

1
2
3
4

Here you added the argument 8 that was prepended to the existing argument 2 forming a complete signature of add(8, 2).

Keyword arguments can also be added later; these are then merged with any existing keyword arguments, but with new arguments taking precedence:

s3 = add.s(2, 2, debug=True)
s3.delay(debug=False) # debug is now False

1
2

As stated, signatures support the calling API: meaning that

sig.apply_async(args=(), kwargs={}, **options): Calls the signature with optional partial arguments and partial keyword arguments. Also supports partial execution options.
sig.delay(*args, **kwargs): Star argument version of apply_async. Any arguments will be prepended to the arguments in the signature, and keyword arguments is merged with any existing keys.

So this all seems very useful, but what can you actually do with these? To get to that i must introduce the canvas primitives....

# The Primitives

These primitives are signature objects themselves, so they can be combined in any number of ways to compose complex work-flows.

These examples retrieve results, so to try them out you need to configure a result backend. The example project above already does that (see the backend argument to Celery).

Let's look at some examples:

# Groups

A group calls a list of tasks in parallel, and it returns a special result instance that lets you inspect the results as a group, and retrieve the return values in order.

from celery import group
from proj.tasks import add

group(add.s(i, i) for i in range(10))().get()
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

1
2
3
4
5

Partial group

g = group(add.s(i) for i in range(10))
g(10).get()
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19]

1
2
3

# Chains

Tasks can be linked together so that after one task returns the other is called:

from celery import chain
from proj.tasks import add, mul

# (4 + 4) * 8
chain(add.s(4, 4) | mul.s(8))().get()
64

1
2
3
4
5
6

or a partial chain:

# (? + 4) * 8
g = chain(add.s(4) | mul.s(8))
g(4).get()
64

1
2
3
4

Chains can also be written like this:

(add.s(4, 4) | mul.s(8))().get()
64

1
2

# Chords

A chord is a group with a callback:

from celery import chord
from proj.tasks import add, xsum

chord((add.s(i, i) for i in range(10)), xsum.s())().get()
90

1
2
3
4
5

A group chained to another task will be automatically converted to a chord:

(group(add.s(i, i) for i in range(10)) | xsum.s())().get()
90

1
2

Since these primitives are all of the signature type they can be combined almost however you want, for example:

upload_document.s(file) | group(apply_filter.s() for filter in filters)

Be sure to read more about work-flows in the Canvas user guide.

# Routing

Celery supports all of the routing facilities provided by AMQP, but it also supports simple routing where messages are sent to named queues.

The Task_routes setting enables you to route tasks by name and keep everything centralized in one location:

app.conf.update(
    task_routes = {
        'proj.tasks.add': {'queue': 'hipri'},
    }
)

1
2
3
4
5

You can also specify the queue at runtime with the queue argument to apply_async:

from proj.tasks import add
add.apply_async((2, 2), queue='hipri')

1
2

You can then make a worker consume from this queue by specifying the celery worker -Q:

celery -A proj worker -Q hipri

You may specify multiple queues by using a comma-separated list. For example, you can make the worker consume from both the default queue and the hipri queue, where the default queue is named celery for historical reasons:

celery -A proj worker -Q hipri,celery

The order of the queues doesn't matter as the worker will give equal weight to the queues.

To learn more about routing, including taking use of the full power of AMQP routing, see the Routing Guide.

# Remote Control

If you're using RabbitMQ(AMQP), Redis, or Qpid as the broker then you can control and inspect the worker at runtime.

For example you can see what tasks the worker is currently working on:

celery -A proj inspect active

This is implemented by using broadcast messaging, so all remote control commands are received by every worker in the cluster.

You can also specify one or more workers to act on the request using the --destination option. This is aa comma-separated list of worker host names:

celery -A proj inspect active --destination=celery@example.com

If a destination isn't provided then every worker will act and reply to the request.

The celery inspect command contains commands that don't change anything in the worker; it only returns information and statistics about what's going on inside the worker. For a list of inspect commands you can execute:

celery -A proj inspect --help

Then there's the celery control command, which contains commands that actually change things in the worker at runtime:

celery -A proj control --help

For example you can force workers to enable event messages(used for monitoring tasks and workers):

celery -A proj control enable_events

When events are enabled you can then start the event dumper to see what the workers are doing:

celery -A proj events --dump

or you can start the curses interface:

celery -A proj events

when you're finished monitoring you can disable events again:

celery -A proj control disable_events

The celery status command also uses remote control commands and shows a list of online workers in the cluster:

celery -A proj status

You can read more about the celery command and monitoring in the Monitoring Guide.

# Timezone

All times and dates, internally and in messages use the UTC timezone.

When the worker receives a message, for example with a countdown set it converts that UTC time to local time. If you wish to use a different timezone that the system timezone then you must configure that using the timezone setting:

app.conf.timezone = 'Europe/London'

# Optimization

The default configuration isn't optimized for throughput. By default, it tries to walk the middle way between many short tasks and fewer long tasks, acompromise between throughput and fair scheduling.

If you have strict fair scheduling requirements, or want to optimize for throughput then you should read the Optimizing Guide.

# What to do now?

Now that you have read this document you should continue to the User Guide. There's also an API reference if you're so inclined.

蒲公英