README 2.72 KB
Newer Older
Philippe Pepiot's avatar
Philippe Pepiot committed
1
2
Summary
-------
Philippe Pepiot's avatar
Philippe Pepiot committed
3
4
5
6

Run, monitor and log celery tasks.


7
8
Installation and setup
----------------------
Philippe Pepiot's avatar
Philippe Pepiot committed
9
10
11

Declare tasks using celery task or cubicweb-celery cwtasks.

12
On worker side, install cw-celerytask-helpers_.
Philippe Pepiot's avatar
Philippe Pepiot committed
13
14
15
16
17
18

celeryconfig.py example::

    BROKER_URL = 'redis://localhost:6379/0'
    CELERY_RESULT_BACKEND = BROKER_URL
    CUBICWEB_CELERYTASK_REDIS_URL = BROKER_URL
19
    CELERY_IMPORTS = ('cw_celerytask_helpers.helpers', 'module.containing.tasks')
Philippe Pepiot's avatar
Philippe Pepiot committed
20
21
22
23
24
    CELERY_TASK_SERIALIZER = 'json'
    CELERY_RESULT_SERIALIZER = 'json'
    CELERY_ACCEPT_CONTENT = ['json', 'msgpack', 'yaml']


David Douard's avatar
David Douard committed
25
26
27
28
29
30
31
In this configuration example, the ``cw_celerytask_helpers`` in
``CELERY_IMPORTS`` is required to have logging data (in the task) sent
back to the Cubicweb instance via Redis. The
``CUBICWEB_CELERYTASK_REDIS_URL`` is the Redis endpoint used for this
logging handling mechanism.

    
Philippe Pepiot's avatar
Philippe Pepiot committed
32
33
34
Start a worker::

    # running cubicweb tasks (celeryconfig.py will be imported from your instance config directory)
35
    celery -A cubicweb_celery -i <CW_INSTANCE_NAME> worker -l info
Philippe Pepiot's avatar
Philippe Pepiot committed
36
37

    # running pure celery tasks
38
    celery worker -l info
Philippe Pepiot's avatar
Philippe Pepiot committed
39
40


41
42
43
44
45
Task state synchronization requires to run the `celery-monitor` command::

    cubicweb-ctl celery-monitor <instance-name>


46
47
48
49
Ensure to have the celeryconfig.py loaded for both cubicweb instance and
celery worker, enforce by settings with CELERY_CONFIG_MODULE environment
variable (it must be an importable python module).

50
.. _cw-celerytask-helpers: https://www.cubicweb.org/project/cw-celerytask-helpers
51

52
53
54
Running tasks
-------------

Philippe Pepiot's avatar
Philippe Pepiot committed
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
Create a task:

.. code-block:: python

    from celery import current_app as app
    from celery.utils.log import get_task_logger

    logger = get_task_logger(__name__)

    @app.task(name='hi_there')
    def my_task(arg, kw=0):
        logger.info('HI %s %s!', arg, kw)
        return 42


Run a task:

.. code-block:: python

Philippe Pepiot's avatar
Philippe Pepiot committed
74
    from cubicweb_celerytask.entities import start_async_task
Philippe Pepiot's avatar
Philippe Pepiot committed
75
76

    cwtask = start_async_task(cnx, 'hi_there', 'THERE', kw=42)
77
    cnx.commit()
78
79
80
81
82


start_async_task() accept task names, task objects or task signatures:
http://docs.celeryproject.org/en/latest/userguide/canvas.html#signatures

83
84
85
For instance, to start the above task in a dedicated queue named `myqueue`:

.. code-block:: python
86

87
    import celery
88

89
90
    start_async_task(cnx, celery.signature('hi_there', args=('THERE',),
                                           kwargs={'kw': 42}, queue='myqueue'))
91
92
93
94
95
96


Testing task based application
------------------------------

In CubicWeb test mode, tasks don't run automatically, use
Philippe Pepiot's avatar
Philippe Pepiot committed
97
98
`cubicweb_celerytask.entities.get_tasks()` to introspect them and
`cubicweb_celerytask.entities.run_all_tasks()` to run them.
99
100
101

Also, CELERY_ALWAYS_EAGER and CELERY_EAGER_PROPAGATES_EXCEPTIONS are set to
True by default.