README.rst 2.6 KB
Newer Older
Philippe Pepiot's avatar
Philippe Pepiot committed
1
2
Summary
-------
Philippe Pepiot's avatar
Philippe Pepiot committed
3
4
5
6

Run, monitor and log celery tasks.


7
8
Installation and setup
----------------------
Philippe Pepiot's avatar
Philippe Pepiot committed
9
10
11

Declare tasks using celery task or cubicweb-celery cwtasks.

12
On worker side, install cw-celerytask-helpers_.
Philippe Pepiot's avatar
Philippe Pepiot committed
13
14
15

celeryconfig.py example::

Philippe Pepiot's avatar
Philippe Pepiot committed
16
    CELERY_BROKER_URL = 'redis://localhost:6379/0'
Philippe Pepiot's avatar
Philippe Pepiot committed
17
    CELERY_RESULT_BACKEND = BROKER_URL
Philippe Pepiot's avatar
Philippe Pepiot committed
18
    CUBICWEB_CELERYTASK_REDIS_URL = CELERY_BROKER_URL
19
    CELERY_IMPORTS = ('cw_celerytask_helpers.helpers', 'module.containing.tasks')
Philippe Pepiot's avatar
Philippe Pepiot committed
20
21


David Douard's avatar
David Douard committed
22
23
24
25
26
27
28
In this configuration example, the ``cw_celerytask_helpers`` in
``CELERY_IMPORTS`` is required to have logging data (in the task) sent
back to the Cubicweb instance via Redis. The
``CUBICWEB_CELERYTASK_REDIS_URL`` is the Redis endpoint used for this
logging handling mechanism.

    
Philippe Pepiot's avatar
Philippe Pepiot committed
29
30
31
Start a worker::

    # running cubicweb tasks (celeryconfig.py will be imported from your instance config directory)
32
    celery -A cubicweb_celery -i <CW_INSTANCE_NAME> worker -l info
Philippe Pepiot's avatar
Philippe Pepiot committed
33
34

    # running pure celery tasks
35
    celery worker -l info
Philippe Pepiot's avatar
Philippe Pepiot committed
36
37


38
39
40
41
42
Task state synchronization requires to run the `celery-monitor` command::

    cubicweb-ctl celery-monitor <instance-name>


43
44
45
46
Ensure to have the celeryconfig.py loaded for both cubicweb instance and
celery worker, enforce by settings with CELERY_CONFIG_MODULE environment
variable (it must be an importable python module).

47
.. _cw-celerytask-helpers: https://www.cubicweb.org/project/cw-celerytask-helpers
48

49
50
51
Running tasks
-------------

Philippe Pepiot's avatar
Philippe Pepiot committed
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
Create a task:

.. code-block:: python

    from celery import current_app as app
    from celery.utils.log import get_task_logger

    logger = get_task_logger(__name__)

    @app.task(name='hi_there')
    def my_task(arg, kw=0):
        logger.info('HI %s %s!', arg, kw)
        return 42


Run a task:

.. code-block:: python

Philippe Pepiot's avatar
Philippe Pepiot committed
71
    from cubicweb_celerytask.entities import start_async_task
Philippe Pepiot's avatar
Philippe Pepiot committed
72
73

    cwtask = start_async_task(cnx, 'hi_there', 'THERE', kw=42)
74
    cnx.commit()
75
76
77
78
79


start_async_task() accept task names, task objects or task signatures:
http://docs.celeryproject.org/en/latest/userguide/canvas.html#signatures

80
81
82
For instance, to start the above task in a dedicated queue named `myqueue`:

.. code-block:: python
83

84
    import celery
85

86
87
    start_async_task(cnx, celery.signature('hi_there', args=('THERE',),
                                           kwargs={'kw': 42}, queue='myqueue'))
88
89
90
91
92
93


Testing task based application
------------------------------

In CubicWeb test mode, tasks don't run automatically, use
Philippe Pepiot's avatar
Philippe Pepiot committed
94
95
`cubicweb_celerytask.entities.get_tasks()` to introspect them and
`cubicweb_celerytask.entities.run_all_tasks()` to run them.
96
97
98

Also, CELERY_ALWAYS_EAGER and CELERY_EAGER_PROPAGATES_EXCEPTIONS are set to
True by default.