README 3.77 KB
Newer Older
Philippe Pepiot's avatar
Philippe Pepiot committed
1
2
Summary
-------
Philippe Pepiot's avatar
Philippe Pepiot committed
3
4
5
6

Run, monitor and log celery tasks.


7
8
Installation and setup
----------------------
Philippe Pepiot's avatar
Philippe Pepiot committed
9
10
11

Declare tasks using celery task or cubicweb-celery cwtasks.

12
On worker side, install cw-celerytask-helpers_.
Philippe Pepiot's avatar
Philippe Pepiot committed
13
14
15
16
17
18

celeryconfig.py example::

    BROKER_URL = 'redis://localhost:6379/0'
    CELERY_RESULT_BACKEND = BROKER_URL
    CUBICWEB_CELERYTASK_REDIS_URL = BROKER_URL
19
    CELERY_IMPORTS = ('cw_celerytask_helpers.redislogger', 'module.containing.tasks')
Philippe Pepiot's avatar
Philippe Pepiot committed
20
21
22
23
24
    CELERY_TASK_SERIALIZER = 'json'
    CELERY_RESULT_SERIALIZER = 'json'
    CELERY_ACCEPT_CONTENT = ['json', 'msgpack', 'yaml']


David Douard's avatar
David Douard committed
25
26
27
28
29
30
31
In this configuration example, the ``cw_celerytask_helpers`` in
``CELERY_IMPORTS`` is required to have logging data (in the task) sent
back to the Cubicweb instance via Redis. The
``CUBICWEB_CELERYTASK_REDIS_URL`` is the Redis endpoint used for this
logging handling mechanism.

    
Philippe Pepiot's avatar
Philippe Pepiot committed
32
33
34
35
36
37
38
39
40
Start a worker::

    # running cubicweb tasks (celeryconfig.py will be imported from your instance config directory)
    celery -A cubicweb_celery -i <CW_INSTANCE_NAME> worker -l info -E

    # running pure celery tasks
    celery worker -l info -E


41
42
43
44
45
Task state synchronization requires to run the `celery-monitor` command::

    cubicweb-ctl celery-monitor <instance-name>


46
47
48
49
Ensure to have the celeryconfig.py loaded for both cubicweb instance and
celery worker, enforce by settings with CELERY_CONFIG_MODULE environment
variable (it must be an importable python module).

50
.. _cw-celerytask-helpers: https://www.cubicweb.org/project/cw-celerytask-helpers
51

52
53
54
Running tasks
-------------

Philippe Pepiot's avatar
Philippe Pepiot committed
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
Create a task:

.. code-block:: python

    from celery import current_app as app
    from celery.utils.log import get_task_logger

    logger = get_task_logger(__name__)

    @app.task(name='hi_there')
    def my_task(arg, kw=0):
        logger.info('HI %s %s!', arg, kw)
        return 42


Run a task:

.. code-block:: python

    from cubes.celerytask.entities import start_async_task

    cwtask = start_async_task(cnx, 'hi_there', 'THERE', kw=42)
77
    cnx.commit()
78
79
80
81
82


start_async_task() accept task names, task objects or task signatures:
http://docs.celeryproject.org/en/latest/userguide/canvas.html#signatures

83
84
85
For instance, to start the above task in a dedicated queue named `myqueue`:

.. code-block:: python
86

87
    import celery
88

89
90
    start_async_task(cnx, celery.signature('hi_there', args=('THERE',),
                                           kwargs={'kw': 42}, queue='myqueue'))
91
92
93
94
95
96
97
98
99
100
101


Testing task based application
------------------------------

In CubicWeb test mode, tasks don't run automatically, use
`cubes.celerytask.entities.get_tasks()` to introspect them and
`cubes.celerytask.entities.run_all_tasks()` to run them.

Also, CELERY_ALWAYS_EAGER and CELERY_EAGER_PROPAGATES_EXCEPTIONS are set to
True by default.
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150


Demo
----

A simple demo is supplied with the source code.

We assume the present ``cubicweb-celerytask`` cube is properly
installed in a working Cubicweb environment, and there is a redis
server available on ``redis://localhost:6379/0`` (you can change this
in ``demo/celeryconfig.py`` is needed).

Then:

.. code-block:: bash

   user@host:~/celerytask$ cubicweb-ctl create celerytask demo

For the sake of simplicity, choose sqlite as database driver, and say
'yes' to the question "aAllow anonymous access ?"

Start the web application:

.. code-block:: bash

   user@host:~/celerytask$ cubicweb-ctl start -D -linfo demo

Open your web browser on http://127.0.0.1:8080/


In another terminal, start a celery worker:


.. code-block:: bash

   user@host:~$ cd celerytask/demo
   user@host:~/celerytask/demo$ celery worker -l info -E


In a third terminal, launch some tasks:

.. code-block:: bash

   user@host:~$ cd celerytask/demo
   user@host:~/celerytask/demo$ cubicweb-ctl shell demo launchtasks.py

You should be able to see 3 tasks on http://127.0.0.1:8080/CeleryTask