Skip to content
  • Philippe Pepiot's avatar
    monitor: can work with multiple instances · f094aff48e8d
    Philippe Pepiot authored
    Previously we handle monitoring celery task by listening to celery event bus
    (celery.events.EventReceiver) that was not persistent. In this case we used a
    dedicated routine (on_monitor_start) to synchronize non finished tasks, but
    this wasn't working in case of an untracked task (eg. not started with
    start_async_task). Also this was a single point of failure because it cannot
    run in multiple instance without concurrency issues (events are sent to all
    instances)
    
    Now we use a redis queue where worker put task_id and task_name to be
    synchronized and celery-monitor use brpop (https://redis.io/commands/brpop) to
    process the queue.
    
    We don't require CELERY_SEND_EVENTS to be enabled anymore (-E or --events in
    worker options).
    We require to add 'cw_celerytask_helpers.helpers' to CELERY_IMPORTS.
    f094aff48e8d