1. 17 Jan, 2017 1 commit
  2. 16 Jan, 2017 3 commits
    • Philippe Pepiot's avatar
      [entities] don't log "cannot deserialize task" · ae5462a6f276
      Philippe Pepiot authored
      This usecase (sending a serialized task signature as result of a task)  is
      exceptional, so avoid flooding the logs with such messages.
    • Philippe Pepiot's avatar
      monitor: can work with multiple instances · f094aff48e8d
      Philippe Pepiot authored
      Previously we handle monitoring celery task by listening to celery event bus
      (celery.events.EventReceiver) that was not persistent. In this case we used a
      dedicated routine (on_monitor_start) to synchronize non finished tasks, but
      this wasn't working in case of an untracked task (eg. not started with
      start_async_task). Also this was a single point of failure because it cannot
      run in multiple instance without concurrency issues (events are sent to all
      Now we use a redis queue where worker put task_id and task_name to be
      synchronized and celery-monitor use brpop (https://redis.io/commands/brpop) to
      process the queue.
      We don't require CELERY_SEND_EVENTS to be enabled anymore (-E or --events in
      worker options).
      We require to add 'cw_celerytask_helpers.helpers' to CELERY_IMPORTS.
    • Philippe Pepiot's avatar
      CeleryTask: move sync task logic in sync_task_state · c97f0e74bb5f
      Philippe Pepiot authored
      - strict policy on sql transaction (commit all or nothing for each task)
      - make it work with on_monitor_start (used to synchronize task states when
        monitor starts in case of missing events)
      - Use serializable() in "spawn" task so it force celery to use the json task
        serializer instead of pickle
      - Don't always update task_name when creating subtasks outside of celerytask
        (eg. by using start_async_task), use a fixed identifier "<unknown>" as task
        name instead and only update these.
  3. 02 Jan, 2017 3 commits
  4. 16 Dec, 2016 4 commits
  5. 14 Dec, 2016 1 commit
  6. 06 Dec, 2016 1 commit
  7. 15 Nov, 2016 1 commit
  8. 06 Dec, 2016 1 commit
  9. 15 Nov, 2016 1 commit
  10. 12 Dec, 2016 1 commit
    • David Douard's avatar
      [entities] improve workflow management of CeleryTasks (by celery-monitor)... · 51e3f4e41879
      David Douard authored
      [entities] improve workflow management of CeleryTasks (by celery-monitor) robustness (related to #16640842)
      the transition may fail (if for some race-like reasons it has already been fired).
      Also ensure some CeleryTask changes are commited (celery-monitor)
      since the transaction may be rolled back during the WF management part of the
      on_event() method.
  11. 04 Nov, 2016 8 commits
  12. 28 Oct, 2016 2 commits
  13. 12 Dec, 2016 1 commit
  14. 04 Nov, 2016 5 commits
  15. 19 Oct, 2016 1 commit
    • Florent Cayré's avatar
      [entities] Determine task's state from db state if final, else on celery... · fdf1a4d0a890
      Florent Cayré authored
      [entities] Determine task's state from db state if final, else on celery backend's if any (closes #16640866)
      Relying on final db states avoids inconsistencies between the CW workflow
      and celery-backend state, which should only be used to add real-time info.
      When no reliable state can be found, return 'unknown state' explicitely.
  16. 20 Oct, 2016 2 commits
  17. 18 Oct, 2016 2 commits
  18. 17 Oct, 2016 2 commits