1. 14 Mar, 2017 1 commit
  2. 22 Feb, 2017 4 commits
  3. 20 Feb, 2017 3 commits
  4. 16 Feb, 2017 4 commits
  5. 17 Jan, 2017 3 commits
    • Philippe Pepiot's avatar
      [entities] force convention for subtask created in a task · b27994d95bc7
      Philippe Pepiot authored
      Convention is to return a dict with a key "celerytask_subtasks".
      So we can remove a "except Exception" that could hide errors or future bugs.
    • Philippe Pepiot's avatar
      [ccplugin] celery-monitor: use only one connection to the database · 495f5c557506
      Philippe Pepiot authored
      Set CW_CONNECTIONS_POOL_SIZE to 1 (by default) to only use one connection to
      the database (no more is needed since it's single threaded).
      Setting the parameter in the "config" object could not work since CubicWeb
      reload it from config files when loading cubes.
    • Philippe Pepiot's avatar
      [ccplugin] celery-monitor: retry failed items · be044d7109a3
      Philippe Pepiot authored
      When multiple instance of celery-monitor are running, we could have an
      integrity errors raised if two instance are working on the same task_id or
      temporary (network, host) failures. In this case we want to retry handling the
      task_id later.
      Put pending processed task_id in a "pending queue" and each minutes and if the
      monitor queue is empty, requeue pending items.
      This change require to handle the "timeout" parameter of loop (only used in
      tests) in a different way to ensure not blocking forever in redis "brpoplpush".
  6. 16 Jan, 2017 2 commits
  7. 17 Jan, 2017 1 commit
  8. 16 Jan, 2017 3 commits
    • Philippe Pepiot's avatar
      [entities] don't log "cannot deserialize task" · ae5462a6f276
      Philippe Pepiot authored
      This usecase (sending a serialized task signature as result of a task)  is
      exceptional, so avoid flooding the logs with such messages.
    • Philippe Pepiot's avatar
      monitor: can work with multiple instances · f094aff48e8d
      Philippe Pepiot authored
      Previously we handle monitoring celery task by listening to celery event bus
      (celery.events.EventReceiver) that was not persistent. In this case we used a
      dedicated routine (on_monitor_start) to synchronize non finished tasks, but
      this wasn't working in case of an untracked task (eg. not started with
      start_async_task). Also this was a single point of failure because it cannot
      run in multiple instance without concurrency issues (events are sent to all
      Now we use a redis queue where worker put task_id and task_name to be
      synchronized and celery-monitor use brpop (https://redis.io/commands/brpop) to
      process the queue.
      We don't require CELERY_SEND_EVENTS to be enabled anymore (-E or --events in
      worker options).
      We require to add 'cw_celerytask_helpers.helpers' to CELERY_IMPORTS.
    • Philippe Pepiot's avatar
      CeleryTask: move sync task logic in sync_task_state · c97f0e74bb5f
      Philippe Pepiot authored
      - strict policy on sql transaction (commit all or nothing for each task)
      - make it work with on_monitor_start (used to synchronize task states when
        monitor starts in case of missing events)
      - Use serializable() in "spawn" task so it force celery to use the json task
        serializer instead of pickle
      - Don't always update task_name when creating subtasks outside of celerytask
        (eg. by using start_async_task), use a fixed identifier "<unknown>" as task
        name instead and only update these.
  9. 02 Jan, 2017 3 commits
  10. 16 Dec, 2016 4 commits
  11. 14 Dec, 2016 1 commit
  12. 06 Dec, 2016 1 commit
  13. 15 Nov, 2016 1 commit
  14. 06 Dec, 2016 1 commit
  15. 15 Nov, 2016 1 commit
  16. 12 Dec, 2016 1 commit
    • David Douard's avatar
      [entities] improve workflow management of CeleryTasks (by celery-monitor)... · 51e3f4e41879
      David Douard authored
      [entities] improve workflow management of CeleryTasks (by celery-monitor) robustness (related to #16640842)
      the transition may fail (if for some race-like reasons it has already been fired).
      Also ensure some CeleryTask changes are commited (celery-monitor)
      since the transaction may be rolled back during the WF management part of the
      on_event() method.
  17. 04 Nov, 2016 6 commits