GitLab

This allows installation of cw-celerytask-helpers 0.8.0 which is still
compatible.

Why upgrading CubicWeb minimum version? We don't support 3.25 and
less, and even 3.26 should disappear but it is the last version
that supports Python 2 and it is in consequence sadly still used.
So it is better not to create illusion and be clear that we won't
add a fix for a too old version of CubicWeb. Projects must
upgrade, maintaining very old stuff is dirty work.

In a client project, we add the case of `result.info` being
`UException` (from `cubicweb_nemo.utils`). It resulted in a
problem, because `result.info` is `True` as boolean, but it has
no key, so it crashed. This patch does not resolve the real
problem of having a clean `info` member or knowing why it could
normally have a different type, but at least it eliminates a
problem for the final users.

Using https://six.readthedocs.io/#six.ensure_text

It can occur, for instance in case of "worker lost", that task events are not
sent in the "MONITOR_KEY" list in redis.
The proposed solution is to regularly check pending tasks status.
Since this is almost the same use, this can replace the old "requeue" behavior
of failed sync_task_state() which was based on a "PENDING_KEY" list in redis.

So, every 10 min, was 1min before, but 10 min seems more reasonable, check all
pending tasks status within the visibility_timeout, since tasks outside this
range cannot have state stored in the broker.

In case of exited worker (segfault), AsyncResult.traceback is None and
information is in AsyncResult.result, record it in the failed transition
comment.

Add a dedicated test class, this require to run the worker in prefork mode,
otherwise, in solo mode the information about worker lost isn't tracked.
Use "import ctypes; ctypes.string_at(0)" to simulate a segfault.

Since the upgrade to celery 4, this test pass.

So the tasks aren't in the scope of the process actually launching tasks.
Dropping this ensure our code works even when tasks aren't registered, this
avoid to introduce code requiring the task to be registered (celery exception
NotRegistered).

This is not backward compatible with celery 3, so pin celery>=4

See http://docs.celeryproject.org/en/4.0/whatsnew-4.0.html for summary of changes

Change attributes from celery.app.config to lower case.
Drop setting serializer to json in README and tests since json is now the
default serializer.

I had issues with test_workflow_group() where AsyncResult().get on the *group*
task_id hang. It looks related to our test infrastructure and AFAIK there is no
known usage of this for cubes using cubicweb-celerytask.
I think we must dig into this bug but I already spent enough time on this, so
let's move forward and skip this part of the test.

Drop assertion "cwtask.eid in results" which is already implicitly checked by "results[cwtask.eid]"
Only run a group(success(), success()), which is enough for the scope of this
test and make it compatible with celery 4 where nested groups are rewrited in a
single group.
Wait and test results of all subtasks, not only the first one.

This code is only used in tests where we check for existence of task_id in database.
Before running the Task, there is no guarantees that the value of Task.id will
be used as task_id, this is the purpose of freeze().
In this particular case Task.id = Task.freeze().id for celery 3.1.25, but this
is no longer true in celery 4.

We should avoid calling freeze() within a task and avoid calling freeze() twice
(once in test and once in ICeleryTask.attach_tasks()). This produce all kinds
of weird issues along celery versions, I think the API and behavior of freeze()
is still somewhat experimental...
Only assert on length of child tasks and test they are all revoked which is the
purpose of this test.

Only Task objects pass through get_task_id() and never AsyncResult, thus get_task_id()
is now useless and is equivalent to task.freeze().id
I think an old version of attach_task() was recursively called with either Task
or AsyncResult thus the handling of both cases, but I cannot find the relevant
changeset (maybe this code was never used ? ...)
I'm pretty confident on this because all known use-cases are covered by tests.

This make code more readable and less error prone.

serializable() and from_serializable() are deprecated in flavor of as_tuple()
and result_from_tuple().
This is required to migrate to celery 4