GitLab

Either use a decorator or ensure that cleanup is performed.

This is so as to prevent global state modification.

When running `cubicweb-ctl newcube`, the generated template provides
debian packaging. So let's add automated tests for that using
autopkgtest.

The new test will generated a new cube, build a source tarball,
build the package, try to install it, see if the Python 3 module is
available and even run autopkgtest against the newly created packages.

Along the way it will also print information about the generated
package: buildinfo, package content, and lintian report.

The rationals behind this decision are:

- while in the past managing all instances sytem wide made a lot of sens,
  pratices have evolved today and we've moved to managing one instance by one
- this makes things easier to debug since commands since using them on several
  instances were making this harder (errors hidden in the middle)
- also solve the problem of the return code to have, before it was always 0
  which prevented to do things like:
      ipython --pdb $(which cubicweb-ctl) $command $instance
  or shell scripts that used it
- this simplify the code and is always good to take

We previously forgot to copy `tox.ini` alongside the tests. As this
file configures pytest pattern matching rules, we were not running
quite a few tests. This is now fixed.

The added tests required some adjustments in the Debian dependencies.

Sadly, not all tests currently pass. It seems there are some ordering
dependencies between the tests in regards to how data are loaded.
Fixing this probably would probably be better debugged by ensuring
the test suite does not fail when using pytest random-order plugin.

The test files shipped as part as the skeleton for `cubeweb-ctl newcube` are
not meant to be run as part of our test suite!

The version of Cryptodome currently in Debian only accepts Blowfish keys
with at least 5 bytes. This is a bug (it should accept 4 as it did
before) that has since been fixed upstream:
https://github.com/Legrandin/pycryptodome/commit/93912230f3c39e09f4038cb17e301468687c6538

So we work around this issue by using a 16 bytes key instead
(which happens to be the minimum recommended length).

This fixes:

  DeprecationWarning: time.clock has been deprecated in Python 3.3 and
  will be removed from Python 3.8: use time.perf_counter or
  time.process_time instead

We add a "separator" argument to JQueryDateTimePicker which will be used
when creating the underlying JQueryTimePicker instance so that the user
can control how time is displayed by the widget. This is very similar to
the "timesteps" argument that's already present.

This follows up on changeset 5b0ce10a7046.

Here are multiple updates to the Debian packaging template provided when
running `cubicweb-ctl newcube`:

 * Switch to source format 3.0 (quilt).
 * Switch to debhelper 9.
 * Switch to dh-python.
 * Switch to Python 3.
 * Name the binary package `python3-cubicweb-CUBENAME` instead of
   just `cubicweb-CUBENAME` (which is still the source package).
 * Populate Depends using dh_python3 support for Python requirements.
 * Run test suite at build time using pytest.
 * Add autopkgtest to run test suite against the installed package.
 * Bump Standards-Version to 4.3.0.

In order to implement automated testing of `cubicweb-ctl newcube`, we
need to avoid any user interaction. So we add a `-s` switch to specify
a short description on the command-line instead of requiring an input.

We can take the opportunity of the switch to Python 3 to get rid
of old transitional packages. Let's do it. :)

As Denis Laxalde pointed out, dh_python can also generate Recommends and
Suggests from Python package names. So let's use that instead of
manually populating these fields in `debian/control`.

Optional dependencies are currently specified in `setup.py` grouped by
feature. These dependency groups are turned into sections in
`requires.txt`. Thankfully `dh_python3` has options to populate
Recommends or Suggests with all package from a given section.

`debian/rules` now contains a list of which sections should go
to Recommends and which section should go to Suggests. Because such
extra list easily gets out-of-sync, we add a third list for ignored
sections, and ensure that all sections currently in `requires.txt`
get a mentioned in `debian/rules`.

Here are the results compared to the previous version with explicit
Recommends and Suggests (only listing Python packages):

|==============================================================|
| only in previous  |           common          | only in new  |
|==============================================================|
|                             Recommends                       |
|--------------------------------------------------------------|
|                   | python3-docutils          |              |
| python3-fyzz      |                           |              |
| python3-imaging   |                           |              |
|                   | python3-pycryptodome      |              |
|                   | python3-pyramid           |              |
|                   | python3-pyramid-multiauth |              |
| python3-pysqlite2 |                           |              |
|                   | python3-rdflib            |              |
|                   | python3-repoze.lru        |              |
| python3-simpletal |                           |              |
|                   | python3-vobject           |              |
|                   | python3-waitress          |              |
| python3-werkzeug  |                           |              |
|                   | python3-wsgicors          |              |
|--------------------------------------------------------------|
|                              Suggests                        |
|--------------------------------------------------------------|
|                   |                           | python3-pil  |


We also lose versioned dependencies which should not really be an issue
for Recommends and Suggests.

PyCryptodome comes in two flavors: “an almost drop-in replacement for the old
PyCrypto library” and “a library independent of the old PyCrypto”. The former
uses the Crypto namespace, and is shipped as `pycryptodome` while the latter
uses Cryptodome instead and lies in the `pycryptodomex` package.

Given the reason to switch to PyCryptodome is that PyCrypto in unmaintained,
its probably better to avoid any mistake and mandate the specific usage of
the Cryptodome namespace by requiring `pycryptodomex` instead of
`pycryptodome`.

A more present reason is that Debian buster will only provide a package
with the separate namespace flavor. The current Recommends is not working with
the current code. Although it's important to note that the package name will
probably have to be changed to `python3-pycryptodomex` once
https://bugs.debian.org/886291 is solved.

The default branch only provides support for Python 3.

As suggested by Denis Laxalde, we can merge python3-cubicweb-pyramid into
python3-cubicweb as all other web backend in the default branch.

As suggested by Denis Laxalde, we can let pybuild generate the list
of Depends from the egg requirements.

The list we get is slightly less specific in terms of needed versions,
but given the age of the previous ones, it's unlikely to cause issues.
Also we lose python3-six from the list… which is correct as it was
removed from the dependencies in 26744ad37953.

As pointed by Lintian, the “Apps” section is only used in menu,
not in doc-base. We can just remove it.

[pkg] Use “dependency package” instead of “virtual package” to describe python3-cubicweb-postgresql-support

The term “virtual” was too imprecise and is not accepted for empty
packages by Lintian anymore.

Just use the now preferred location.

cubicweb-ctl shipped an empty /etc/bash_completion.d as a leftover
of shipping a bash completion script. As pointed by Lintian, this
directory is not obsolete, so let's get rid of it for good.

The documentation package was missing a Built-Using field to record
the provenance of the templates.

Empty packages or meta packages do not require Depends on a specific
Python version, so we can remove the relevant substvars.

Quoting Lintian, “Since Debian Policy version 4.0.1, the priority extra
has been deprecated.” So let's use “optional” instead.

As suggested by Lintian, remove build dependency on obsolete package
dh-systemd. Instead, specify a more precise version of debhelper 9 that
added support for systemd.

Add support for autopkgtest. The test “unittest” will run the test
suite using pytest. As we need PostgreSQL in the test environment
and open network connection, the test is marked with
`isolation-container` and requires at least LXC to be run, e.g:

    autopkgtest -s -- lxc -s autopkgtest-sid

This forces us to base the Debian package on the source of the Python
package (as produced by running `python3 setup.py sdist`). While
it might seem cumbersome, this greatly reduces likelyhood of mismatch
between an installation via `pip` and one using the Debian package.

`dpkg-source` will also document for us what is currently in the
source repository but not in the Python package. Currently the missing
file are:

PKG-INFO
cubicweb/sobjects/test/data/cubicweb_comment/schema.py
cubicweb/sobjects/test/data/cubicweb_comment/__init__.py
cubicweb/sobjects/test/data/cubicweb_comment/__pkginfo__.py
cubicweb/sobjects/test/data/cubicweb_card/schema.py
cubicweb/sobjects/test/data/cubicweb_card/entities.py
cubicweb/sobjects/test/data/cubicweb_card/__init__.py
cubicweb/sobjects/test/data/cubicweb_card/__pkginfo__.py
cubicweb/server/test/data-migractions/cubicweb_comment/schema.py
cubicweb/server/test/data-migractions/cubicweb_comment/__init__.py
cubicweb/server/test/data-migractions/cubicweb_comment/__pkginfo__.py
cubicweb/server/test/data-migractions/cubicweb_localperms/schema.py
cubicweb/server/test/data-migractions/cubicweb_localperms/__init__.py
cubicweb/server/test/data-migractions/cubicweb_localperms/__pkginfo__.py
cubicweb/server/test/data-migractions/cubicweb_basket/schema.py
cubicweb/server/test/data-migractions/cubicweb_basket/__init__.py
cubicweb/server/test/data-migractions/cubicweb_basket/__pkginfo__.py
cubicweb/server/test/data-migractions/migratedapp/cubicweb_comment/schema.py
cubicweb/server/test/data-migractions/migratedapp/cubicweb_comment/__init__.py
cubicweb/server/test/data-migractions/migratedapp/cubicweb_comment/__pkginfo__.py
cubicweb/server/test/data-migractions/migratedapp/cubicweb_localperms/schema.py
cubicweb/server/test/data-migractions/migratedapp/cubicweb_localperms/__init__.py
cubicweb/server/test/data-migractions/migratedapp/cubicweb_localperms/__pkginfo__.py
cubicweb/server/test/data-migractions/migratedapp/cubicweb_basket/schema.py
cubicweb/server/test/data-migractions/migratedapp/cubicweb_basket/__init__.py
cubicweb/server/test/data-migractions/migratedapp/cubicweb_basket/__pkginfo__.py
cubicweb/server/test/data-migractions/migratedapp/cubicweb_tag/schema.py
cubicweb/server/test/data-migractions/migratedapp/cubicweb_tag/__init__.py
cubicweb/server/test/data-migractions/migratedapp/cubicweb_tag/__pkginfo__.py
cubicweb/server/test/data-migractions/migratedapp/cubicweb_card/schema.py
cubicweb/server/test/data-migractions/migratedapp/cubicweb_card/__init__.py
cubicweb/server/test/data-migractions/migratedapp/cubicweb_card/__pkginfo__.py
cubicweb/server/test/data-migractions/migratedapp/cubicweb_file/schema.py
cubicweb/server/test/data-migractions/migratedapp/cubicweb_file/__init__.py
cubicweb/server/test/data-migractions/migratedapp/cubicweb_file/__pkginfo__.py
cubicweb/server/test/data-migractions/cubicweb_tag/schema.py
cubicweb/server/test/data-migractions/cubicweb_tag/__init__.py
cubicweb/server/test/data-migractions/cubicweb_tag/__pkginfo__.py
cubicweb/server/test/data-migractions/cubicweb_card/schema.py
cubicweb/server/test/data-migractions/cubicweb_card/__init__.py
cubicweb/server/test/data-migractions/cubicweb_card/__pkginfo__.py
cubicweb/server/test/data-migractions/cubicweb_file/schema.py
cubicweb/server/test/data-migractions/cubicweb_file/__init__.py
cubicweb/server/test/data-migractions/cubicweb_file/__pkginfo__.py
cubicweb/test/data-rewrite/cubicweb_localperms/schema.py
cubicweb/test/data-rewrite/cubicweb_localperms/__init__.py
cubicweb/test/data-rewrite/cubicweb_localperms/__pkginfo__.py
cubicweb/test/data-rewrite/cubicweb_card/schema.py
cubicweb/test/data-rewrite/cubicweb_card/entities.py
cubicweb/test/data-rewrite/cubicweb_card/__init__.py
cubicweb/test/data-rewrite/cubicweb_card/__pkginfo__.py

The current manifest will duplicate several files that are stored as
symlinks in the Mercurial repository, hence the long list of ignored
files in `extend-diff-ignore`.

With changeset 234ca3cbbb46, clicking in the schema of an entity with
cubicweb in py3 on "vue en boite" will probably result in an infinite
spinner (which implies cw > 3.26)

What happened ?

This "vue en boite" used to work at least until...

hg diff -c a8c1ea390400 cubicweb/schema.py


    @@ -993,10 +992,6 @@ class CubicWebRelationSchema(PermissionM
                         return False
             return True

    -    @deprecated('use .rdef(subjtype, objtype).role_cardinality(role)')
    -    def cardinality(self, subjtype, objtype, target):
    -        return self.rdef(subjtype, objtype).role_cardinality(target)
    -

     class CubicWebSchema(Schema):
         """set of entities and relations schema defining the possible data sets


But, wait ...
If I open a shell on an instance of cw 3.24 something seems off

>>> list(schema['CWUniqueTogetherConstraint'].relation_definitions())[0][0].cardinality
# <bound method CubicWebRelationSchema.wrapped of <constraint_of [CWUniqueTogetherConstraint,CWEType]>>

We have been sorting on a method the whole time ? Is it possible what
were the effects ?

1) We cannot sort function can't we ?

>>> def adder(i): return lambda x: x+i
>>> sorted(map(adder,range(10)))
[<function __main__.<lambda>>,
 <function __main__.<lambda>>,
...

Yes we can.

2) what does it means.

>>> { adder(1) : 1 }
Out[19]: {<function __main__.<lambda>>: 1}

In fact the function object as a __hash__ method (which is practical for making
memoizers (cache)), and return truly random results (pseudo random).

My take on this patch is relations have NEVER been sorted by cardinality.
No one never ever noticed. Hence, I propose to not fix a bug that never was
reported.

py2 support being dropped, passing the code in full py3 str/bytes syntax

--HG--
branch : 3.26