GitLab

Closes #17219673

While this toolbar isn't yet very integrated with CW, it integrates an inline
debugging shell like werkzeug for flask or django_extensions which greatly
helps debugging on exception.

It can already be manually activated by writting "pyramid.includes =
pyramid_debugtoolbar" but it's hidden somewhere in the documentation and not
very accessible and annoying to do.

Closes #17219765

--HG--
branch : 3.26

This will display lines like:

2019-06-12 16:26:57 - (cubicweb.pyramid.bwcompat) INFO: REQUEST [view] '/wiki/334037' selected controller <cubicweb.web.views.basecontrollers.ViewController object at 0x7f133802ae90> at /root/.virtualenvs/abreton/local/lib/python2.7/site-packages/cubicweb/web/views/basecontrollers.py:120

2019-06-12 16:26:59 - (cubicweb.pyramid.bwcompat) INFO: REQUEST [data] '/data/ea5eb3b6f89fba0ff4b671093c88d3dd/images/cartouche-mur.jpg' selected controller <cubicweb.web.views.staticcontrollers.DataController object at 0x7f1332a84d50> at /root/.virtualenvs/abreton/local/lib/python2.7/site-packages/cubicweb/web/views/staticcontrollers.py:179

The make debugging easier as this isn't done already. Most other web frameworks
do something similar for the developper to be able to understand what is
happening right now.

Closes #17219741

Closes #17219653

Daemonization used to make sens in the past, but today "pyramid" command is
only used for dev and should never be used for deployement (you should use a
wsgi server for that instead) so daemonization doesn't fit any use case
anymore.

Closes #17232923

This warning isn't useful anymore.

Followup fa292e905edc which require cubicweb.session.secret to be set.

Followup fa292e905edc

DeprecationWarning: In future versions of Waitress clear_untrusted_proxy_headers will be set to True by default. You may opt-out by setting this value to False, or opt-in explicitly by setting this to True.

Source: https://docs.pylonsproject.org/projects/waitress/en/stable/arguments.html?highlight=clear_untrusted_proxy_headers

> This tells Waitress to remove any untrusted proxy headers ("Forwarded",
> "X-Forwared-For", "X-Forwarded-By", "X-Forwarded-Host", "X-Forwarded-Port",
> "X-Forwarded-Proto") not explicitly allowed by trusted_proxy_headers.

According to grep we don't use any of those headers so let's turn it on for
security reasons.

DeprecationWarning: The default pickle serializer is deprecated as of Pyramid 1.9 and it will be changed to use pyramid.session.JSONSerializer in version 2.0. Explicitly set the serializer to avoid future incompatibilities
. See "Upcoming Changes to ISession in Pyramid 2.0" for more information about this change.

As describe here https://docs.pylonsproject.org/projects/pyramid/en/1.10-branch/narr/sessions.html#pickle-session-deprecation
use a serializer that fallback on pickle if needed to avoid impacting users.

DeprecationWarning: The SafeConfigParser class has been renamed to ConfigParser in Python 3.2. This alias will be removed in future versions. Use ConfigParser directly instead.

--HG--
branch : 3.26

Now that we use Python 3 only, this is possible.

This mostly consists in removing the dependency on "six" and updating
the code to use only Python3 idioms.

Notice that we previously used TemporaryDirectory from
cubicweb.devtools.testlib for compatibility with Python2. We now
directly import it from tempfile.

This follows up on changeset a8c1ea390400, in which code deprecated
using logilab.common.deprecation got dropped. Now we also drop code
deprecated using stdlib's warn(<msg>, DeprecationWarning).

Notice that, as a consequence of dropping old/new etypes aliases in
cubicweb/schema.py, we drop the import ETYPE_NAME_MAP (no longer
needed); but since other modules imported that name from
cubicweb.schema, we need to update the import statement to use
"cubicweb" directly.

Twisted web server is not used anymore and has been superseded by
pyramid many years ago. Furthermore, our usage is not compatible with
Python 3. So we drop the "etwist" sub-package.

As a consequence, "all-in-one" configuration type gets dropped as it was
Twisted-specific. We resurrect it in cubicweb/pyramid/config.py by only
keeping options used by the "pyramid". Similarly, we introduce a
AllInOneCreateHandler in cubicweb/pyramid/pyramidctl.py that is
basically the one that lived in cubicweb/etwist/twctl.py and is used to
create the "all-in-one" instance. Added a TODO here about "pyramid.ini"
that could be generated at the end of bootstrap() method.

In cubicweb/devtools/httptest.py, CubicWebServerTC is now equivalent to
CubicWebWsgiTC and the latter is dropped.

As in previous changesets, most code has been deprecated since a very
long time. This changeset concerns "core" cubicweb modules
(e.g. "server", "appobjects") for which it's not easy to handle change
atomically.

In cubicweb/server/querier.py, we adjust empty_rset() function as a
result of "rqlst" argument of ResultSet being dropped. (There was no use
of the keyword argument anyways.)

Based on django source code in case SystemRandom is not available.

According to python documentation:

https://docs.python.org/2/library/random.html
> Warning
> The pseudo-random generators of this module should not be used for security
> purposes. Use os.urandom() or SystemRandom if you require a cryptographically
> secure pseudo-random number generator.

--HG--
branch : 3.26

--HG--
branch : 3.26

Flake8 had a new release which raise new issues, namely:

W504: line break after binary operator
F841: local variable 'ex' is assigned to but never used
W605: invalid escape sequence
F821: undefined name 'buffer' (noqa seems the only way to avoid this false positive)

Also pin flake8>=3.6 in our tests and make explicit that we use python3 to run
flake8 tests.

--HG--
branch : 3.26

Flake8 had a new release which raise new issues, namely:

W504: line break after binary operator
F841: local variable 'ex' is assigned to but never used
W605: invalid escape sequence
F821: undefined name 'buffer' (noqa seems the only way to avoid this false positive)

Also pin flake8>=3.6 in our tests and make explicit that we use python3 to run
flake8 tests.

For a new-style cube, if we import "cubicweb_<name>" and then import
"cubes.<name>", the cube will be imported twice. cubes.<name> and then
cubicweb_name is ok though...

When using pyramid, we try to find which cube define a "includeme" by importing
them with the name "cubes.<name>", so we (possibly ?) double import all new-style cube.
This case may also occur in tests with PyramidCWTest.

Touching the import loader to fix this scares me, so let's fix this by testing
"cubicweb_<name>" *before* "cubes.name" (for old style cubes, importing
cubicweb_<name> raise ImportError).

--HG--
branch : 3.26

Using six is better than the try/except because it clearly indicates
compatibility code. Also, when using tools such as vermin which relies
on AST, it helps getting rid of false compatibility errors.

--HG--
branch : 3.26

As noted in comment, it might occur that we receive a "sessioneid" from
a valid cookie while respective CWSession got dropped (typical case is
db being recreated while users got preserved). In such case, we just
recreate the CWSession entity as if it did not exist.

--HG--
branch : 3.26

We do not initialize logging in CubicWebPyramidConfiguration thus
preventing logging to be started with cubicweb's configuration. On the
other hand, any logging configuration defined in ``development.ini``
file will be loaded by whatever start the instance using this file (i.e.
pserve, gunicorn, etc.). The benefit is that logging can now be
configured per "qualified name" in this file (i.e. one can easily set
the DEBUG level for the application cube while keeping all dependencies'
level to WARNING). In the development.ini template, we add logger
configurations for "logilab" and "cubicweb" qualified names (along with
those of the application cube).

When some session or authtk secret is missing in Pyramid settings,
scary "!! SECURITY WARNING !!" are issued. This is arguably pointless in
tests. So disable them in this case.

See https://docs.pylonsproject.org/projects/waitress/en/latest/#using-behind-a-reverse-proxy

Since we do not configure trusted_proxy waitress ignore X-Forwarded-Proto
header and may generate http urls (pyramid request.url) when the instance is
behind a https reverse proxy. This cause cubicweb-signredrequest to not work
with since it rely on the url (including scheme).

Set url_scheme to 'https' when CubicWeb base-url scheme is https as a
workaround.

--HG--
branch : 3.25

Also refactor a bit the method to restrict the try/except block to the
code for which we actually want to catch an exception (i.e. the
cnx.execute() + cnx.commit()).

--HG--
branch : 3.25

The "pyramid" instance configuration does not work with "cubiwceb.bwcompat"
mode (on purpose). Yet, having the setting exposed in development.ini file
(generate by `cubicweb-ctl create --config pyramid <cube> <instance>` command)
is misleading and we want to remove it.

Thus, we only query this setting when cubicweb configuration is "all-in-one" and
drop the setting line from templated development.ini file. If the option is
found and True for any other configuration type, we issue a user warning (and
ignore the option).

--HG--
branch : 3.25