GitLab

The merge of the repo-side and web-side entity caches doesn't actually work,
because entities have a _cw attribute which is sometimes a connection and
sometimes a web request, and whose nature changes not-really-predictably
whenever ResultSet.get_entity() is called and brings it out of the cache.

This reverts commit 59a79300f213.

The BFSS API changes in python 3:

* 'defaultdir' MUST be a unicode object
* 'fsencoding' MUST NOT be set

In python 2, fsencoding handles both the encoding of file paths on the
file system (utf-8 by default, but the system may actually be using
something else) and the encoding of file paths that will be stored in
the database.

So in python 3, we wipe the slate clean:

* rely on sys.getfilesystemencoding() to convert unicode objects to
  bytes
* always encode paths to utf-8 for storage in the database

Caveat emptor / here be dragons:

* sys.getfilesystemencoding() depends on the current locale, which
  therefore MUST be set properly
* when migrating an existing instance from py2 to py3, one MAY need
  to reencode file paths stored in the database

We no longer deal with Session objects here.

Avoids race condition, at the cost of slightly less predictable file names.

The ClientConnection/Connection duality brought some painful hacks
that are no longer needed.

Hence we can send entity._cw_update_attr_cache and ._cw_dont_cache_attribute
to the grave.

Related to #3837233

Connection replaces ClientConnection everywhere.

Some notes:

* testlib: .client_cnx and .repo_cnx become aliases of .cnx (we might
  not want to tell people to update their tests again for just no real
  benefit, so we'll live with these aliases for a while)

* entity.as_rset must not be cached because we risk caching result
  sets having a Connection object as .req (helps unittest_breadcrumbs)

* entity._cw_dont_cache_attributes loses its repo/request special
  paths and only keeps its storage/bfss user (this helps
  unittest_wfobjs)

* moreover, entity.cw_instantiate and .cw_set stop overriding the
  attributes cache *after* the before_*_entity hooks have run, because
  there is no need to (it is now actually harmful to do it and
  unittest_hooks.test_html_tidy* tests remain green because of this)

* rset._build_entity sticks its .req onto the entity just fetched from
  the cache, because otherwise it might carry a _cw that is a
  Connection object where a Request is expected (helps
  unittest_views_actions)

* we get overall better cache usages (entity caches were split over
  Request + ClientConnection and Connection), hence the changes
  unittest_entity and unittest_wfobjs

* void the ecache when providing the cnx to a request object

  Having the entity cache pre-filled when we bind it to the request
  object hurts because these entities are bound to Connection objects,
  that lack e.g. `.form` or `.add_js` and crash the views subsystem.

  Thus, the unittest_testlib.test_error_raised test will are kept
  green.


Closes #3837233

since it now handles a connection to the system source only

Related to #2919300

[jcr: adjust 3.17.11 migration, fix a number of bugs in new
 ConnectionsSet implementation, fix
 source.{open,close}_source_connections]

We have a clean public API now.

The older form is not necessary since we dropped python 2.5 compability. This
will help Python 3.3 experimentation.

This will finally fix the `test_bfss_fs_importing_transparency` pass again.

Regression was introduced by 4ba11607d84a

When the <old value> == <new value> (not really and update but can happen), the
value was popped but never restored. The cw_edited attribut had wrong content
for a small time lapse.

--HG--
branch : stable

In the process binary.to_file now takes a file instead of a file-path. As
storage was currently the only user it should not impact anyone.

otherwise you can get crashed with AssertionError: Binary objects must use raw strings, not <type 'unicode'>

--HG--
branch : stable

files on windows network share must be read / written by chunks of reasonable size
or you get some unusual os level errors.

--HG--
branch : stable

The current code assume attr value is alway a valide Binary value. But when the
attr is optional, the value might be None. The change handle this case.

--HG--
branch : oldstable

This will also catch IOError (as file not found) that might be encountered expectable too.

--HG--
branch : stable

current_fs_path was returning the future fspath when no value was defined.

* This does not make any sense
* This tricks the storage into believing that updating the attribute content is
  not necessary. The file was written but the attributes was not updated. as a
  result the assignement was not made.

(close #1725446)

--HG--
branch : stable

Also document session's data storage and some other internals.
Hopefuly things will get clearer.

Closes #1684860: vocabulary confusion in repository code: notion of 'pool'

[server] move EditedEntity class to its own module, to avoid cyclic dependency when needed from e.g. session.py

--HG--
branch : stable

--HG--
branch : stable

Operation api
~~~~~~~~~~~~~
* commit_event killed, recently introduced postcommit_event is enough and has a better name
* kill SingleOperation class, it's a) currently never used b) superseeded by set_operation if needed.


Entity edition api
~~~~~~~~~~~~~~~~~~
edited_attributes turned into a special object holding edition specific attributes:

- attributes to be edited (simply mirrored in cw_attr_cache, actual values are there)
- former _cw_skip_security set (cw_edited) and querier_pending_relations
It has also been renamed to `cw_edited` on the way (it may also contains inlined relations)

The entity dict interface has been deprecated. One should explicitly use either
cw_attr_cache or cw_edited according to the need.

Also, there is now a control that we don't try to hi-jack edited attributes
once this has no more effect (eg modification have already been saved)

At last, _cw_set_defaults/cw_check internal methods have been moved to this
special object


Hook api
~~~~~~~~
hook.entity_oldnewvalue function now moved to a method of cw_edited object.

and not only at commit time. So it's consistent with entity creation behaviour.

The new file is created at assignement time and removed if the commit is
rollbacked.

--HG--
branch : stable

--HG--
branch : stable

--HG--
branch : stable

--HG--
branch : stable

--HG--
branch : stable

--HG--
branch : oldstable

--HG--
branch : stable

[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941

[storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding

--HG--
branch : stable

The previous implementation was bugged (prefixing the whole path with '_'
instead of the base name). A new version (using number) replace it.
* * *
Improve BytesFileSystemStorage.new_fs_path to use available metadata

This version try to get some hint about how to name the file using metadata.
Using the real file name and extension when available.  Keeping the extension
might be usefull for exemple in the case of processing that use filename
extension to detect content-type.

--HG--
branch : stable

--HG--
branch : stable

When creating a new File object, if fs_importing is set,
we want entity.data to be the file content instead of the
filepath for the rest of the transaction.
(see test_bfss_fs_importing_transparency) for test implementation

To make this possible, the storage hooks (entity_added / entity_updated)
must return the correct value to set in the entity dict.

--HG--
branch : stable