GitLab

--HG--
branch : seda21

feat: wrap BinaryDataObject and PhysicalDataObject must be wrapped in a DataObjectGroup object is defined in SEDAArchiveTransfer options (related to #36)

--HG--
branch : seda21

--HG--
branch : seda21

--HG--
branch : seda21

This is to fix SEDA exports tests on python3.

We first dump XML documents as unicode using the _encoding argument
introduced in previous changeset. Accordingly, expected files are
changed to remove the XML declaration because dumped values will not
contain them.

Then we open files in text mode (using io.open() for consistency between
python2 and python3) and do string formatting using unicode values.
(Especially, six.binary_type(<some integer>) produces different values
in python2 and python3, and we actually want to stringify the integer --
the eid value.)

We use io.open() to consistently open files in text mode in both python2
and python3 (we use text mode because we need to .format() the read
string, which is only possible with text on python3). Accordingly, we
set encoding=text_type when dumping XML to string, in order to produce a
unicode string.

Apparently, this is what python2 does.

[profile gen] Account for bdo's user cardinality or its parent's in "Integrity" element (CONSEJIRA-663)

See also https://jira.mtpl.bs.fr.atos.net/browse/CONSEJIRA-663

For more details, quoting verbatim the customer request for future reference:

| Sur les documents multi occurrences (0-n ou 1-n) ou même si l’archive
| est multi occurrence (0-n ou 1-n), il y a une balise integrity qui est
| liée au document dans le bordereau de versement que l’on génère.
| Or, dans le XSD du profil, la balise integrity du document ne possède
| pas l'attribut : maxOccurs="unbounded"
| Donc lors de l'envoi à Asalae du versement, Asalae fait une vérification
| par rapport au profil et affiche une erreur a cause de la balise
| integrity qui est présente plusieurs fois.
|
| Pour moi le correctif à faire doit porter sur le XSD et le RNG sur la
| balise integrity pour qu'elle soit multi-occurrence possible dans les cas
| suivants :
|
|   *   Si le document (Document) est en multi occurrence (0-n ou 1-n)
|       donc avec un attribut maxOccurs="unbounded"
|   *   Si une UA (Contains) supérieure au document est en multi
|       occurrence (0-n ou 1-n) donc avec un attribut
|       maxOccurs="unbounded"

| si une UA supérieur est en cardinalité max unbounded, alors
| l'integrity sur les documents en dessous doit forcément être en
| unbounded également

| il faut que l'integrity soit en facultative ou multi occurrence si une
| UA supérieure est facultative ou en multi occurrence.

This is essentially achieved by looking for the upper parent archive
units of a data object then their cardinality to produce the Integrity element.

Adding tests for new integrity_cardinality() function based on customer
examples.

In the next changeset, binary data object's user_cardinality will be
accounted for when inserting the "Integrity" element in profile whereas
it is currently hard-coded to "0..1" (see xsd_integrity() method of
SEDA1XSDExport and SEDA02XSDExport classes). In order to prevent changes
to generated profile with respect to the "seda_02_bordereau_ref.xml"
file, we set the value of user_cardinality created bdo in this test.

The idea is to have the generation of xml:id attribute configurable
based on entity context. So we create an entity adapter IXmlId that
default to the current implementation. This will be extended in
cubicweb-saem_ref where we'd want to use the ARK identifier to produce
this xml:id attribute.

In profile_generation module, we rename eid2xmlid function as xmlid and
make it accept an entity.

There may be cases where several concepts use the same label, mostly in the case
of format identifier but do the same thing for mime types as a safety belt.

because they don't need the heavy setup of the former class.

rather use ORM controlled order, consistently with the UI. Let the doctor
diagnose ambiguities (done in previous commit).

This backout most parts of 4941eab6a289.

so by default it's the master vocabulary and MIME types / PRONOM vocabularies
don't have to be kept in sync. Keep the synchronization mecanism though, and
test it separatly.

so client (e.g. GED sas) may rely on this to identify nodes unambiguously.

Closes extranet #39304385

In later commit we will insert identifiers using xml:id on more node. It sounds
like there are no benefit in using a custom attribute, so uniformize toward
using xml:id everywhere.

Related to extranet #39304385

which will occurs when the profile has some data object linked to several mime
types or format ids. Tests setup is shared with profile generation, so modify it
there, it won't hurt the original functional case to cover this as well.

Closes #36331831

since this may now occurs with values deducted from file category. In such case,
generates proper rng:choices / xsd:enumeration.

Related to #36331831

on a binary data object. This is done to prepare automatic synchronization of
these entities'values (concept) according to file_category. At some point we may
want to remove them in favor of direct relations to the binary data object,
since these are there to handle a cardinality which will be automatically
handled, provided the implemented behaviour fine with business people.

Related to #36331831

As explained in the previous commit, we want items with cardinality = 1 first
then the one with cardinality != 1 if any. See previous commit for detailed
explanation.

Closes #17098404

By trying to validate several profiles using Jing (in Asalae), it appears that
it isn't possible to validate a profile where there are several elements with
the same name with optional or/and multiple cardinality, e.g. with a profile
telling we expect  1..n Document followed by 1 Document, a transfer with two
documents won't be accepted, telling a required document is missing. This is
much probably because the validator "eats" the two documents for the first rule,
then it misses one to satisfy the second rule.

In order to avoid generating such insatisfyable profiles, while not changing
much the UI for now, we decided to add hook preventing usage of several sibling
children with a cardinality != 1 during edition, and then at RNG profile
generation time to put the one with cardinality != 1 as last children. That way
profiles should always be validable.

This patch introduces the hooks preventing insatisfyable profiles to be created.
The next one will ensure the order of RNG export.

Two hooks are added to check for unhandled case one for new watched relations
and the other for update cardinalities.

Added test cases for the 3 most prominent cases: archive unit, document and
keywords, with more coverage in the first one and only simple check for the two
others. Also, some test have to be updated to follow this change. Notice diff of
exported RNG isn't nice to read but is actually only removal of a surrounding
"oneOrMore" element.

Related to #17098404

This has several advantages:
* it's consistent with xsd/rng defaults
* it simplifies code
* it may avoid undesired validation error once #17098404 will be fixed

This requires several test update, fixed by a mix of test case update or
expected data update to minimize the changes.

Related to #17098404

else this won't be conform to the SEDA 0.2schema. But notice this is the
opposite in SEDA 1.0. In this case, test cases were fine but the problem was
actually the same: data-objects and sub-units order was random, and it's not
anymore.

As we are here, add eid so we get a total ordering, which is an expected
property so that exported profiles are consistent during time. This should be
enough until we want to control ordering from the UI.

Closes extranet #33070904

Ensure language priority.

We should override the concepts_language class attribute else we'll use the
inherited 'seda-1' language.

this will ease testing of other things in this module (as in following cset).

It should appears first for SEDA 1.0 and after custodial history in SEDA 0.2.

Update export test accordingly and add Description to SEDA 0.2 validation test.

Closes extranet #23707701

RNG doesn't handle ID/IDREF as XSD: it doesn't handle having some tag's
*content* of such type, as it is the case in SEDA 2 ObjectReferenceId element.
See https://www.oasis-open.org/committees/relax-ng/compatibility.html#id for
details.

To fix this, 'downgrade' ID into NCName. This was already done when some data
object was referenced, but it should actually be done systematically.

Closes #17078970

This is an enum in the original xsd while we don't output possible values
currently, so in the mean time make it mandatory, it's still a bit better than
relying on the original schema for that, and it makes sense from a business POV
to force specification of description level of an archive unit.

A lot of tests have to be updated, some of them because output value is now
different, others because we've to introduce a scheme+concept to handle the
validation, but they were not expecting that.

Closes #16687907

not text. Consistently with SEDA 1/0.2 export.

as this was already done for SEDA 1/0.2 export.

Move method from SEDA 1 export to the common base class with SEDA 2 export to
make it available there and avoid duplication. Also SEDA 2 test needs update
since this method return cwuri instead of absolute_url.

Related to extranet #15934739

Export used to chose concept value by searching for label with the following
language code:

1. version specific (seda-2, seda-1 or seda-02)
2. seda
3. en

and crash if no matching label was found. But it seems we don't want it to crash
because we use some concept which has e.g. only a concept in 'fr' language code.
At some point we may want to improve this, depending on the relation (internal
seda vocabulary vs open vocabulary such as keyword reference), but for now,
let's add 'fr' to the expected language list and fallback to any language if
none has been found so the export doesn't fail.

Related to extranet #15934212