s3storage issueshttps://forge.extranet.logilab.fr/cubicweb/cubes/s3storage/-/issues2020-08-19T14:50:12Zhttps://forge.extranet.logilab.fr/cubicweb/cubes/s3storage/-/issues/1Provide ccplugin to test s3 storage configuration2020-08-19T14:50:12ZArthur LutzProvide ccplugin to test s3 storage configurationIt would be nice to have a ccplugin that makes it easier to test if an s3 storage configuration is correct.
`cubicweb-ctl s3-test instance` would create a file, so some checks on it, and then remove it
I don't know if it could be in ...It would be nice to have a ccplugin that makes it easier to test if an s3 storage configuration is correct.
`cubicweb-ctl s3-test instance` would create a file, so some checks on it, and then remove it
I don't know if it could be in two steps so that the user can go and look at the bucket and see the `cubicweb-hello-world.txt` file exists and then go back and have another command to remove the test.https://forge.extranet.logilab.fr/cubicweb/cubes/s3storage/-/issues/2Dependency problem with urllib3 & python3.7 & botocore2021-01-20T10:08:55ZArthur LutzDependency problem with urllib3 & python3.7 & botocoreAfter adding s3storage as a dependency :
```
INFO[0175] Running: [/bin/sh -c docker-cubicweb-helper create-instance]
Traceback (most recent call last):
File "/usr/bin/cubicweb-ctl", line 11, in <module>
load_entry_point('cubicwe...After adding s3storage as a dependency :
```
INFO[0175] Running: [/bin/sh -c docker-cubicweb-helper create-instance]
Traceback (most recent call last):
File "/usr/bin/cubicweb-ctl", line 11, in <module>
load_entry_point('cubicweb==3.26.17', 'console_scripts', 'cubicweb-ctl')()
File "/usr/lib/python3/dist-packages/cubicweb/cwctl.py", line 1085, in run
cwcfg.load_cwctl_plugins()
File "/usr/lib/python3/dist-packages/cubicweb/cwconfig.py", line 694, in load_cwctl_plugins
for cube in cls.available_cubes():
File "/usr/lib/python3/dist-packages/cubicweb/cwconfig.py", line 472, in available_cubes
module = entry_point.load()
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2410, in load
self.require(*args, **kwargs)
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2433, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (urllib3 1.24.3 (/home/cubicweb/.local/lib/python3.7/site-packages), Requirement.parse('urllib3<1.26,>=1.25.4; python_version != "3.4"'), {'botocore'})
```https://forge.extranet.logilab.fr/cubicweb/cubes/s3storage/-/issues/3Document access from pyramid part2020-12-08T10:54:52ZArthur LutzDocument access from pyramid partOften cubicweb application also have a "pure pyramid" part which doesn't use the cubicweb part.
It would be nice to document how to use s3storage in conjunction with that.Often cubicweb application also have a "pure pyramid" part which doesn't use the cubicweb part.
It would be nice to document how to use s3storage in conjunction with that.https://forge.extranet.logilab.fr/cubicweb/cubes/s3storage/-/issues/4Document access to s3storage ressources from nginx2020-12-08T10:56:28ZArthur LutzDocument access to s3storage ressources from nginxIn some use cases when s3storage is used we might want to bypass completely pyramid/cubicweb and serve files from nginx (or even in public bucket domain), document that usecase.In some use cases when s3storage is used we might want to bypass completely pyramid/cubicweb and serve files from nginx (or even in public bucket domain), document that usecase.https://forge.extranet.logilab.fr/cubicweb/cubes/s3storage/-/issues/6Faciliter la surcharge des DeleteFileOp.postcommit_event et AddFileOp.postco...2021-02-23T17:00:04ZKatia SaurfeltFaciliter la surcharge des DeleteFileOp.postcommit_event et AddFileOp.postcommit_eventSur FranceArchives nous avons besoin de surcharger les DeleteFileOp.postcommit_event et AddFileOp.postcommit_event pour chaque fichier traité. Serait-il possible d'avoir la possibilité d'appeler du code sur chaque donnée :
```
class...Sur FranceArchives nous avons besoin de surcharger les DeleteFileOp.postcommit_event et AddFileOp.postcommit_event pour chaque fichier traité. Serait-il possible d'avoir la possibilité d'appeler du code sur chaque donnée :
```
class S3DeleteFileOp(DataOperationMixIn, LateOperation):
containercls = list
def new_func_process_data(self, storage, key, eid, attr):
pass
def postcommit_event(self):
for storage, key, eid, attr in self.get_data():
self.info('Deleting object %s.%s (%s/%s) from S3',
eid, attr, storage.bucket, key)
resp = storage.s3cnx.delete_object(Bucket=storage.bucket, Key=key)
if resp.get('ResponseMetadata', {}).get('HTTPStatusCode') >= 300:
self.error('S3 object deletion FAILED: %s', resp)
else:
self.debug('S3 object deletion OK: %s', resp)
self.new_func_process_data(storage, key, eid, attr)
```https://forge.extranet.logilab.fr/cubicweb/cubes/s3storage/-/issues/8Check new_key2021-02-23T17:03:33ZKatia SaurfeltCheck new_keyVerify file names doesn't contain "Characters to avoid" :
* https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html
* doesn't start with "/"Verify file names doesn't contain "Characters to avoid" :
* https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html
* doesn't start with "/"Elouan MartinetElouan Martinet