GitLab

Hooks require uploaded files to be accessible and open

Instead of doing storage attribution in test setup, we implement a test
hook (in test/data/hook.py) as would be done in real application. As a
consequence, we change the way s3 requests are mocked (using moto).
Namely, we introduce a "_s3_client" class method in S3Storage that is
mocked in test setup so that mocking operates in the test hook.

In migration tests, instead of instantiating a new storage object, we
reuse the one bound to Image's data attribute so as to benefit from the
active mock.

We now use an "Image" entity type defined in test/data/schema.py with a
"data" bytes attribute along with a "thumbnail" one. The former is used
as File's data attribute was. The latter is now used within
S3StorageMigrationTC (instead of "data" attribute); this is in
preparation of the next changeset in which s3 storage setup will be
handled in a test hook (instead of test setup).

So as to ease testing in downstream projects.

This seems more appropriate as we're actually not doing any database
setup.

Otherwise, reading the Binary object with .read() (not .getvalue())
returns nothing since the stream is positioned at the end.

Accordingly, use .read() method in test.

Version 3.24.9 previously used does not exist and 3.24.7 is the latest
one available at the moment.

This is the API I'm used to and it makes some operations simpler.

Also notice that we now use a fake region name ("somewhere"), this
should at least avoid doing real requests whereas using "eu-west-1"
might have side effects if the user has boto credentials and moto's
mocks fail.

See https://github.com/spulec/moto/issues/1793 for details.

no need for complicated code for parallel file upload.

so it can be used by other test classes without having to use test class
heritage.

S3 object deletion can be disabled via a 's3-auto-delete' config option.

This implementation provides an S3Storage that can be used to store entities
attributes in an Amazon S3 storage.

It comes with an STKEY rql function to be able to retrieve the S3 object key
for the entity attribute (similar to BFSS's FSPATH rql function).

S3 connection configurations are attached to the S3Storage instance, like:

  s3_storage = S3Storage(s3_bucket)
  storages.set_attribute_storage(repo, 'File', 'data', s3_storage)