Design: External storage options

[cco3]

At the moment, if the containers are destroyed, information about scans are stored in the database (assuming a remote DB has been configured). However, the original image will be lost. It would be nice if the image could also be stored in a remote storage solution. E.g., AWS, google cloud storage, ftp, etc.

Example usage:

1. Configure remote DB
2. Configure remote storage
3. Do some scans
4. Delete containers
5. Bring up new containers with same configuration
6. Still have access to original images and scan info

[pombredanne]

@cco3 sorry for the late reply... I missed this one.
How would you see this done?

1. a global feature with per-instance setting that would systematically safeguard a copy of any project input files using some file services
2. something to be configured on a per-pipeline basis?

[cco3]

The former.

I'm imagining it would be analogous to how you configure alternate DBs, except using standards for file storage. https://scancodeio.readthedocs.io/en/latest/scancodeio-settings.html#database

[pombredanne]

Adding this to the latest milestone as this needs some minimal design first. @cco3 could you help there?

[cco3]

Ideally what I would like is just to specify a URL to a google storage bucket in the config. If I understand correctly, there are a number of services that have compatible apis so you can use the same clients on all of them.