feature: saving files as blob in database

[DanielAuerX]

• files (e.g. branding pictues, attachments) can be saved as a blob in a database. The feature can be turned on and off (feature toggle) with the env var FILE_STORAGE_MODE

• a new controller with the endpoints /file/branding etc. had to be created since the files are statically linked when fs is used.

• the files plus metadata are stored in the table 'file'

[DanielAuerX]

some clean up tasks etc are needed. just wanted to see if there is interest in a feature like this.

[LinkinStars]

Although it is possible to save files in the database, we do not recommend it. Since the number of files can be large, saving them in the database not only causes performance problems with queries, it can also be very troublesome when it comes time to back up database data. You should be able to find some bad things about storing files in databases in the web. All in all we still don't recommend users to store files in the database. Storing them in the file system or a third-party cloud provider using a plugin is good.

[mgerhardy]

+1 for this feature.

It is quite troublesome to keep backups and restores in sync if you have multiple infrastructure components. It's also a cost aspect for us.

[columbo28]

+1 for this PR

I would also like to have that ability as we are using Apache Answers also.

At the moment we have to align the backups of the filesystem AND the database backup as well.
Both of them must match to each other.

If we use the ability to store the files as blobs within the database, we would only have to care about a single backup component.

We could use the toggle that @DanielAuerX already provided to explicitly enable the blob feature, by default we simply set FILE_STORAGE_MODE to 'fs'.

What do you say @LinkinStars ? :)

Cheers,
Steven

[LinkinStars]

@mgerhardy @columbo28

Thanks for the feedback, it's important for us to hear more varied suggestions. Of course, there are advantages and disadvantages to putting files in the database. I understand that this can be turned off by default by setting ENV to keep the existing feature unaffected. Other than that, we'll mainly be looking at maintenance costs.

Let's hear more ideas from others before we decide on this feature. @sy-records @kumfo

[DanielAuerX]

When I was looking into how files are deleted, I noticed some issues with the file_record table. This is actually related to the problem of keeping the file system and the db in sync.

• Not all uploaded files (branding and avatar) are added to the file_record, leading to the file_record being incomplete
• If a question (or answer) is deleted, it is not being checked if the text body contained a reference to a file/image. Therefore, the question is deleted, but the file_record entry is not being updated and the image remains in the file system.
• The file record_entity is not registered in init_data.go, therefore the table is not being created in the first place and no files are deleted at all. This also causes an error log whenever the clean up cron job is running.

I might missed or misunderstood some things, but I am quite sure there is something wrong here

[LinkinStars]

@DanielAuerX Thanks for the feedback.

1. Yes, for these files were not tried to record before, it may be better to add them.
2. Cleanup is based on a time set by the user, not immediately after deletion. FYI:

   answer/internal/base/cron/cron.go

   Line 81 in 3f1ed50

   if s.serviceConfig.CleanUpUploads {

3. Very important feedback. Would you be interested in submitting another separate PR to help fix this?

[DanielAuerX]

Yes sure, I will prepare a PR

[sy-records]

Thank you for your contribution.

I also think it would add extra maintenance costs, after all, most use local storage or 3rd party cloud storage. If this can exist as a plugin, that's good.

[mgerhardy]

For me it's not about the extra maintenance costs - but the extra infrastructure costs that we would have to pay when we have to use local storage (persistent volumes in k8s) and a postgres database. It's a lot easier to maintain and reduces production costs if we only have to use the (already existing) postgres database. Adding S3 or even local storage is not that easy in cluster environments.